CN113158882B - Bionic compound eye privacy protection intelligent binary modeling method for home video monitoring - Google Patents
Bionic compound eye privacy protection intelligent binary modeling method for home video monitoring Download PDFInfo
- Publication number
- CN113158882B CN113158882B CN202110418738.XA CN202110418738A CN113158882B CN 113158882 B CN113158882 B CN 113158882B CN 202110418738 A CN202110418738 A CN 202110418738A CN 113158882 B CN113158882 B CN 113158882B
- Authority
- CN
- China
- Prior art keywords
- visual
- privacy protection
- bionic
- compound eye
- image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
- G06F18/2411—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on the proximity to a decision surface, e.g. support vector machines
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Health & Medical Sciences (AREA)
- Evolutionary Computation (AREA)
- Bioinformatics & Computational Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Artificial Intelligence (AREA)
- Life Sciences & Earth Sciences (AREA)
- Evolutionary Biology (AREA)
- General Health & Medical Sciences (AREA)
- Psychiatry (AREA)
- Social Psychology (AREA)
- Human Computer Interaction (AREA)
- Multimedia (AREA)
- Image Analysis (AREA)
Abstract
The invention relates to a bionic compound eye privacy protection intelligent binary modeling method for home video monitoring, which comprises the following steps: (1) selecting a video segment based on the public behavior recognition data set, referring to a compound eye insect, and performing bionic compound eye visual processing on the video data; (2) respectively extracting image visual characteristics and statistical characteristics of a video frame sequence with bionic visual privacy protection; (3) mapping the image features to image visual privacy protection scores; (4) extracting a moving target area in a video from a video frame sequence with bionic visual privacy protection; (5) extracting HOG characteristics from the foreground moving target and obtaining the human body posture recognition rate through an SVM classifier; (6) and establishing a binary statistical model between the visual privacy protection score and the human posture recognition rate, and balancing the relationship between the visual privacy protection score and the human posture recognition rate. The invention balances the conflict between privacy protection and intelligent application while carrying out privacy protection on the video data, and has higher practical application value.
Description
Technical Field
The invention belongs to the field of computer vision and pattern recognition, and particularly relates to a bionic compound eye privacy protection intelligent binary modeling method for home video monitoring, in particular to a bionic compound eye privacy protection and typical human body posture recognition binary statistical modeling method for home video monitoring.
Background
The continuous development of technologies such as the internet and the like in the big data era improves the life quality of people, but simultaneously, a plurality of new problems are generated, and the personal privacy safety problem is brought to the first time. The image and video data contain a large amount of personal behavior habits and life styles, and the leakage of the data causes serious privacy security problems, so that the security of the image and video data is widely concerned.
In addition, with the continuous development of computer vision technology, video monitoring has been expanded to the scene of living at home from public places such as airports and banks, admittedly, intelligent video monitoring system can realize all-weather monitoring, reaches and discovers that abnormal conditions in time send out functions such as warning, and has important application prospect particularly to health monitoring of solitary old people such as falling down or danger early warning of unattended children such as falling down from bed. However, the dependence of the current mainstream video application on video quality is often accompanied by the personal privacy disclosure risk generated by video content, especially the foreground object easily involved in video intelligent processing, such as human face signs or close actions, and the background environment, such as money jewelry or famous and precious artworks, which are actually related to the most sensitive personal privacy problem in a home scene. Therefore, the dependence of the current intelligent processing on scene content can be said to be the biggest obstacle of the popularization and application of video monitoring at home.
ZL 2020107571413 discloses a robot visual privacy behavior identification and protection method, which is characterized in that segments containing privacy in a robot camera video stream are replaced by corresponding guardian element text information, approximate equivalence conversion of privacy scenes is achieved, and therefore the privacy scenes are protected.
ZL 2019101700704 discloses a passive human body identification method based on narrow-band radio frequency link sampling, and the method is characterized in that biological characteristic information generated by walking of a pedestrian to be identified is obtained through constructed multi-angle interlaced radio frequency link sampling, forced cooperation and visual privacy invasion of users in a limited environment are effectively avoided, the method needs pedestrian walking movement as a foreground target, and the problem of personal privacy disclosure still exists.
The nature is the source of various major inventions of human beings, and the human intelligence not only stays in observing and recognizing the biology world, but also can imitate the biology. There is an interesting phenomenon about the relationship of biological vision to cognitive performance: there are a large number of environmentally highly adaptive creatures in nature, although their vision level can only be classified to the level of human legal blindness, which means that simulating low-level biological vision may provide an effective solution to the privacy security problem of video surveillance for home applications.
Disclosure of Invention
In order to solve the problem that the existing household application intelligent video monitoring relates to privacy invasion, a bionic compound eye method is provided for visual privacy protection, and a correlation statistical model between the visual privacy protection level and human body posture recognition is established.
In order to achieve the purpose, the invention is realized by the following technical scheme:
the invention relates to a bionic compound eye privacy protection intelligent binary modeling method for home video monitoring, which comprises the following steps:
(1) selecting an effective video segment based on a public behavior recognition data set, referring to several common compound eye insects, performing bionic compound eye vision processing on video data to realize a visual privacy protection effect, simulating the compound eye vision of the insects by adopting an AcuityView method in the step, and controlling the bionic vision effect by controlling the visual acuity and the visual observation distance of different insects;
(2) respectively extracting image vision and statistical characteristics of a video frame sequence with bionic vision privacy protection, wherein in the step, a vision significance weighting GCS-LBP operator is adopted to extract the visual characteristics of the image, and fractal dimension distribution and Benford's law of wavelet domain and gradient domain are adopted as the statistical characteristics of the image;
(3) mapping the image features to image visual privacy protection scores by adopting a support vector regression model;
(4) for the video frame sequence with bionic visual privacy protection, extracting a moving target area in a video by using a low-rank sparse decomposition theory;
(5) extracting HOG characteristics from the foreground target and obtaining the recognition rate of human posture recognition through an SVM classifier;
(6) and a binary statistical model between the visual privacy protection score and the human body posture recognition rate is established by adopting a statistical modeling method, the relation between the visual privacy protection score and the human body posture recognition rate is balanced, and the effective bionic visual privacy protection processing range is obtained through the model.
The invention provides a new visual privacy concept for performing privacy protection on bionic compound eye vision, and provides a binary model for bionic compound eye visual privacy and human body posture identification in a statistical form. Firstly, the study tries to ensure the retention of enough cognitive information at a level far lower than the visual level of human eyes by using the compound eye visual mechanism of insects for reference, and realizes the effect of visual privacy protection; secondly, by referring to the idea of image quality evaluation, the image visual characteristics, local fractal dimension and other statistical characteristics are fused, and visual privacy objective evaluation indexes suitable for bionic vision are provided; and finally, taking human body posture recognition with a high common value in home monitoring as an example, providing a set of binary statistical model construction method for penetrating through bionic compound eye vision privacy and intelligent application.
Compared with the prior art, the invention has the following beneficial effects:
(1) aiming at the privacy leakage risk of the traditional video in the visual aspect, the imaging improvement is carried out by using the compound eye insect visual mechanism for reference, so that the purpose of protecting the visual privacy is achieved;
(2) according to the invention, from the aspect of vision and statistics, quantitative evaluation facing to the protection level of visual privacy is formed by adopting visual significance weighted GCS-LBP and support vector regression fused with Benford's law-fractal dimension;
(3) the invention selects human body pose recognition as an example of household video application, and establishes a set of binary statistical model which is communicated with bionic binocular vision privacy and intelligent application, so as to realize quantitative balance discussion of the two on a statistical level.
Drawings
FIG. 1 is a schematic flow diagram of the process of the present invention.
Fig. 2 is a schematic view of image visual privacy protection of bionic bee compound eyes.
Fig. 3 is a schematic view of image visual privacy protection of a bionic butterfly compound eye.
Fig. 4 is a diagram of foreground extraction results of an original video and a bionic bee vision video, where the first column on the left side is an original video frame and a foreground frame, the second column is a bionic video frame and a foreground frame, and the third column and the fourth column are a bionic video frame and a foreground frame.
Detailed Description
In the following description, for purposes of explanation, numerous implementation details are set forth in order to provide a thorough understanding of the embodiments of the invention. It should be understood, however, that these implementation details are not to be interpreted as limiting the invention. That is, in some embodiments of the invention, such implementation details are not necessary.
The invention relates to a bionic compound eye privacy protection intelligent binary modeling method for home video monitoring, which comprises the following steps:
(1) based on the public behavior recognition data set, an effective video segment is selected, and bionic compound eye visual processing is carried out on the video data by referring to several common compound eye insects such as bees and butterflies, so that the visual privacy protection effect is realized.
The insect has a visual effect which is far inferior to that of human beings due to the unique compound eye structure, and researches show that the visual acuity determines details which can be seen by organisms in a visual scene, insect animals with the compound eye structure have the visual acuity which is far lower than that of other animals, the compound eye is evolved by insects in order to adapt to a living environment and consists of an indefinite number of small eyes, and the more the small eyes, the clearer the visual image and the more the visual image is, the more the insect is, the visual image is, the more the insect is, the visual image is. The bionic insect vision research is the research of an optical imaging system under a complex visual scene, in the optical system, a perception image of the scene is the convolution of the scene and a Point Spread Function (PSF), the Fourier transform of the PSF is also an important characteristic of the imaging system, namely, a Modulation Transfer Function (MTF), the MTF is given by an equation (1), wherein ν is a spatial frequency, R is a Minimum Resolvable Angle (MRA) of the visual system, and the angle width of a recognizable narrowest black-white line pair.
In this embodiment, an AcuityView, r-package, which simulates the vision of an animal, is used, in which the Minimum Resolvable Angle (MRA) of different animals, in degrees, the actual width of the image and the visual observation distance, needs to be input, wherein the minimum resolvable angle is the reciprocal of the visual acuity, which is expressed in cycles/degree, which is the number of black and white stripe pairs that an observer can distinguish at a single viewing angle, and a higher visual acuity means that a small object can be recognized at a farther place.
Therefore, a bionic compound eye visual privacy protection human posture recognition data set is constructed, the vision of bionic bees with visual acuity of 1.9 weeks/degree and the vision of butterflies with visual acuity of 2.7 weeks/degree are selected, the bionic visual observation distance is set to be 0.2m to 1m in sequence, the interval is 0.2m, and effective video segments are selected for recognition based on the public behavior recognition data set in a bionic visual privacy protection state.
(2) For the video frame sequence with bionic visual privacy protection, image vision and statistical characteristics are respectively extracted, and texture characteristics of an image which can be extracted by an LBP operator are common image vision characteristics.
In the embodiment, a generalized central symmetry local binary pattern (GCS-LBP) operator is adopted, and compared with a traditional LBP operator, the GCS-LBP has better robustness for illumination change, image noise and small image distortion, and can better extract texture structure characteristics of an image.
Wherein, I m And I m+(m/2) Respectively, two pixel points s which are symmetric about a central pixel point in a GCS-LBP window containing m pixel points and having a radius R T Is an assignment function with a threshold value of T:
the GCS-LBP operator extracts structural features of the image, in order to enable the extracted features to better reflect and accord with perception characteristics of a human visual system, a visual saliency model GBVS is introduced, the distance between any two points in the image is described through a feature value, a saliency map is obtained through a Markov random field steady state of distance weight, the GBVS saliency model is combined with the GCS-LBP operator to extract visual features of the image which accord with human visual perception for evaluating visual privacy protection degree of the image, and the statistical histogram of the visual saliency model weighted GCS-LBP operator is represented as follows:
H＝{h(0)，h(1)，......h(2 m /2-1)} (4)
h(k)＝∑ i ∑ j I LBP (i，j)·Δ(I GBVS (i，j)·k) (5)
in this embodiment, the average value of the GCS-LBP operator statistical histogram is weighted by the visual saliency model as the extracted visual features of the image:
on the basis of extracting the visual features of the image, extracting fractal dimension distribution of the image and Benford's law as the statistical features of the image, which can be used for obtaining the difference between the original image and the statistical mode of the visual privacy protection image, Benford's law, also called first numerical law, indicates that in many practical data sets, the first significant digit d (d e 1, …, 9) appears with probability, and formula (8) is a first numerical distribution calculation formula.
Benford's law applies to digital distributions if the distribution spans a considerable order of magnitude.
The present embodiment extracts feature vectors using the normalized first number distribution in the wavelet domain and the image gradient domain, respectively.
Fractal analysis is a source of fractal geometric ideas, which is a study on processing irregular and self-similar objects, and formula (9) for calculating fractal dimension is as follows:
d＝lim ε→0 [logB(ε)/log(1/ε)] (9)
where ε is the length of one side of the minicubes and N (ε) is the number of test forms covered by the minicubes.
Each pixel in the original image is used as the center of a 7 multiplied by 7 rectangular neighborhood to generate a fractal dimension image, the fractal dimension is calculated from the neighborhood, and a box counting technology is applied to determine the fractal dimension of the gray level image block.
(3) And mapping the image features to image visual privacy protection scores by adopting a support vector regression model: the support vector regression is widely applied to quality score regression in image quality evaluation, and an SVR support vector regression model is adopted for extracted visual-statistical characteristics to map image characteristics to image scores.
This example uses the ε -SVR regression equation to evaluate the quality of an image
Wherein K (v) i V) is a kernel function, Lagrange's operator and parameters need to be determined during training, and the solution of the formula can be converted into an optimization problem
The radial basis function kernels used are:
and (3) taking the characteristics of the training set and corresponding Difference Mean (DMOS) values as input, training an SVR model, and finally taking the characteristics of the test set as the input of the training model to obtain the predicted bionic compound eye visual privacy protection score.
(4) For the video frame sequence with the bionic visual privacy protection function, a low-rank sparse decomposition theory is used for extracting a moving target area in a video, human body posture identification with high value in intelligent monitoring is taken as an example, and the bionic compound eye processing degree can be limited through an identification result in the visual privacy protection state.
Firstly, a low-rank sparse decomposition method is adopted, separation of a foreground and a background is realized, background information is removed, the recognition rate of human body postures can be improved, the low-rank sparse theory is that an original low-rank matrix of a given matrix is recovered, a moving target in the foreground of a video sequence can be regarded as a sparse part, a static background can be regarded as a low-rank part, X in a formula (13) represents a video sequence matrix, A is a low-rank matrix corresponding to a constant background, and E is a sparse matrix corresponding to a moving target.
X＝A+E (13)
For how to recover the matrices a and E from the known X, an optimization solution model converts the matrices a and E into an l1 norm solution problem, as shown in equation (14), since the optimization model can recover the matrices a and E under the condition of a large error, the method is suitable for separating the foreground and the background of the video under the bionic visual privacy protection state.
s.t.D＝A+E (14)
(5) Extracting HOG characteristics from the foreground target and obtaining the recognition rate of human posture recognition through an SVM classifier;
in the embodiment, the HOG features are extracted from the foreground video frame obtained by low-rank sparse decomposition, and the directional gradient Histogram (HOG) can well describe the features of the local target region.
The HOG algorithm divides an image into small cell units, then acquires gradient or edge direction histograms of all pixel points in the cell units, and finally combines the histograms to form a feature descriptor.
(6) And (3) establishing a binary statistical model between the visual privacy protection score and the human posture recognition rate by adopting a statistical modeling method, and balancing the relation between the visual privacy protection score and the human posture recognition rate.
In order to balance the bionic compound eye vision privacy and the human body posture recognition, the embodiment establishes a binary statistical model between the bionic compound eye vision privacy and the human body posture recognition. When the visual privacy protection score of the image is low, the visual information contained in the image does not meet the requirement of privacy protection, and when the visual privacy protection score of the image is high, the recognition rate of the image is greatly reduced, so that the application value of the image in the visual privacy protection state is lost.
Therefore, it is desirable to find a balance between the visual privacy score and the recognition rate, i.e. ensure high recognition rate while high visual privacy protection, and establish a quantitative association statistical model between visual security and human body posture recognition rate.
The invention balances the conflict between privacy protection and intelligent application while carrying out privacy protection on the video data, and has higher practical application value.
The above description is only an embodiment of the present invention, and is not intended to limit the present invention. Various modifications and alterations to this invention will become apparent to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of the claims of the present invention.
Claims (4)
1. A bionic compound eye privacy protection intelligent binary modeling method facing home video monitoring is characterized in that: the method comprises the following steps:
(1) bionic insect compound eye visual privacy protection: selecting effective video segments based on the public behavior recognition data set, referring to compound eye insects, and performing bionic compound eye visual processing on the video data;
(2) respectively extracting image visual characteristics and statistical characteristics of a video frame sequence with bionic visual privacy protection;
(3) mapping the image visual features and the statistical features to image visual privacy protection scores;
(4) extracting a moving target area in a video from a video frame sequence with bionic visual privacy protection;
(5) extracting HOG characteristics from the foreground moving target and obtaining the human body posture recognition rate through an SVM classifier;
(6) establishing a binary statistical model between the visual privacy protection score and the human body posture recognition rate, and balancing the relationship between the two;
wherein: in the step (2), the visual features of the image are extracted through a visual saliency weighting GCS-LBP operator, fractal dimension distribution, Benford's law of wavelet domain and gradient domain are taken as statistical features of the image, and the visual saliency weighting GCS-LBP operator is expressed as:
GCS-LBP operator:
wherein the content of the first and second substances,
and
respectively comprising m pixel points, two pixel points symmetrical about a central pixel point in a GCS-LBP window with radius R,
introducing a visual saliency model GBVS, combining the visual saliency model GBVS with a GCS-LBP operator, and weighting a statistical histogram of the GCS-LBP operator by the visual saliency model:
and weighting the average value of the GCS-LBP operator statistical histogram by using a visual significance model as the extracted visual features of the image:
2. the intelligent binary modeling method for bionic compound eye privacy protection oriented to home video monitoring as claimed in claim 1, is characterized in that: in the step (1), the compound eye vision of the compound eye insects is simulated by adopting an AcuityView method, and the bionic visual effect is controlled by controlling the visual acuity and the visual observation distance of different insects.
3. The intelligent binary modeling method for bionic compound eye privacy protection oriented to home video monitoring as claimed in claim 1, is characterized in that: and (4) performing low-rank sparse decomposition on the bionic visual privacy protection state video data to obtain an interested moving target foreground area.
4. The intelligent binary modeling method for bionic compound eye privacy protection oriented to home video monitoring as claimed in claim 1, is characterized in that: in the step (6), a statistical modeling method is adopted to establish the relationship between the visual privacy protection score and the human body posture recognition rate, and an effective bionic visual privacy protection processing range is obtained through a model.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110418738.XA CN113158882B (en) | 2021-04-19 | 2021-04-19 | Bionic compound eye privacy protection intelligent binary modeling method for home video monitoring |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110418738.XA CN113158882B (en) | 2021-04-19 | 2021-04-19 | Bionic compound eye privacy protection intelligent binary modeling method for home video monitoring |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113158882A CN113158882A (en) | 2021-07-23 |
| CN113158882B true CN113158882B (en) | 2022-08-05 |
Family
ID=76868814
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110418738.XA Active CN113158882B (en) | 2021-04-19 | 2021-04-19 | Bionic compound eye privacy protection intelligent binary modeling method for home video monitoring |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113158882B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113807275B (en) * | 2021-09-22 | 2023-10-24 | 南京邮电大学 | Household video fall detection method based on privacy protection GAN feature enhancement |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111199538A (en) * | 2019-12-25 | 2020-05-26 | 杭州中威电子股份有限公司 | Privacy protection degree evaluation method for multilayer compressed sensing image |
| CN111860434A (en) * | 2020-07-31 | 2020-10-30 | 贵州大学 | Robot vision privacy behavior identification and protection method |
-
2021
- 2021-04-19 CN CN202110418738.XA patent/CN113158882B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111199538A (en) * | 2019-12-25 | 2020-05-26 | 杭州中威电子股份有限公司 | Privacy protection degree evaluation method for multilayer compressed sensing image |
| CN111860434A (en) * | 2020-07-31 | 2020-10-30 | 贵州大学 | Robot vision privacy behavior identification and protection method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113158882A (en) | 2021-07-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Sun et al. | Abnormal event detection for video surveillance using deep one-class learning | |
| Saeedi et al. | Infrared and visible image fusion using fuzzy logic and population-based optimization | |
| Lu et al. | Learning optimal seeds for diffusion-based salient object detection | |
| Xu et al. | Crowd density estimation based on rich features and random projection forest | |
| Sang et al. | Improved crowd counting method based on scale-adaptive convolutional neural network | |
| Horng et al. | Multilevel image thresholding selection based on the firefly algorithm | |
| CN110458038B (en) | Small data cross-domain action identification method based on double-chain deep double-current network | |
| Nagi et al. | A MATLAB based face recognition system using image processing and neural networks | |
| CN109101865A (en) | A kind of recognition methods again of the pedestrian based on deep learning | |
| Wang et al. | Blink detection using Adaboost and contour circle for fatigue recognition | |
| CN109635636B (en) | Pedestrian re-identification method based on fusion of attribute characteristics and weighted blocking characteristics | |
| Siraj et al. | Digital image classification for Malaysian blooming flower | |
| Wu et al. | Manifold kernel sparse representation of symmetric positive-definite matrices and its applications | |
| CN113963032A (en) | Twin network structure target tracking method fusing target re-identification | |
| Janku et al. | Fire detection in video stream by using simple artificial neural network | |
| CN108921215A (en) | A kind of Smoke Detection based on local extremum Symbiotic Model and energy spectrometer | |
| CN108320281B (en) | Image significance detection method and terminal based on multi-feature diffusion | |
| CN113158882B (en) | Bionic compound eye privacy protection intelligent binary modeling method for home video monitoring | |
| Miao et al. | Abnormal behavior learning based on edge computing toward a crowd monitoring system | |
| Nga et al. | Combining binary particle swarm optimization with support vector machine for enhancing rice varieties classification accuracy | |
| CN117437691A (en) | Real-time multi-person abnormal behavior identification method and system based on lightweight network | |
| CN113449550A (en) | Human body weight recognition data processing method, human body weight recognition method and device | |
| Chen | Evaluation technology of classroom students’ learning state based on deep learning | |
| CN113591797B (en) | Depth video behavior recognition method | |
| Kuang et al. | Learner posture recognition via a fusing model based on improved SILTP and LDP |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |