CN109165575B - Pyrotechnic recognition algorithm based on SSD frame - Google Patents
Pyrotechnic recognition algorithm based on SSD frame Download PDFInfo
- Publication number
- CN109165575B CN109165575B CN201810885395.6A CN201810885395A CN109165575B CN 109165575 B CN109165575 B CN 109165575B CN 201810885395 A CN201810885395 A CN 201810885395A CN 109165575 B CN109165575 B CN 109165575B
- Authority
- CN
- China
- Prior art keywords
- smoke
- fire
- layer
- network
- training
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 239000000779 smoke Substances 0.000 claims abstract description 57
- 238000001514 detection method Methods 0.000 claims abstract description 35
- 238000012549 training Methods 0.000 claims abstract description 26
- 238000011176 pooling Methods 0.000 claims abstract description 12
- 238000012545 processing Methods 0.000 claims abstract description 9
- 238000013500 data storage Methods 0.000 claims abstract description 7
- 238000012360 testing method Methods 0.000 claims description 9
- 238000005286 illumination Methods 0.000 claims description 3
- 238000002372 labelling Methods 0.000 claims description 3
- 238000007781 pre-processing Methods 0.000 claims description 3
- 238000012544 monitoring process Methods 0.000 abstract description 6
- 238000011423 initialization method Methods 0.000 abstract description 4
- 238000013135 deep learning Methods 0.000 abstract 1
- 238000000034 method Methods 0.000 description 6
- 238000013528 artificial neural network Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000000605 extraction Methods 0.000 description 2
- 230000001133 acceleration Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 230000004807 localization Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000004148 unit process Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/40—Scenes; Scene-specific elements in video content
- G06V20/41—Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/29—Graphical models, e.g. Bayesian networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/40—Scenes; Scene-specific elements in video content
- G06V20/44—Event detection
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Multimedia (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Artificial Intelligence (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- General Engineering & Computer Science (AREA)
- Fire-Detection Mechanisms (AREA)
- Image Analysis (AREA)
Abstract
Aiming at the problems of complex background, more noise, easy exposure of pictures and the like of a high-speed rail monitoring video, a smoke and fire recognition algorithm based on an image deep learning SSD frame is researched, wherein a detection model training network is a reconstructed VGG16 network, and 6 convolution layers and 1 pooling layer are added on the basis of the VGG 16. Parameters to be designed for realizing the convolution layer include the number of filters, the size of the filters, the initialization method of the parameters, the initialization method of the offset, whether to start the offset item, how many pixels are added to each side of the input and the step length of the filters. The detection model trained by the network is used for designing a smoke and fire identification system, and the smoke and fire identification system comprises a video acquisition unit, an image enhancement processing unit, a smoke and fire identification unit and a data storage unit. The system is used in cooperation with the high-speed rail monitoring video, and even if the resolution ratio of the image is relatively low and a complex background exists in the high-speed rail monitoring video, the accuracy of detecting the pyrotechnic target can be ensured.
Description
Technical Field
The invention belongs to the field of flame detection, and particularly relates to a smoke and fire identification algorithm based on SSD.
Background
In a high-speed railway carriage running at a high speed, once smoke and fire alarm occur, the loss caused by the smoke and fire alarm can not be estimated, so that the smoke and fire alarm can be prevented and timely found out. The most widely used so far are mainly temperature-sensitive and smoke-sensitive fire detectors. The temperature-sensing type fire detector and the smoke-sensing type fire detector are used for judging whether alarm information is sent or not by sensing the temperature and smoke concentration around the flame and comparing the sensed information quantity with a threshold value.
With the development of technology, a flame and smoke recognition system based on image information appears, and the recognition system can process the acquired images in real time, so that the early warning time can be greatly shortened, and the early prediction and control of fire can be realized. The traditional algorithm can realize flame identification, but has low identification accuracy and higher missing report rate.
Disclosure of Invention
The high-speed rail monitoring video has the problems of complex background, more noise, easy exposure of pictures and the like, and the problems of low flame detection precision caused by the complex environmental factors are solved, and a smoke and fire recognition algorithm is provided based on the problems, and the following technical scheme is adopted:
the utility model provides a smoke and fire recognition algorithm based on SSD, the detection model that uses in the smoke and fire recognition algorithm is trained by the model network after the reconfiguration, the model network has reconfigured VGG16 network, the model network has reduced one full tie layer on VGG 16's basis, has kept two full tie layers, has increased 6 convolution layers and 1 pooling layer.
Further, the step of training the detection model includes:
step 1, acquiring smoke and fire video;
step 2, image preprocessing, which specifically comprises the steps of converting an acquired pyrotechnical video into an image sequence of a group of 10 frames, labeling pyrotechnical in the image sequence into an image set with labels, randomly distributing the image set with labels into a test set and a training set by using a classifier according to the proportion of 1:3, and converting images in the test set and the training set into an LMDB format;
step 3, training the reconstructed smoke and fire detection model by using training data until the model converges;
step 4, verifying the smoke and fire detection model by using the test data set;
further, when training the smoke and fire detection model, the 10 th layer convolution layer, the 15 th layer convolution layer, the 17 th layer convolution layer, the 19 th layer convolution layer, the 2 nd layer full-connection layer and the 6 th pooling layer in the model network use two 3*3 convolution kernels to carry out convolution, so that 6 groups of characteristic values are obtained.
Further, one of the two 3*3 convolution kernels is used for detecting whether a target exists, and the other is used for judging whether the target is smoke or fire.
Further, when training the smoke and fire detection model, the second full-connection layer in the model network is obtained by convolving the first full-connection layer with 1024 convolution kernels of 1*1, and the first full-connection layer in the model network is obtained by convolving the pooling layer connected with the first full-connection layer with 1024 convolution kernels of 3*3.
Further, the confidence level of the model network output is determined through fusion calculation of the 6 sets of characteristic values.
Compared with the prior art, the invention has the beneficial effects that: aiming at the problems of complex background, more noise, easy exposure of pictures and the like in a high-speed rail video, 6 convolution layers and 1 pooling layer are added on the basis of VGG16, and the layering characteristic extraction process can be accumulated, so that the neural network has strong characteristic extraction capability, and a trained model can obtain more complex characteristics. The added pooling layer enables the neural network to ignore the change of the relative position such as inclination and rotation of the target, reduces the dimension of the feature map, avoids over fitting and improves the precision of the model.
Another object of the present invention is to provide a smoke and fire identification system, which includes a video capturing unit, an image enhancement processing unit, a smoke and fire identification unit, and a data storage unit, wherein the video capturing unit is used for capturing a high-speed rail monitoring video, the image enhancement processing unit is used for processing a picture with insufficient illumination, the smoke and fire identification unit uses a smoke and fire detection model trained by a reconstructed detection model training network, when the confidence of smoke and fire detection is greater than 0.5, the smoke and fire is determined, and the data storage unit is used for storing history information when a fire occurs.
Further, the smoke and fire recognition system further comprises a high-definition camera and an alarm device.
Compared with the prior art, the system has the beneficial effects that the system utilizes the detection model trained by the smoke and fire recognition algorithm, and even if the resolution ratio of the acquired image is low and interference factors exist in the acquired image, the accuracy of smoke and fire target detection can be ensured, and the alarm accuracy is greatly improved.
Drawings
FIG. 1 is a diagram of a training network of a detection model after reconstruction in accordance with the present invention;
FIG. 2 is a flow chart of training a detection model;
FIG. 3 is a flow chart of the smoke detection system operation.
Detailed Description
As shown in fig. 1, the detection model training network in the present invention is a model network based on VGG16 network reconstruction, which reduces one full connection layer based on VGG16, reserves two full connection layers fc6 and fc7, and adds 6 convolution layers conv8_1, conv8_2, conv9_1, conv9_2, conv10_1, conv10_2 and 1 pooling layer pool11.
In this embodiment, the detection model is built and trained based on a caffe framework. The step of implementing the model network comprises:
the first step: creating a header file, and adding a layer header file to be placed under an include/cache/layers. Inherited from vision_laminates. The parameters of this layer are added in the caffe.
And a second step of: creating a corresponding source file, placing the source file under src/caffe/layers to realize a layerSetUp method, reading parameters of layers, initializing weights and the like, wherein the method is called when the layers are set Up and is used for initializing the layers. The Reshape method is realized, and the forward_cpu and backward_cpu methods are realized.
And a third step of: the. Cu file is created at src/caffe/layers/since GPU acceleration is required. The forward_gpu and backward_gpu methods are implemented.
Fourth step: the caffe code is recompiled.
Parameters to be designed for realizing the convolution layer include the number of filters, the size of the filters, an initialization method of the parameters, an initialization method of offset, whether to start an offset item, how many pixels are added to each side of input and the step size of the filters.
As shown in fig. 2, the step of training the model includes:
step 1, acquiring smoke and fire video;
step 2, image preprocessing, which specifically comprises the steps of converting an acquired pyrotechnical video into an image sequence of a group of 10 frames, labeling pyrotechnical in the image sequence into an image set with labels, randomly distributing the image set with labels into a test set and a training set by using a classifier according to the proportion of 1:3, and converting images in the test set and the training set into an LMDB format;
step 3, training the reconstructed smoke and fire detection model by using training data until the model converges;
and 4, verifying the smoke and fire detection model by using the test data set.
When training the smoke-fire detection model, as shown in fig. 1, the second full-connection layer (fc 7) in the model network is obtained by convolving the first full-connection layer (fc 6) with 1024 convolution kernels of 1*1, and the first full-connection layer (fc 6) in the VGG21 is obtained by convolving the pooling layer (pool 5) connected with the first full-connection layer (fc 6) with 1024 convolution kernels of 3*3. The 10 th layer convolution layer (Conv4_3), the 15 th layer convolution layer (Conv8_2), the 17 th layer convolution layer (Conv9_2), the 19 th layer convolution layer (Conv10_2), the 2 nd full connection layer (fc 7) and the 6 th pooling layer (pool 11) in the model network are convolved by using two parallel 3*3 convolution kernels, so that 6 sets of eigenvalues are obtained, and the confidence of the final output of the model network is determined by the 6 sets of eigenvalues. Wherein two parallel convolution kernels are Localization for detecting a positioning target and Confidence for judging whether the target is the target respectively.
The invention also designs a smoke and fire recognition system which comprises a video acquisition unit, an image enhancement processing unit, a smoke and fire recognition unit, a data storage unit, a high-definition camera and an alarm device.
As shown in fig. 3, the operation of the system includes:
step 1, a video acquisition unit is used for acquiring a high-speed rail monitoring video;
step 2, the image enhancement processing unit processes the pictures with insufficient illumination;
step 3, carrying out smoke and fire detection by using the trained smoke and fire detection model, and determining that smoke and fire is generated when the confidence coefficient of smoke and fire detection is greater than 0.5;
step 4, the data storage unit stores history information when fire occurs;
and 5, alarming by an alarm device.
The above embodiments are merely preferred embodiments of the present invention and are not intended to limit the present invention, and any modifications, equivalent substitutions, improvements, etc. within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (3)
1. The SSD-based smoke and fire recognition algorithm is characterized in that a detection model used in the smoke and fire recognition algorithm is trained by a redesigned model network, the model network reconstructs a VGG16 network, the model network reduces one full connection layer on the basis of VGG16, two full connection layers are reserved, and 6 convolution layers and 1 pooling layer are added;
the step of training the detection model comprises:
step 1, acquiring smoke and fire video;
step 2, image preprocessing, which specifically comprises the steps of converting an acquired pyrotechnical video into an image sequence of a group of 10 frames, labeling pyrotechnical in the image sequence into an image set with labels, randomly distributing the image set with labels into a test set and a training set by using a classifier according to the proportion of 1:3, and converting images in the test set and the training set into an LMDB format;
step 3, training the reconstructed smoke and fire detection model by using training data until the model converges;
step 4, verifying the smoke and fire detection model by using the test data set;
when training a smoke and fire detection model, a 10 th layer convolution layer, a 15 th layer convolution layer, a 17 th layer convolution layer, a 19 th layer convolution layer, a 2 nd layer full-connection layer and a 6 th pooling layer in the model network are convolved by using two 3*3 convolution kernels to obtain 6 groups of characteristic values;
one of the two 3*3 convolution kernels is used for detecting whether a target exists, and the other is used for judging whether the target is smoke or not;
when training the smoke and fire detection model, the second full-connection layer in the model network is obtained by convolving the first full-connection layer with 1024 convolution kernels of 1*1, and the first full-connection layer in the model network is obtained by convolving the pooling layer connected with the first full-connection layer with 1024 convolution kernels of 3*3.
2. A smoke and fire identification system using the SSD-based smoke and fire identification algorithm of claim 1, wherein the smoke and fire identification system includes a video acquisition unit, an image enhancement processing unit, a smoke and fire identification unit, and a data storage unit, wherein the video acquisition unit is used for acquiring a high-speed rail surveillance video, the image enhancement processing unit is used for processing a picture of insufficient illumination, the smoke and fire identification unit uses a smoke and fire detection model trained by a reconstructed detection model training network, when the confidence of smoke and fire detection is greater than 0.5, the smoke and fire is determined, and the data storage unit is used for storing history information when a fire occurs.
3. A smoke and fire identification system as claimed in claim 2 wherein the smoke and fire identification system further comprises a high definition camera and an alarm device.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201810885395.6A CN109165575B (en) | 2018-08-06 | 2018-08-06 | Pyrotechnic recognition algorithm based on SSD frame |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201810885395.6A CN109165575B (en) | 2018-08-06 | 2018-08-06 | Pyrotechnic recognition algorithm based on SSD frame |
Publications (2)
Publication Number | Publication Date |
---|---|
CN109165575A CN109165575A (en) | 2019-01-08 |
CN109165575B true CN109165575B (en) | 2024-02-20 |
Family
ID=64895071
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN201810885395.6A Active CN109165575B (en) | 2018-08-06 | 2018-08-06 | Pyrotechnic recognition algorithm based on SSD frame |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN109165575B (en) |
Families Citing this family (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN110390673B (en) * | 2019-07-22 | 2021-04-27 | 福州大学 | Cigarette automatic detection method based on deep learning in monitoring scene |
CN110379136A (en) * | 2019-07-26 | 2019-10-25 | 北京讯腾智慧科技股份有限公司 | Supervisory control for gas leak device and method for combustion gas station |
CN111091551A (en) * | 2019-12-12 | 2020-05-01 | 哈尔滨市科佳通用机电股份有限公司 | Method for detecting loss fault of brake beam strut opening pin of railway wagon |
CN111414969B (en) * | 2020-03-26 | 2022-08-16 | 西安交通大学 | Smoke detection method in foggy environment |
CN112052744B (en) * | 2020-08-12 | 2024-02-09 | 成都佳华物链云科技有限公司 | Environment detection model training method, environment detection method and environment detection device |
Citations (11)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN101751744A (en) * | 2008-12-10 | 2010-06-23 | 中国科学院自动化研究所 | Detection and early warning method of smoke |
CN103956016A (en) * | 2014-04-25 | 2014-07-30 | 蚌埠依爱消防电子有限责任公司 | Smoke and flame combined type automatic detection method based on video image |
CN106250845A (en) * | 2016-07-28 | 2016-12-21 | 北京智芯原动科技有限公司 | Flame detecting method based on convolutional neural networks and device |
CN106845410A (en) * | 2017-01-22 | 2017-06-13 | 西安科技大学 | A kind of flame identification method based on deep learning model |
CN107609470A (en) * | 2017-07-31 | 2018-01-19 | 成都信息工程大学 | The method of outdoor fire disaster early-stage smog video detection |
CN107704820A (en) * | 2017-09-28 | 2018-02-16 | 深圳市鑫汇达机械设计有限公司 | A kind of effective coal-mine fire detecting system |
CN107749067A (en) * | 2017-09-13 | 2018-03-02 | 华侨大学 | Fire hazard smoke detecting method based on kinetic characteristic and convolutional neural networks |
CN107944359A (en) * | 2017-11-14 | 2018-04-20 | 中电数通科技有限公司 | Flame detecting method based on video |
CN108052865A (en) * | 2017-07-06 | 2018-05-18 | 同济大学 | A kind of flame detecting method based on convolutional neural networks and support vector machines |
CN108257347A (en) * | 2018-01-10 | 2018-07-06 | 安徽大学 | A kind of flame image sequence sorting technique and device using convolutional neural networks |
CN108256496A (en) * | 2018-02-01 | 2018-07-06 | 江南大学 | A kind of stockyard smog detection method based on video |
Family Cites Families (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2009136894A1 (en) * | 2008-05-08 | 2009-11-12 | Utc Fire & Security | System and method for ensuring the performance of a video-based fire detection system |
US20110058706A1 (en) * | 2008-05-08 | 2011-03-10 | Utc Fire & Secunity | System and method for video detection of smoke and flame |
-
2018
- 2018-08-06 CN CN201810885395.6A patent/CN109165575B/en active Active
Patent Citations (11)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN101751744A (en) * | 2008-12-10 | 2010-06-23 | 中国科学院自动化研究所 | Detection and early warning method of smoke |
CN103956016A (en) * | 2014-04-25 | 2014-07-30 | 蚌埠依爱消防电子有限责任公司 | Smoke and flame combined type automatic detection method based on video image |
CN106250845A (en) * | 2016-07-28 | 2016-12-21 | 北京智芯原动科技有限公司 | Flame detecting method based on convolutional neural networks and device |
CN106845410A (en) * | 2017-01-22 | 2017-06-13 | 西安科技大学 | A kind of flame identification method based on deep learning model |
CN108052865A (en) * | 2017-07-06 | 2018-05-18 | 同济大学 | A kind of flame detecting method based on convolutional neural networks and support vector machines |
CN107609470A (en) * | 2017-07-31 | 2018-01-19 | 成都信息工程大学 | The method of outdoor fire disaster early-stage smog video detection |
CN107749067A (en) * | 2017-09-13 | 2018-03-02 | 华侨大学 | Fire hazard smoke detecting method based on kinetic characteristic and convolutional neural networks |
CN107704820A (en) * | 2017-09-28 | 2018-02-16 | 深圳市鑫汇达机械设计有限公司 | A kind of effective coal-mine fire detecting system |
CN107944359A (en) * | 2017-11-14 | 2018-04-20 | 中电数通科技有限公司 | Flame detecting method based on video |
CN108257347A (en) * | 2018-01-10 | 2018-07-06 | 安徽大学 | A kind of flame image sequence sorting technique and device using convolutional neural networks |
CN108256496A (en) * | 2018-02-01 | 2018-07-06 | 江南大学 | A kind of stockyard smog detection method based on video |
Also Published As
Publication number | Publication date |
---|---|
CN109165575A (en) | 2019-01-08 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN109165575B (en) | Pyrotechnic recognition algorithm based on SSD frame | |
CN108470332B (en) | Multi-target tracking method and device | |
CN111899227A (en) | Automatic railway fastener defect acquisition and identification method based on unmanned aerial vehicle operation | |
CN108256404B (en) | Pedestrian detection method and device | |
Yu et al. | Railway obstacle detection algorithm using neural network | |
CN111881853B (en) | Method and device for identifying abnormal behaviors in oversized bridge and tunnel | |
CN111709285A (en) | Epidemic situation protection monitoring method and device based on unmanned aerial vehicle and storage medium | |
CN109584213A (en) | A kind of selected tracking of multiple target number | |
CN107563299B (en) | Pedestrian detection method using RecNN to fuse context information | |
US10861172B2 (en) | Sensors and methods for monitoring flying objects | |
CN112734731A (en) | Livestock temperature detection method, device, equipment and storage medium | |
CN109815884A (en) | Unsafe driving behavioral value method and device based on deep learning | |
CN106056624A (en) | Unmanned aerial vehicle high-definition image small target detecting and tracking system and detecting and tracking method thereof | |
CN113688709B (en) | Intelligent detection method, system, terminal and medium for wearing safety helmet | |
CN113468996A (en) | Camouflage object detection method based on edge refinement | |
CN113378649A (en) | Identity, position and action recognition method, system, electronic equipment and storage medium | |
Henrio et al. | Anomaly detection in videos recorded by drones in a surveillance context | |
CN116824335A (en) | YOLOv5 improved algorithm-based fire disaster early warning method and system | |
CN114359838A (en) | Cross-modal pedestrian detection method based on Gaussian cross attention network | |
CN111027555A (en) | License plate recognition method and device and electronic equipment | |
CN115410324A (en) | Car as a house night security system and method based on artificial intelligence | |
Cao et al. | YOLO-SF: YOLO for fire segmentation detection | |
Roy et al. | Annotation and benchmarking of a video dataset under degraded complex atmospheric conditions and its visibility enhancement analysis for moving object detection | |
Chauhan et al. | Study of moving object detection and tracking for video surveillance | |
CN115131826B (en) | Article detection and identification method, and network model training method and device |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant |