CN110020987B - Medical image super-resolution reconstruction method based on deep learning - Google Patents
Medical image super-resolution reconstruction method based on deep learning Download PDFInfo
- Publication number
- CN110020987B CN110020987B CN201910224792.3A CN201910224792A CN110020987B CN 110020987 B CN110020987 B CN 110020987B CN 201910224792 A CN201910224792 A CN 201910224792A CN 110020987 B CN110020987 B CN 110020987B
- Authority
- CN
- China
- Prior art keywords
- resolution
- data
- training
- medical image
- super
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 41
- 238000013135 deep learning Methods 0.000 title claims abstract description 9
- 238000012549 training Methods 0.000 claims abstract description 37
- 238000005070 sampling Methods 0.000 claims description 5
- 230000003321 amplification Effects 0.000 claims description 4
- 238000003199 nucleic acid amplification method Methods 0.000 claims description 4
- 238000012545 processing Methods 0.000 claims description 3
- 230000003416 augmentation Effects 0.000 claims description 2
- PCHJSUWPFVWCPO-UHFFFAOYSA-N gold Chemical compound [Au] PCHJSUWPFVWCPO-UHFFFAOYSA-N 0.000 claims description 2
- 239000011159 matrix material Substances 0.000 claims description 2
- 238000013519 translation Methods 0.000 claims description 2
- 238000004364 calculation method Methods 0.000 claims 1
- 238000004140 cleaning Methods 0.000 claims 1
- 238000002560 therapeutic procedure Methods 0.000 claims 1
- 238000003384 imaging method Methods 0.000 abstract description 2
- 230000000694 effects Effects 0.000 description 8
- 238000011156 evaluation Methods 0.000 description 6
- 230000003902 lesion Effects 0.000 description 6
- 230000007547 defect Effects 0.000 description 3
- 238000011161 development Methods 0.000 description 2
- 238000002474 experimental method Methods 0.000 description 2
- 238000002372 labelling Methods 0.000 description 2
- 238000012360 testing method Methods 0.000 description 2
- 230000000903 blocking effect Effects 0.000 description 1
- 238000012512 characterization method Methods 0.000 description 1
- 238000007796 conventional method Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000010200 validation analysis Methods 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4046—Scaling of whole images or parts thereof, e.g. expanding or contracting using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H30/00—ICT specially adapted for the handling or processing of medical images
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Health & Medical Sciences (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Radiology & Medical Imaging (AREA)
- Evolutionary Computation (AREA)
- Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
- Artificial Intelligence (AREA)
- Epidemiology (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Primary Health Care (AREA)
- Public Health (AREA)
- Image Processing (AREA)
- Measuring And Recording Apparatus For Diagnosis (AREA)
Abstract
The invention provides a method for reconstructing medical image super-resolution based on deep learning, which is characterized in that a correct super-resolution result can be obtained by directly inputting medical images into a system, a network is trained by a large amount of high-quality medical image data, position information of focuses in the medical image data is introduced in a training stage, the position information refers to center coordinates and sizes of the focuses or fine edge marks of the focuses, and the problem that medical images after super-resolution caused by training of the medical image super-resolution network of other methods lose original imaging significance can be solved by adding the prior information.
Description
Technical Field
The invention belongs to the technical field of computer vision, and particularly relates to a medical image super-resolution reconstruction method based on deep learning.
Background
With the development of deep learning, the level of computer vision and image processing is continually breaking through the bottleneck of traditional methods in some scenarios. Image super-resolution is also being developed as an important technology in computer vision. The super-resolution method based on deep learning in a natural scene is endless, various different network design modes are derived, the network depth is continuously increased, the feature multiplexing capability is continuously enhanced, PSNR indexes under different scales are continuously increased, and the method has wide application in actual scenes such as video monitoring, video restoration, digital high definition, satellite images and the like.
Because of limitations of medical instrument hardware or limitations of dose during medical image shooting, super-resolution has a great development space in the medical image field, and some researchers migrate methods in natural images directly to medical images at present, although some improvements are obtained in PSNR indexes and visual effects as if the methods in the natural images are also improved, the methods in the natural images are migrated directly to more serious medical images, and the methods are not correspondingly adjusted according to specific characteristics of the medical images are doubtful. Unlike natural images, medical images have the problems of positive areas (focus) and negative areas (normal tissues), and the conventional method for training images in a blocking way by using the super-resolution of natural image images directly leads to model study to be askew to the normal tissues due to imbalance of yin and yang data, so that medical image information with imaging significance is damaged. And PSNR is used as an evaluation index of an image global information signal layer, and only PSNR is used as an evaluation index for measuring the super-resolution effect of the medical image, so that the super-resolution effect of a truly important area cannot be evaluated well, and the signals of a focus area can be annihilated in a large number of signals of normal tissue areas.
Disclosure of Invention
The invention provides a medical image super-resolution reconstruction method based on deep learning, which aims to solve the problems as mentioned in the background art, and mainly comprises the following steps:
(1) For the medical image characteristics, the additional labeling information of the medical data set is utilized, a new data set making and training strategy is provided to solve the problem of unbalanced yin and yang samples, and the method is hereinafter referred to as PN-sample balance.
(2) As no known and effective evaluation index for the super-resolution reconstruction effect of the medical image exists at present, the invention provides an evaluation index for jointly evaluating the super-resolution reconstruction effect of the medical image by combining a CAD system with PSNR.
The introduction of the CAD system can make up the defect that PSNR can not measure whether super-resolution effectively enhances focus areas really focused by doctors. Only if the super-resolution technique is performed in the correct direction, the improvement of the PSNR index is significant.
Compared with the prior art, the invention has the following obvious advantages:
1. the PN-sample balance method can effectively ensure the training stability of the model and ensure that the yin and yang areas can obtain a better balanced learning effect.
2. The provided joint evaluation index can effectively identify the situation of overall PSNR improvement caused by the improvement of most negative areas when the network training is askew. The effectiveness of the medical image super-resolution effect can be ensured.
Description of the drawings:
FIG. 1 is a flow chart of the present invention
FIG. 2 is a 3D dense network architecture for use with the present invention;
FIG. 3 is a network Block diagram of the Dense Block and Compressor of the present invention;
FIG. 4 illustrates the performance of the validation PSNR, the left graph is the prior art method, and the right graph is the method proposed by the present invention 1;
FIG. 5 is a graph comparing CAD system effects for verification of the present invention 2;
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention will be further described in detail with reference to the accompanying drawings 1 to 5 and examples. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention. The technical features of the embodiments of the present invention described below may be combined with each other as long as they do not collide with each other.
The invention provides a medical image super-resolution reconstruction method and system based on deep learning. The medical image is directly input into the system to obtain a correct super-resolution result. The network is trained by a large amount of high-quality medical image data, wherein the difference between the training method and other methods is that the focus position information in the medical image data is introduced in the training stage, but the testing is not needed, and the practical requirements are met. The location information here refers to the center coordinates and dimensions of the lesion or the subtle edge labeling of the lesion. By adding the prior information, the problem that medical images with super resolution are lost after the super resolution caused by the training of medical image super resolution networks by other methods, which is called PN-sample balance method, can be solved. And the method provides that the CAD system is combined with PSNR to perform real and effective evaluation on the super-resolution performance of the medical image, the super-resolution method has significance in improving the PSNR on the premise of ensuring the improvement of the performance of the CAD system, if the PSNR of which the performance of the CAD system is reduced and improved due to the data after the super-resolution is also meaningless, only a few irrelevant areas can be enhanced, and focus area information really concerned by people is destroyed, which is also the defect of PSNR, and the introduction of the CAD system can effectively make up for the defect.
In the embodiment of the present invention, the step (1) mainly includes: acquisition of high quality medical images, and introduction and processing of lesion location information.
Specifically, in the step (1-1), a high-quality medical image data set is collected, the data is cleaned, and the poor data in the data set is deleted. Since our task is super-resolution reconstruction, the high resolution in the three directions z, y, x is the primary factor we consider, and we generally choose spacing in all three directions less than 1mm as high quality data. Spacing means spatial information in the actual physical space contained by a voxel in the medical image, and smaller Spacing in three directions results in an object of actual size that will be composed of more voxels, i.e. with higher resolution. And we need to collect the location information of the lesions in the data as a priori information that we additionally consider when training.
Step (1-2), after collecting the data, we start to make super-resolution training data set, for higher applicability, we use multi-scale simultaneous training mode, firstly, downsampling the high quality data set in the target direction of three directions by corresponding scale, then interpolate to original resolution by bicubic interpolation method or cubic spline method, as simulated low resolution data, and the original data form low resolution (I LR ) And high resolution (I HR ) Data pairs (data-pairs). For lesion location information, we convert the real coordinate information to voxel coordinate information that matches the three-dimensional image by Vo pixel coordinates= (World coordinates-Origin)/Spacing. The coordinates are specifically classified into two-dimensional coordinates (y, x) or three-dimensional coordinates (z, y, x) according to the dimensions of the specific medical image. Vo pixel coordinates are the corresponding coordinates in the medical image matrix, world coordinates are the World coordinate system of the lesion, origin is the origin of coordinates of the photographing instrument, and spacing is the spatial information contained in one voxel mentioned above. We need to record the pixel corresponds that match the images we use, save them in memory for guiding the sampling strategy when training.
And (2) training and optimizing the main packet network.
Specifically, step (2-1) is different from other methods in that data are diced directly before training and used for training, an on-line dicing strategy guided by position information is adopted, random numbers are generated according to the manually set yin-yang data proportion when training data are obtained each time, the random numbers 0,1 and 2 can be generated first assuming that the yin-yang ratio 2:1 is a reasonable ratio, if 0 is generated, coordinate information is randomly obtained from positive area coordinate information stored in a memory in step (1-2), disturbance of 20 voxels is added in each dimension of a space, the disturbed coordinates are used as center coordinates, and the low-resolution high-resolution data pair (I LR 、I HR data-pairs), wherein the larger data blocks are larger than the data blocks in formal training, and redundant parts are cut off after the amplification, so that the situation that the training data has padding at the edge after online data amplification is completed is prevented. By performing the following steps on the training data block: the method comprises the steps of cutting off redundant parts of edges to 42 x 42 (3-dimensional data and 42 x 42 two-dimensional data) after random augmentation operations such as horizontal overturning, vertical overturning, front-back overturning, translation, scaling, multi-angle rotation and the like for network training. If the random number is 1 or 2, the random effective position is sampled in the image, and then the same block cutting and amplification operation is carried out on the positive area for training.
And (2-2) sending a batch of data blocks which meet the yin-yang ratio and are obtained by sampling in the step (1-2) into a subsequent network for training. Obtaining a super-resolution reconstructed high resolution image (I SR ) The obtained I SR And I as gold standard HR MSE loss is performed
And calculating reconstruction loss, and optimizing network parameters through a back propagation algorithm (back propagation). Where i, j, k are coordinate numbers in three directions, respectively. And after the training is stable, selecting a proper super-resolution network model as a follow-up use.
The network is designed by using a dense connection mode of the dense with excellent performance, multiplexing and fusing are performed on low-layer and high-layer features to achieve a better result, as shown in fig. 2, and the parameters used under the same performance are smaller, so that the characterization capability of the limited parameters can be developed maximally, as shown in fig. 3, the basic structures of our dense block and compact are shown, the dense block is a dense connection mode, the main feature extraction work is completed, and the compact mainly plays a role in data dimension reduction because 3D data is very easy to cause the shortage of video memory.
When the first invention point is stated so far and the second invention point is not related, namely whether the superdivision network enhances the medical image correctly, the PSNR index is far beyond the traditional training method, and the existing result of the superdivision network is compared with that of the superdivision network. The problem of unbalance of yin and yang samples in medical images is not considered, and the performance is limited although the problem is greatly improved. And problems with existing superdivision methods can be found in the present invention 2.
And 3, verifying the prior super-resolution method on the invention 2, and verifying the practical effectiveness of the super-resolution method. While the validity of the present invention 1 can be verified. Note that the present invention 2 has been verified in terms of actual data, i.e., the low-resolution data used in the experiment is truly collected low-resolution data, not artificial false low-resolution data obtained by sampling. Wherein:
step (3-1), training and testing the CAD system on the data without any superdivision operation, as the baseline of the performance achieved by the batch of data.
Step (3-2) using the model trained in step (2) to perform super-division operation on the real low-resolution data to enhance the real low-resolution data to obtain I SR And use I SR And (3) replacing the original low-resolution data, and training a CAD system algorithm according to the same parameters to obtain the performance index of the invention 1.
Step (3-3) training was performed using the same configuration as in step (2) except for the yin-yang balancing method of the present invention 1, as a representative of the existing method.
Comparing the three experiments in the step (3-4), the existing superdivision method is found to cause a large number of false positives, which means that the medical information in the influence is destroyed. The method of the present invention 1 is effective in avoiding this problem and the performance is improved as shown in fig. 5.
Step (3-5) shows that the invention 2 can find out the problem of error learning existing in the existing super-resolution method, and can evaluate the super-resolution performance of the medical image more comprehensively. The invention 1 can effectively solve the problems, and accurately learn the characteristics of the medical image, thereby achieving the effect of accurately improving the quality of the medical image.
The above embodiments are only exemplary embodiments of the present invention and are not intended to limit the present invention, the scope of which is defined by the claims. Various modifications and equivalent arrangements of this invention will occur to those skilled in the art, and are intended to be within the spirit and scope of the invention.
Claims (1)
1. A medical image super-resolution reconstruction method based on deep learning is characterized by comprising the following steps:
step 1, obtaining high-quality medical images
Step (1-1), collecting a high-quality medical image dataset and cleaning the dataset;
step (1-2), preparing a super-resolution training data set, adopting a multi-scale simultaneous training mode, firstly performing downsampling operation of corresponding scales on a high-quality data set in a target direction of three directions, interpolating to original resolution through a bicubic interpolation method or a cubic spline method, and forming low-resolution (I) by using the data as simulated low-resolution data and the original data LR ) And high resolution (I HR ) Data pairs (data-pairs), for focus position information, converting real coordinate information into Voxel coordinate information matched with a three-dimensional image through Voxel coordinates= (World coordinates-Origin)/Spacing; the coordinates are specifically classified into two-dimensional coordinates (y, x) or three-dimensional coordinates (z, y, x) according to the dimension of the specific medical image, and the Voxel coordinates are the medical scienceThe corresponding coordinates in the therapy image matrix, world coordinates as the focus, origin as the coordinate origin of the shooting instrument, and Spacing meaning the spatial information in the actual physical space contained in a voxel in the medical image;
step 2, introduction and processing of focus position information
Step (2-1), cutting data into blocks for training before training, adopting an on-line dicing strategy guided by position information, generating random numbers according to manually set yin-yang data proportion when training data are acquired each time, and assuming that the yin-yang ratio is 2:1, generating random numbers 0,1,2 firstly, if 0 is generated, randomly acquiring coordinate information from positive area coordinate information stored in a memory in step (1-2), adding disturbance of 20 voxels in each dimension of a space, taking the disturbed coordinate as a central coordinate, and performing low resolution (I) in step (1-2) LR ) And high resolution (I HR ) Intercepting a larger data block in a data pair (data-pair); by performing the following steps on the training data block: cutting off redundant parts of edges to 42 x 42 size for network training after horizontal overturning, vertical overturning, front-back overturning, translation, scaling and multi-angle rotation random augmentation operation; if the random number is 1 or 2, sampling at random effective positions in the image, and then performing the same block cutting and amplification operation in a positive area for training;
step (2-2), sending the data blocks which meet the yin-yang ratio and are obtained by sampling in the step (1-2) into a subsequent network for training to obtain a reconstructed high-resolution image (I) after super resolution SR ) The obtained I SR And I as gold standard HR Reconstruction losses by MSE loss calculation
And optimizing network parameters by a back propagation algorithm (back propagation), wherein i, j and k are coordinate serial numbers in three directions respectively.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910224792.3A CN110020987B (en) | 2019-03-24 | 2019-03-24 | Medical image super-resolution reconstruction method based on deep learning |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910224792.3A CN110020987B (en) | 2019-03-24 | 2019-03-24 | Medical image super-resolution reconstruction method based on deep learning |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110020987A CN110020987A (en) | 2019-07-16 |
| CN110020987B true CN110020987B (en) | 2023-06-30 |
Family
ID=67189877
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910224792.3A Active CN110020987B (en) | 2019-03-24 | 2019-03-24 | Medical image super-resolution reconstruction method based on deep learning |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110020987B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110610459A (en) * | 2019-09-17 | 2019-12-24 | 中国气象局公共气象服务中心(国家预警信息发布中心) | Image processing method and device |
| CN111192255B (en) * | 2019-12-30 | 2024-04-26 | 上海联影智能医疗科技有限公司 | Index detection method, computer device, and storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105654425A (en) * | 2015-12-07 | 2016-06-08 | 天津大学 | Single-image super-resolution reconstruction method applied to medical X-ray image |
| CN108615236A (en) * | 2018-05-08 | 2018-10-02 | 上海商汤智能科技有限公司 | A kind of image processing method and electronic equipment |
| CN109360152A (en) * | 2018-10-15 | 2019-02-19 | 天津大学 | 3 d medical images super resolution ratio reconstruction method based on dense convolutional neural networks |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2018053340A1 (en) * | 2016-09-15 | 2018-03-22 | Twitter, Inc. | Super resolution using a generative adversarial network |
-
2019
- 2019-03-24 CN CN201910224792.3A patent/CN110020987B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105654425A (en) * | 2015-12-07 | 2016-06-08 | 天津大学 | Single-image super-resolution reconstruction method applied to medical X-ray image |
| CN108615236A (en) * | 2018-05-08 | 2018-10-02 | 上海商汤智能科技有限公司 | A kind of image processing method and electronic equipment |
| CN109360152A (en) * | 2018-10-15 | 2019-02-19 | 天津大学 | 3 d medical images super resolution ratio reconstruction method based on dense convolutional neural networks |
Non-Patent Citations (1)
| Title |
|---|
| 刘蓬博.基于深度学习的医学图像增强算法的研究.《中国优秀硕士学位论文全文数据库》.2020,全文. * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110020987A (en) | 2019-07-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Chen et al. | Super-resolution of real-world rock microcomputed tomography images using cycle-consistent generative adversarial networks | |
| CN111105352B (en) | Super-resolution image reconstruction method, system, computer equipment and storage medium | |
| Liu et al. | Trident dehazing network | |
| Zhang et al. | Light-field depth estimation via epipolar plane image analysis and locally linear embedding | |
| Ma et al. | Image retargeting quality assessment: A study of subjective scores and objective metrics | |
| US20120313937A1 (en) | Coupled reconstruction of hair and skin | |
| SG188646A1 (en) | Multi-view ray tracing using edge detection and shader reuse | |
| CN110020987B (en) | Medical image super-resolution reconstruction method based on deep learning | |
| TWI331731B (en) | Method of utilizing multi-view images to solve occlusion problem for photorealistic model reconstruction | |
| CN112802046B (en) | Image generation system for generating pseudo CT from multi-sequence MR based on deep learning | |
| CN114972332B (en) | Bamboo laminated wood crack detection method based on image super-resolution reconstruction network | |
| Xu et al. | Quality assessment of stereoscopic 360-degree images from multi-viewports | |
| TW201123076A (en) | Three-dimensional display method of medical images | |
| Luo et al. | Bi-GANs-ST for perceptual image super-resolution | |
| Dai et al. | High-quality surface reconstruction using gaussian surfels | |
| Shen et al. | Depth map enhancement method based on joint bilateral filter | |
| Zhuang et al. | Spdet: Edge-aware self-supervised panoramic depth estimation transformer with spherical geometry | |
| CN111145280B (en) | OCT image speckle suppression method | |
| Fu et al. | ADGAN: An asymmetric despeckling generative adversarial network for unpaired OCT image speckle noise reduction | |
| Sahasrabudhe et al. | Structured spatial domain image and data comparison metrics | |
| CN110322548B (en) | Three-dimensional grid model generation method based on geometric image parameterization | |
| JP4082718B2 (en) | Image recognition method, image display method, and image recognition apparatus | |
| Li et al. | A Bayesian approach to uncertainty-based depth map super resolution | |
| CN115311135A (en) | 3 DCNN-based isotropic MRI resolution reconstruction method | |
| Zhang et al. | Light field occlusion removal network via foreground location and background recovery |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |