CN110490813A - Characteristic pattern Enhancement Method, device, equipment and the medium of convolutional neural networks - Google Patents
Characteristic pattern Enhancement Method, device, equipment and the medium of convolutional neural networks Download PDFInfo
- Publication number
- CN110490813A CN110490813A CN201910605387.6A CN201910605387A CN110490813A CN 110490813 A CN110490813 A CN 110490813A CN 201910605387 A CN201910605387 A CN 201910605387A CN 110490813 A CN110490813 A CN 110490813A
- Authority
- CN
- China
- Prior art keywords
- subcharacter
- channel
- characteristic pattern
- enhancement factor
- attention
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000013527 convolutional neural network Methods 0.000 title claims abstract description 33
- 238000000034 method Methods 0.000 title claims abstract description 31
- 239000013598 vector Substances 0.000 claims abstract description 36
- 230000002708 enhancing effect Effects 0.000 claims abstract description 29
- 239000003623 enhancer Substances 0.000 claims abstract description 25
- 238000012545 processing Methods 0.000 claims abstract description 12
- 230000004913 activation Effects 0.000 claims description 26
- 230000009467 reduction Effects 0.000 claims description 15
- 238000004590 computer program Methods 0.000 claims description 13
- 238000010606 normalization Methods 0.000 claims description 13
- 230000006870 function Effects 0.000 claims description 8
- 230000008859 change Effects 0.000 claims description 2
- 238000001514 detection method Methods 0.000 abstract description 6
- 230000011218 segmentation Effects 0.000 abstract description 6
- 238000004364 calculation method Methods 0.000 description 8
- 238000010586 diagram Methods 0.000 description 8
- 230000007246 mechanism Effects 0.000 description 8
- 230000004044 response Effects 0.000 description 5
- 238000004422 calculation algorithm Methods 0.000 description 4
- 238000013528 artificial neural network Methods 0.000 description 2
- 230000006835 compression Effects 0.000 description 2
- 238000007906 compression Methods 0.000 description 2
- 230000007423 decrease Effects 0.000 description 2
- 230000001965 increasing effect Effects 0.000 description 2
- 230000005291 magnetic effect Effects 0.000 description 2
- 239000011159 matrix material Substances 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 241000208340 Araliaceae Species 0.000 description 1
- 235000005035 Panax pseudoginseng ssp. pseudoginseng Nutrition 0.000 description 1
- 235000003140 Panax quinquefolius Nutrition 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 230000000712 assembly Effects 0.000 description 1
- 238000000429 assembly Methods 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000005611 electricity Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 235000008434 ginseng Nutrition 0.000 description 1
- 238000009434 installation Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
- 230000017105 transposition Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10004—Still image; Photographic image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Molecular Biology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Image Analysis (AREA)
Abstract
This application discloses a kind of characteristic pattern Enhancement Methods of convolutional neural networks, device, equipment and medium, convolution operation is carried out to the original image of input, obtain corresponding multilayer feature figure, for certain layer of characteristic pattern, it is grouped according to channel dimension, obtain multiple subcharacter figures, for each subcharacter figure, global average pondization and global maximum pond parallel processing are carried out using embedded airspace grouping enhancing SGE module, obtain corresponding two channel dimension vectors, according to corresponding two channel dimension vectors, obtain the attention enhancement factor in each channel in corresponding subcharacter figure, according to the attention enhancement factor and corresponding subcharacter figure, obtain corresponding enhancer characteristic pattern, according to whole enhancer characteristic patterns, obtain the corresponding Enhanced feature figure of certain layer of characteristic pattern, so as to preferably expressor spy The semantic information for levying figure interchannel significance level, improves the mission performances such as image classification, segmentation, the detection of convolutional neural networks.
Description
Technical field
This application involves technical field of computer vision, in particular to the characteristic pattern enhancing side of a kind of convolutional neural networks
Method, device, equipment and medium.
Background technique
With the rise of deep learning, as one of depth learning technology, CNN (Convolutional Neural
Network, convolutional neural networks) more and more to be developed and applied in computer vision field, researcher proposes
Many convolution operations, such as transposition convolution, expansion convolution, grouping convolution, depth separate convolution, point-by-point convolution, deformable convolution
Deng.Wherein, grouping convolution has great advantage in terms of reducing calculation amount and parameter amount, preventing, and counts with early stage
Calculation machine visual field artificial design features are using the grouping unity of thinking, such as HOG (Histogram of Oriented
Gradient, histograms of oriented gradients), SIFT (Scale-invariant feature transform, scale invariant feature
Transformation), LBP (Local Binary Pattern, local binary patterns) etc., therefore many classic network AlexNet,
ResNeXt, MobileNet, ShuffleNet, CapsuleNet etc. have used grouping thought, are along channel dimension to spy
Sign figure grouping, then carries out convolution or Regularization to every group of subcharacter figure, can preferably express the semanteme of specific region
Characteristic information achieves excellent performance boost in computer vision field.
Currently, major part CNN network structure is being counted by introducing attention mechanism come the feature representation ability of lift scheme
The fields such as calculation machine vision become very popular, and a variety of neural network structures all introduce channel dimension or airspace dimension attention machine
It makes to enhance useful channel information, the useless channel information of compression, Analysis On Multi-scale Features can also be merged or global context information is come
The enhancing ability to characteristic pattern specific region is further increased, allows neural network that there is the mechanism of more interpretation.As it can be seen that will
Attention mechanism is added in subcharacter figure after grouping, can further enhance the semantic feature information learning and table to specific region
Danone power, and further compression noise and interference.
However, existing SGE (Spatial Group-wise Enhance, the airspace point being embedded into CNN network structure
Group enhancing) the semantic feature information representation and insufficient that extracts of module, so as to cause the image classification of CNN network, segmentation, detection
Etc. mission performances decline.
Summary of the invention
A kind of characteristic pattern Enhancement Method, device, equipment and the medium for being designed to provide convolutional neural networks of the application,
With mission performances such as the image classification, segmentation, the detections that improve convolutional neural networks.
In a first aspect, the embodiment of the present application provides a kind of characteristic pattern Enhancement Method of convolutional neural networks, comprising:
Convolution operation is carried out to the original image of input, obtains corresponding multilayer feature figure；
For certain layer of characteristic pattern, it is grouped according to channel dimension, obtains multiple subcharacter figures；
For each subcharacter figure, global average pond and the overall situation are carried out using embedded airspace grouping enhancing SGE module
Maximum pond parallel processing obtains corresponding two channel dimension vectors；
According to corresponding two channel dimension vectors, the attention enhancing in each channel in corresponding subcharacter figure is obtained
The factor；
According to the attention enhancement factor and corresponding subcharacter figure, corresponding enhancer characteristic pattern is obtained；
According to whole enhancer characteristic patterns, the corresponding Enhanced feature figure of certain layer of characteristic pattern is obtained.
In one possible implementation, described according to described right in the above method provided by the embodiments of the present application
The two channel dimension vectors answered obtain the attention enhancement factor in each channel in corresponding subcharacter figure, comprising:
Using 1 × 1 convolution to corresponding two channel dimensions vector dimensionality reduction；
It is added after being activated two channel dimension vectors after dimensionality reduction using ReLU activation primitive, it is special to obtain corresponding son
Levy the attention enhancement factor in each channel in figure.
In one possible implementation, described according to the note in the above method provided by the embodiments of the present application
Power enhancement factor of anticipating and corresponding subcharacter figure, obtain corresponding enhancer characteristic pattern, comprising:
The attention enhancement factor is risen into dimension to the port number for corresponding to subcharacter figure using 1 × 1 convolution；
Utilize attention enhancement factor described in SoftMax function normalization；
Attention enhancement factor after normalization is multiplied with corresponding subcharacter figure, the first subcharacter figure enhanced；
Regularization is carried out to the first subcharacter figure, obtains the second subcharacter figure；
The second subcharacter figure, the third subcharacter figure enhanced are activated using Sigmoid activation primitive；
Using the third subcharacter figure as enhancer characteristic pattern.
In one possible implementation, described to utilize Sigmoid in the above method provided by the embodiments of the present application
Activation primitive activates the second subcharacter figure, the third subcharacter figure enhanced, comprising:
The important coefficient in the second subcharacter figure channel is obtained using Sigmoid activation primitive；
It is logical to re-scale the second subcharacter figure using the sized second subcharacter figure of the important coefficient
Spatial feature importance on road, the third subcharacter figure enhanced.
Second aspect, the embodiment of the present application provide a kind of characteristic pattern enhancement device of convolutional neural networks, comprising:
Convolution module obtains corresponding multilayer feature figure for carrying out convolution operation to the original image of input；
It is grouped according to channel dimension for being directed to certain layer of characteristic pattern, obtains multiple subcharacters by grouping module
Figure；
Enhance module, for being directed to each subcharacter figure, is carried out using embedded airspace grouping enhancing SGE module global flat
Equal pondization and global maximum pond parallel processing, obtain corresponding two channel dimension vectors；It is logical according to described corresponding two
Road dimension vector obtains the attention enhancement factor in each channel in corresponding subcharacter figure；According to the attention enhancement factor
With corresponding subcharacter figure, corresponding enhancer characteristic pattern is obtained；
Output module, for according to whole enhancer characteristic patterns, obtaining the corresponding Enhanced feature figure of certain layer of characteristic pattern.
In one possible implementation, in above-mentioned apparatus provided by the embodiments of the present application, the enhancing module, tool
Body is used for:
Using 1 × 1 convolution to corresponding two channel dimensions vector dimensionality reduction；
It is added after being activated two channel dimension vectors after dimensionality reduction using ReLU activation primitive, it is special to obtain corresponding son
Levy the attention enhancement factor in each channel in figure.
In one possible implementation, in above-mentioned apparatus provided by the embodiments of the present application, the enhancing module, tool
Body is used for:
The attention enhancement factor is risen into dimension to the port number for corresponding to subcharacter figure using 1 × 1 convolution；
Utilize attention enhancement factor described in SoftMax function normalization；
Attention enhancement factor after normalization is multiplied with corresponding subcharacter figure, the first subcharacter figure enhanced；
Regularization is carried out to the first subcharacter figure, obtains the second subcharacter figure；
The second subcharacter figure, the third subcharacter figure enhanced are activated using Sigmoid activation primitive；
Using the third subcharacter figure as enhancer characteristic pattern.
In one possible implementation, in above-mentioned apparatus provided by the embodiments of the present application, the enhancing module, tool
Body is used for:
The important coefficient in the second subcharacter figure channel is obtained using Sigmoid activation primitive；
It is logical to re-scale the second subcharacter figure using the sized second subcharacter figure of the important coefficient
Spatial feature importance on road, the third subcharacter figure enhanced.
The third aspect, the embodiment of the present application provide a kind of electronic equipment, comprising: memory and processor；
The memory, for storing computer program；
Wherein, the processor executes the computer program in the memory, to realize above-mentioned first aspect and
Method described in each embodiment of one side.
Fourth aspect, the embodiment of the present application provide a kind of computer readable storage medium, the computer-readable storage
Computer program is stored in medium, for realizing above-mentioned first aspect and when the computer program is executed by processor
Method described in each embodiment of one side.
Compared with prior art, characteristic pattern Enhancement Method, device, equipment and Jie of convolutional neural networks provided by the present application
Matter carries out convolution operation to the original image of input, obtains corresponding multilayer feature figure, will wherein certain layer of characteristic pattern according to channel
Dimension is grouped, and obtains multiple subcharacter figures, for each subcharacter figure, utilizes embedded airspace grouping enhancing SGE module
Global average pondization and global maximum pond parallel processing are carried out, corresponding two channel dimension vectors are obtained, according to described right
The two channel dimension vectors answered obtain the attention enhancement factor in each channel in corresponding subcharacter figure, according to the attention
Power enhancement factor and corresponding subcharacter figure, obtain corresponding enhancer characteristic pattern, according to whole enhancer characteristic patterns, obtain certain layer
The corresponding Enhanced feature figure of characteristic pattern, thus by extracting channel dimension with the average pondization of the overall situation and global maximum pondization operation
Attention enhancement factor, can preferably expressor characteristic pattern interchannel significance level semantic information, redesign simultaneously
Airspace grouping enhancing modular structure, so that the calculating of the attention enhancement factor of subcharacter figure channel dimension is more efficient, into one
Step improves the mission performances such as image classification, segmentation, the detection of convolutional neural networks.
Detailed description of the invention
Fig. 1 is the flow diagram of the characteristic pattern Enhancement Method for the convolutional neural networks that the embodiment of the present application one provides；
Fig. 2 is the algorithm flow schematic diagram provided by the embodiments of the present application enhanced sub- characteristic pattern；
Fig. 3 is the structural schematic diagram of the characteristic pattern enhancement device for the convolutional neural networks that the embodiment of the present application two provides；
Fig. 4 is the structural schematic diagram for the electronic equipment that the embodiment of the present application three provides.
Specific embodiment
With reference to the accompanying drawing, the specific embodiment of the application is described in detail, it is to be understood that the guarantor of the application
Shield range is not limited by the specific implementation.
Unless otherwise explicitly stated, otherwise in entire disclosure and claims, term " includes " or its change
Changing such as "comprising" or " including " etc. will be understood to comprise stated element or component, and not exclude other members
Part or other component parts.
Fig. 1 is the flow diagram of the characteristic pattern Enhancement Method for the convolutional neural networks that the embodiment of the present application one provides.It is real
In the application of border, the executing subject of the present embodiment can be the characteristic pattern enhancement device of convolutional neural networks, the convolutional neural networks
Characteristic pattern enhancement device can realize that such as software code related can also execute generation by being written with by virtual bench
The entity apparatus of code is realized, such as USB flash disk, then alternatively, can also be realized by being integrated with the related entity apparatus for executing code, example
Such as, chip, computer etc..
As shown in Figure 1, this approach includes the following steps S101~S106:
S101, convolution operation is carried out to the original image of input, obtains corresponding multilayer feature figure.
In the present embodiment, the convolutional neural networks constructed in advance carry out multilayer convolution operation to the original image of input, can
To obtain corresponding multilayer feature figure.It is appreciated that wherein one layer of characteristic pattern includes a certain number of channels.
S102, it is directed to certain layer of characteristic pattern, it is grouped according to channel dimension, obtains multiple subcharacter figures.
In the present embodiment, since in convolutional neural networks learning process, grouping convolution can gradually capture special semanteme
Response, so that the response of concern position is bigger, and other positions are not then activated or do not respond to, while utilizing grouping
Convolution can also reduce calculation amount and parameter amount, therefore, can first be grouped to features described above figure, preferably to enhance pair
Channels a certain number of in characteristic pattern are specifically grouped, obtain and be grouped by the semantic feature information learning of specific region
The identical multiple subcharacter figures of quantity.
Assuming that the output characteristic pattern after multilayer convolution isWherein, C is characterized the port number of figure, H and W difference
The length and width for indicating characteristic pattern, are divided into G group along channel dimension for characteristic pattern first, then the obtained each airspace of subcharacter figure
The vector of position is denoted as X={ x1,...,H×W, wherein each element is
S103, it is directed to each subcharacter figure, carries out global average Chi Huahe using embedded airspace grouping enhancing SGE module
Global maximum pond parallel processing, obtains corresponding two channel dimension vectors.
S104, according to corresponding two channel dimension vectors, obtain the attention in each channel in corresponding subcharacter figure
Power enhancement factor.
In the present embodiment, step S104 can be implemented are as follows: using 1 × 1 convolution to corresponding two channel dimensions
Vector dimensionality reduction；It is added after being activated two channel dimension vectors after dimensionality reduction using ReLU activation primitive, obtains corresponding son
The attention enhancement factor in each channel in characteristic pattern.
In practical application, some CNN network structures by introducing attention mechanism come the feature representation ability of lift scheme,
This attention mechanism not only tells which important feature network model pays close attention to, but also the expression energy of specific region can be enhanced
Power.But the mode of channel dimension and airspace dimension cascade attention enhancing module also increases the calculation amount and ginseng of network model
Quantity.
Fig. 2 is the algorithm flow schematic diagram provided by the embodiments of the present application enhanced sub- characteristic pattern.As shown in Fig. 2,
Figure left side X column, which are represented, is grouped into 3 sub- characteristic patterns for characteristic pattern, carries out to each subcharacter figure by the algorithm flow of figure lower section
Enhancing obtains corresponding enhancer characteristic pattern in the V column of figure right side.
It describes in detail below to above-mentioned algorithm flow.In view of calculation amount and model size problem, the application only makes
With channel dimension attention mechanism, response is generated to each airspace position of sub- characteristic pattern using global average pond, in conjunction with
Only there is gradient feedback in global maximum pond to the maximum position of characteristic response in backpropagation.Airspace grouping is redesigned simultaneously
Enhance modular structure, by using global statistics characteristic information to sub- characteristic pattern parallel processing, respectively with global average Chi Huahe
Global maximum pondization operates to extract the attention enhancement factor of channel dimension, is used for expressor characteristic pattern interchannel significance level
Semantic information.
The extraction process of global statistics characteristic information is as follows:
Wherein, a and b respectively indicates the channel dimensional vector after global average pondization and global maximum pond parallel processing, max
() indicates to take all positions of channel dimensional vector the maximum operation of response, the maximum activation information in available each channel.This
Mean that the dimension of the channel grouping subcharacter figure is compressed to from C × H × W after sample operationSimultaneously in the dimensional vector of channel
Each value each grouping subcharacter figure interchannel importance is indicated with global statistics characteristic information.
It is calculated it is appreciated that global maximum pond information is effectively dissolved into attention enhancement factor in the present embodiment
In, enhance the semantic feature information representation ability of airspace grouping enhancing module.
In order to model the interdependency of subcharacter figure interchannel, the application uses volume 1 × 1 with ReLU activation primitive
Product carries out dimensionality reduction to channel dimensional vector, decreases calculation amount, table while increasing the non-linear interaction capabilities of interchannel information
Up to formula are as follows:
E=ReLU (W1a) (3)
F=ReLU (W2b) (4)
Wherein, W1And W2The weight matrix for respectively indicating the operation of 1 × 1 convolution dimensionality reduction, is denoted asWithAnd dimension of the channel meets relationship:
Two channel dimensional vectors are added, it can the corresponding attention enhancement factor in each channel of subcharacter figure is obtained,
Expression formula is as follows:
S105, according to the attention enhancement factor and corresponding subcharacter figure, obtain corresponding enhancer characteristic pattern.
In the present embodiment, step S105 can be implemented are as follows: utilize 1 × 1 convolution by the attention enhancement factor liter
Tie up the port number of corresponding subcharacter figure；Utilize attention enhancement factor described in SoftMax function normalization；After normalization
Attention enhancement factor is multiplied with corresponding subcharacter figure, the first subcharacter figure enhanced；To the first subcharacter figure into
Row regularization obtains the second subcharacter figure；The second subcharacter figure is activated using Sigmoid activation primitive, is enhanced
Third subcharacter figure；Using the third subcharacter figure as enhancer characteristic pattern.
It is preferably fitted interchannel complex relationship in order to increase more multi-non-linear, rises dimension behaviour using 1 × 1 convolution first
Make, the dimension of attention enhancement factor is allowed to reach the port number of subcharacter figure, so as to when subsequent weight average treatment and subcharacter
Figure port number matches.Rise the expression formula of the attention enhancement factor after tieing up and normalizing are as follows:
Wherein, W3It indicates that 1 × 1 convolution rises the weight matrix of dimension operation, is denoted as
For each of subcharacter figure airspace position, with the attention enhancement factor u after above-mentioned normalization to xiAdd
Weight average, the available subcharacter figure enhanced with attention mechanism channel, expression formula are as follows:
yi=uxi (7)
Further, the second subcharacter figure, the third subcharacter enhanced are activated using Sigmoid activation primitive
Figure, it is specific to can be achieved are as follows: the important coefficient in the second subcharacter figure channel is obtained using Sigmoid activation primitive；It utilizes
The sized second subcharacter figure of important coefficient, to re-scale the spy of the airspace on the second subcharacter figure channel
Levy importance, the third subcharacter figure enhanced.
In the present embodiment, to eliminate interference of the amplitude difference to result between sample, the above-mentioned note of Regularization will be used
The meaning enhanced subcharacter figure in power mechanism channel.For each airspace position of subcharacter figure, the expression formula of regularization operation
It is as follows:
Wherein, ziIt indicates to tie up channel the subcharacter figure after regularization, μcIndicate the mean value of subcharacter figure,Indicate that son is special
Levy the variance of figure.
Then the important coefficient of the subcharacter Tu Tongdaowei after regularization is obtained using Sigmoid activation primitive, is used
This important coefficient carrys out the subcharacter figure after sized above-mentioned Regularization, re-scales the sky on subcharacter Tu Tongdaowei
Characteristic of field importance.The expression formula for describing this treatment process is as follows:
vi=xi·Sigmoid(αzi+β) (11)
Wherein, α and β respectively indicates the parameter to the scale operation of subcharacter figure and translation after regularization, for every
For one grouping, the two parameters have identical fixed value.
It is appreciated that having redesigned grouping enhancing modular structure in airspace in the present embodiment, uses 1 × 1 convolution and merge
Dimension operation is risen after preceding dimensionality reduction fusion, not only reduces calculation amount, and enhance the information exchange ability of subcharacter Tu Tongdaowei, and
And the probability of channel dimension spatial feature significance level is obtained to indicate attention enhancement factor using SoftMax activation primitive.
S106, according to whole enhancer characteristic patterns, obtain the corresponding Enhanced feature figure of certain layer of characteristic layer.
The characteristic pattern Enhancement Method of convolutional neural networks provided in this embodiment, by with the average pondization of the overall situation and the overall situation most
The attention enhancement factor of channel dimension is extracted in great Chiization operation, can preferably expressor characteristic pattern interchannel significance level
Semantic information, while redesign airspace grouping enhancing modular structure so that the attention of subcharacter figure channel dimension enhances
The calculating of the factor is more efficient, further improves the mission performances such as image classification, segmentation, the detection of convolutional neural networks.
Following is the application Installation practice, can be used for executing the application embodiment of the method.It is real for the application device
Undisclosed details in example is applied, the application embodiment of the method is please referred to.
Fig. 3 is the structural schematic diagram of the characteristic pattern enhancement device for the convolutional neural networks that the embodiment of the present application two provides, such as
Shown in Fig. 3, the apparatus may include:
Convolution module 310 obtains corresponding multilayer feature figure for carrying out convolution operation to the original image of input；
It is grouped according to channel dimension for being directed to certain layer of characteristic pattern, it is special to obtain multiple sons by grouping module 320
Sign figure；
Enhance module 330, for being directed to each subcharacter figure, is carried out using embedded airspace grouping enhancing SGE module complete
The average pondization of office and global maximum pond parallel processing, obtain corresponding two channel dimension vectors；According to described corresponding two
A channel dimension vector obtains the attention enhancement factor in each channel in corresponding subcharacter figure；Enhanced according to the attention
The factor and corresponding subcharacter figure, obtain corresponding enhancer characteristic pattern；
Output module 340, for according to whole enhancer characteristic patterns, obtaining the corresponding Enhanced feature of certain layer of characteristic pattern
Figure.
The characteristic pattern enhancement device of convolutional neural networks provided in this embodiment, by with the average pondization of the overall situation and the overall situation most
The attention enhancement factor of channel dimension is extracted in great Chiization operation, can preferably expressor characteristic pattern interchannel significance level
Semantic information, while redesign airspace grouping enhancing modular structure so that the attention of subcharacter figure channel dimension enhances
The calculating of the factor is more efficient, further improves the mission performances such as image classification, segmentation, the detection of convolutional neural networks.
In some embodiments, the enhancing module 330, is specifically used for:
Using 1 × 1 convolution to corresponding two channel dimensions vector dimensionality reduction；
It is added after being activated two channel dimension vectors after dimensionality reduction using ReLU activation primitive, it is special to obtain corresponding son
Levy the attention enhancement factor in each channel in figure.
In some embodiments, the enhancing module 330, is specifically used for:
The attention enhancement factor is risen into dimension to the port number for corresponding to subcharacter figure using 1 × 1 convolution；
Utilize attention enhancement factor described in SoftMax function normalization；
Attention enhancement factor after normalization is multiplied with corresponding subcharacter figure, the first subcharacter figure enhanced；
Regularization is carried out to the first subcharacter figure, obtains the second subcharacter figure；
The second subcharacter figure, the third subcharacter figure enhanced are activated using Sigmoid activation primitive；
Using the third subcharacter figure as enhancer characteristic pattern.
In some embodiments, the enhancing module 330, is specifically used for:
The important coefficient in the second subcharacter figure channel is obtained using Sigmoid activation primitive；
It is logical to re-scale the second subcharacter figure using the sized second subcharacter figure of the important coefficient
Spatial feature importance on road, the third subcharacter figure enhanced.
Fig. 4 is the structural schematic diagram for the electronic equipment that the embodiment of the present application three provides, as shown in figure 4, the equipment includes: to deposit
Reservoir 401 and processor 402；
Memory 401, for storing computer program；
Wherein, processor 402 executes the computer program in memory 401, to realize each method embodiment as described above
Provided method.
In embodiment, it is carried out with characteristic pattern enhancement device of the electronic equipment to convolutional neural networks provided by the present application
Example.Processor can be central processing unit (CPU) or with other of data-handling capacity and/or instruction execution capability
The processing unit of form, and can control the other assemblies in electronic equipment to execute desired function.
Memory may include one or more computer program products, and computer program product may include various forms
Computer readable storage medium, such as volatile memory and/or nonvolatile memory.Volatile memory for example can be with
Including random access memory (RAM) and/or cache memory (cache) etc..Nonvolatile memory for example can wrap
Include read-only memory (ROM), hard disk, flash memory etc..It can store one or more computers on computer readable storage medium
Program instruction, processor can run program instruction, method in each embodiment to realize the application above and/or
Other desired functions of person.Such as input signal, signal component, noise point can also be stored in a computer-readable storage medium
The various contents such as amount.
The embodiment of the present application four provides a kind of computer readable storage medium, stores in the computer readable storage medium
There is computer program, for realizing side provided by each method embodiment as described above when which is executed by processor
Method.
In practical application, the computer program in the present embodiment can be with any group of one or more programming languages
It closes to write the program code for executing the embodiment of the present application operation, programming language includes object-oriented programming
Language, Java, C++ etc. further include that conventional procedural programming language, such as " C " language or similar program are set
Count language.Program code can be executed fully on the user computing device, partly be executed on a user device, as one
Independent software package executes, part executes on a remote computing or completely long-range on the user computing device for part
It calculates and is executed on equipment or server.
In practical application, computer readable storage medium can be using any combination of one or more readable mediums.It can
Reading medium can be readable signal medium or readable storage medium storing program for executing.Readable storage medium storing program for executing for example can include but is not limited to electricity,
Magnetic, optical, electromagnetic, infrared ray or semiconductor system, device or device, or any above combination.Readable storage medium storing program for executing
More specific example (non exhaustive list) includes: electrical connection with one or more conducting wires, portable disc, hard disk, random
It accesses memory (RAM), read-only memory (ROM), erasable programmable read only memory (EPROM or flash memory), optical fiber, portable
Formula compact disk read-only memory (CD-ROM), light storage device, magnetic memory device or above-mentioned any appropriate combination.
The description of the aforementioned specific exemplary embodiment to the application is in order to illustrate and illustration purpose.These descriptions
It is not wishing to for the application to be limited to disclosed precise forms, and it will be apparent that according to the above instruction, can much be changed
And variation.The purpose of selecting and describing the exemplary embodiment is that explaining the specific principle of the application and its actually answering
With so that those skilled in the art can be realized and utilize the application a variety of different exemplary implementation schemes and
Various chooses and changes.Scope of the present application is intended to be limited by claims and its equivalents.
Claims (10)
1. a kind of characteristic pattern Enhancement Method of convolutional neural networks characterized by comprising
Convolution operation is carried out to the original image of input, obtains corresponding multilayer feature figure；
For certain layer of characteristic pattern, it is grouped according to channel dimension, obtains multiple subcharacter figures；
For each subcharacter figure, global average pondization is carried out using embedded airspace grouping enhancing SGE module and the overall situation is maximum
Pond parallel processing obtains corresponding two channel dimension vectors；
According to corresponding two channel dimension vectors, obtain the attention enhancing in each channel in corresponding subcharacter figure because
Son；
According to the attention enhancement factor and corresponding subcharacter figure, corresponding enhancer characteristic pattern is obtained；
According to whole enhancer characteristic patterns, the corresponding Enhanced feature figure of certain layer of characteristic pattern is obtained.
2. the method according to claim 1, wherein described according to corresponding two channel dimension vectors,
Obtain the attention enhancement factor in each channel in corresponding subcharacter figure, comprising:
Using 1 × 1 convolution to corresponding two channel dimensions vector dimensionality reduction；
It is added after being activated two channel dimension vectors after dimensionality reduction using ReLU activation primitive, obtains corresponding subcharacter figure
In each channel attention enhancement factor.
3. according to the method described in claim 2, it is characterized in that, described special according to the attention enhancement factor and corresponding son
Sign figure, obtains corresponding enhancer characteristic pattern, comprising:
The attention enhancement factor is risen into dimension to the port number for corresponding to subcharacter figure using 1 × 1 convolution；
Utilize attention enhancement factor described in SoftMax function normalization；
Attention enhancement factor after normalization is multiplied with corresponding subcharacter figure, the first subcharacter figure enhanced；
Regularization is carried out to the first subcharacter figure, obtains the second subcharacter figure；
The second subcharacter figure, the third subcharacter figure enhanced are activated using Sigmoid activation primitive；
Using the third subcharacter figure as enhancer characteristic pattern.
4. according to the method described in claim 3, it is characterized in that, described utilize Sigmoid activation primitive activation described second
Subcharacter figure, the third subcharacter figure enhanced, comprising:
The important coefficient in the second subcharacter figure channel is obtained using Sigmoid activation primitive；
Using the sized second subcharacter figure of the important coefficient, to re-scale on the second subcharacter figure channel
Spatial feature importance, the third subcharacter figure enhanced.
5. a kind of characteristic pattern enhancement device of convolutional neural networks characterized by comprising
Convolution module obtains corresponding multilayer feature figure for carrying out convolution operation to the original image of input；
It is grouped according to channel dimension for being directed to certain layer of characteristic pattern, obtains multiple subcharacter figures by grouping module；
Enhance module, for being directed to each subcharacter figure, carries out global average pond using embedded airspace grouping enhancing SGE module
Change and global maximum pond parallel processing, obtain corresponding two channel dimension vectors；It is tieed up according to corresponding two channels
Vector is spent, the attention enhancement factor in each channel in corresponding subcharacter figure is obtained；According to the attention enhancement factor and right
Subcharacter figure is answered, corresponding enhancer characteristic pattern is obtained；
Output module, for according to whole enhancer characteristic patterns, obtaining the corresponding Enhanced feature figure of certain layer of characteristic pattern.
6. device according to claim 5, which is characterized in that the enhancing module is specifically used for:
Using 1 × 1 convolution to corresponding two channel dimensions vector dimensionality reduction；
It is added after being activated two channel dimension vectors after dimensionality reduction using ReLU activation primitive, obtains corresponding subcharacter figure
In each channel attention enhancement factor.
7. device according to claim 6, which is characterized in that the enhancing module is specifically used for:
The attention enhancement factor is risen into dimension to the port number for corresponding to subcharacter figure using 1 × 1 convolution；
Utilize attention enhancement factor described in SoftMax function normalization；
Attention enhancement factor after normalization is multiplied with corresponding subcharacter figure, the first subcharacter figure enhanced；
Regularization is carried out to the first subcharacter figure, obtains the second subcharacter figure；
The second subcharacter figure, the third subcharacter figure enhanced are activated using Sigmoid activation primitive；
Using the third subcharacter figure as enhancer characteristic pattern.
8. device according to claim 7, which is characterized in that the enhancing module is specifically used for:
The important coefficient in the second subcharacter figure channel is obtained using Sigmoid activation primitive；
Using the sized second subcharacter figure of the important coefficient, to re-scale on the second subcharacter figure channel
Spatial feature importance, the third subcharacter figure enhanced.
9. a kind of electronic equipment, comprising: memory and processor；
The memory, for storing computer program；
Wherein, the processor executes the computer program in the memory, to realize such as any one of claim 1-4 institute
The method stated.
10. a kind of computer readable storage medium, which is characterized in that be stored with computer in the computer readable storage medium
Program, for realizing method such as of any of claims 1-4 when the computer program is executed by processor.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910605387.6A CN110490813B (en) | 2019-07-05 | 2019-07-05 | Feature map enhancement method, device, equipment and medium for convolutional neural network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910605387.6A CN110490813B (en) | 2019-07-05 | 2019-07-05 | Feature map enhancement method, device, equipment and medium for convolutional neural network |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110490813A true CN110490813A (en) | 2019-11-22 |
| CN110490813B CN110490813B (en) | 2021-12-17 |
Family
ID=68546677
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910605387.6A Active CN110490813B (en) | 2019-07-05 | 2019-07-05 | Feature map enhancement method, device, equipment and medium for convolutional neural network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110490813B (en) |
Cited By (19)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111027670A (en) * | 2019-11-04 | 2020-04-17 | 重庆特斯联智慧科技股份有限公司 | Feature map processing method and device, electronic equipment and storage medium |
| CN111161195A (en) * | 2020-01-02 | 2020-05-15 | 重庆特斯联智慧科技股份有限公司 | Feature map processing method and device, storage medium and terminal |
| CN111274999A (en) * | 2020-02-17 | 2020-06-12 | 北京迈格威科技有限公司 | Data processing method, image processing method, device and electronic equipment |
| CN111325751A (en) * | 2020-03-18 | 2020-06-23 | 重庆理工大学 | CT image segmentation system based on attention convolution neural network |
| CN111444957A (en) * | 2020-03-25 | 2020-07-24 | 腾讯科技（深圳）有限公司 | Image data processing method, image data processing device, computer equipment and storage medium |
| CN111539325A (en) * | 2020-04-23 | 2020-08-14 | 四川旅游学院 | Forest fire detection method based on deep learning |
| CN111967478A (en) * | 2020-07-08 | 2020-11-20 | 特斯联科技集团有限公司 | Feature map reconstruction method and system based on weight inversion, storage medium and terminal |
| CN112001248A (en) * | 2020-07-20 | 2020-11-27 | 北京百度网讯科技有限公司 | Active interaction method and device, electronic equipment and readable storage medium |
| CN112149694A (en) * | 2020-08-28 | 2020-12-29 | 特斯联科技集团有限公司 | Image processing method, system, storage medium and terminal based on convolutional neural network pooling module |
| CN112183645A (en) * | 2020-09-30 | 2021-01-05 | 深圳龙岗智能视听研究院 | Image aesthetic quality evaluation method based on context-aware attention mechanism |
| CN112465828A (en) * | 2020-12-15 | 2021-03-09 | 首都师范大学 | Image semantic segmentation method and device, electronic equipment and storage medium |
| CN112668656A (en) * | 2020-12-30 | 2021-04-16 | 深圳市优必选科技股份有限公司 | Image classification method and device, computer equipment and storage medium |
| CN112767406A (en) * | 2021-02-02 | 2021-05-07 | 苏州大学 | Deep convolution neural network suitable for corneal ulcer segmentation of fluorescence staining slit lamp image |
| CN113011465A (en) * | 2021-02-25 | 2021-06-22 | 浙江净禾智慧科技有限公司 | Household garbage throwing intelligent supervision method based on grouping multi-stage fusion |
| CN113052173A (en) * | 2021-03-25 | 2021-06-29 | 北京百度网讯科技有限公司 | Sample data feature enhancement method and device |
| CN113052760A (en) * | 2021-01-29 | 2021-06-29 | 成都商汤科技有限公司 | Pooling method, chip, equipment and storage medium |
| CN113361529A (en) * | 2020-03-03 | 2021-09-07 | 北京四维图新科技股份有限公司 | Image semantic segmentation method and device, electronic equipment and storage medium |
| CN113486898A (en) * | 2021-07-08 | 2021-10-08 | 西安电子科技大学 | Radar signal RD image interference identification method and system based on improved ShuffleNet |
| CN113920099A (en) * | 2021-10-15 | 2022-01-11 | 深圳大学 | Polyp segmentation method and device, computer equipment and storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109299268A (en) * | 2018-10-24 | 2019-02-01 | 河南理工大学 | A kind of text emotion analysis method based on dual channel model |
| CN109376804A (en) * | 2018-12-19 | 2019-02-22 | 中国地质大学（武汉） | Based on attention mechanism and convolutional neural networks Classification of hyperspectral remote sensing image method |
| CN109685199A (en) * | 2017-10-18 | 2019-04-26 | 斯特拉德视觉公司 | The method and apparatus of table of the creation comprising the information about pond type and the test method and test device for using it |
-
2019
- 2019-07-05 CN CN201910605387.6A patent/CN110490813B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109685199A (en) * | 2017-10-18 | 2019-04-26 | 斯特拉德视觉公司 | The method and apparatus of table of the creation comprising the information about pond type and the test method and test device for using it |
| CN109299268A (en) * | 2018-10-24 | 2019-02-01 | 河南理工大学 | A kind of text emotion analysis method based on dual channel model |
| CN109376804A (en) * | 2018-12-19 | 2019-02-22 | 中国地质大学（武汉） | Based on attention mechanism and convolutional neural networks Classification of hyperspectral remote sensing image method |
Cited By (33)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111027670B (en) * | 2019-11-04 | 2022-07-22 | 重庆特斯联智慧科技股份有限公司 | Feature map processing method and device, electronic equipment and storage medium |
| CN111027670A (en) * | 2019-11-04 | 2020-04-17 | 重庆特斯联智慧科技股份有限公司 | Feature map processing method and device, electronic equipment and storage medium |
| CN111161195A (en) * | 2020-01-02 | 2020-05-15 | 重庆特斯联智慧科技股份有限公司 | Feature map processing method and device, storage medium and terminal |
| CN111161195B (en) * | 2020-01-02 | 2023-10-13 | 重庆特斯联智慧科技股份有限公司 | Feature map processing method and device, storage medium and terminal |
| CN111274999A (en) * | 2020-02-17 | 2020-06-12 | 北京迈格威科技有限公司 | Data processing method, image processing method, device and electronic equipment |
| CN111274999B (en) * | 2020-02-17 | 2024-04-19 | 北京迈格威科技有限公司 | Data processing method, image processing device and electronic equipment |
| CN113361529B (en) * | 2020-03-03 | 2024-05-10 | 北京四维图新科技股份有限公司 | Image semantic segmentation method and device, electronic equipment and storage medium |
| CN113361529A (en) * | 2020-03-03 | 2021-09-07 | 北京四维图新科技股份有限公司 | Image semantic segmentation method and device, electronic equipment and storage medium |
| CN111325751A (en) * | 2020-03-18 | 2020-06-23 | 重庆理工大学 | CT image segmentation system based on attention convolution neural network |
| CN111444957A (en) * | 2020-03-25 | 2020-07-24 | 腾讯科技（深圳）有限公司 | Image data processing method, image data processing device, computer equipment and storage medium |
| CN111444957B (en) * | 2020-03-25 | 2023-11-07 | 腾讯科技（深圳）有限公司 | Image data processing method, device, computer equipment and storage medium |
| CN111539325A (en) * | 2020-04-23 | 2020-08-14 | 四川旅游学院 | Forest fire detection method based on deep learning |
| CN111967478A (en) * | 2020-07-08 | 2020-11-20 | 特斯联科技集团有限公司 | Feature map reconstruction method and system based on weight inversion, storage medium and terminal |
| CN111967478B (en) * | 2020-07-08 | 2023-09-05 | 特斯联科技集团有限公司 | Feature map reconstruction method, system, storage medium and terminal based on weight overturn |
| CN112001248A (en) * | 2020-07-20 | 2020-11-27 | 北京百度网讯科技有限公司 | Active interaction method and device, electronic equipment and readable storage medium |
| CN112001248B (en) * | 2020-07-20 | 2024-03-01 | 北京百度网讯科技有限公司 | Active interaction method, device, electronic equipment and readable storage medium |
| CN112149694A (en) * | 2020-08-28 | 2020-12-29 | 特斯联科技集团有限公司 | Image processing method, system, storage medium and terminal based on convolutional neural network pooling module |
| CN112149694B (en) * | 2020-08-28 | 2024-04-05 | 特斯联科技集团有限公司 | Image processing method, system, storage medium and terminal based on convolutional neural network pooling module |
| CN112183645A (en) * | 2020-09-30 | 2021-01-05 | 深圳龙岗智能视听研究院 | Image aesthetic quality evaluation method based on context-aware attention mechanism |
| CN112465828B (en) * | 2020-12-15 | 2024-05-31 | 益升益恒(北京)医学技术股份公司 | Image semantic segmentation method and device, electronic equipment and storage medium |
| CN112465828A (en) * | 2020-12-15 | 2021-03-09 | 首都师范大学 | Image semantic segmentation method and device, electronic equipment and storage medium |
| CN112668656A (en) * | 2020-12-30 | 2021-04-16 | 深圳市优必选科技股份有限公司 | Image classification method and device, computer equipment and storage medium |
| CN112668656B (en) * | 2020-12-30 | 2023-10-13 | 深圳市优必选科技股份有限公司 | Image classification method, device, computer equipment and storage medium |
| CN113052760A (en) * | 2021-01-29 | 2021-06-29 | 成都商汤科技有限公司 | Pooling method, chip, equipment and storage medium |
| CN112767406A (en) * | 2021-02-02 | 2021-05-07 | 苏州大学 | Deep convolution neural network suitable for corneal ulcer segmentation of fluorescence staining slit lamp image |
| CN112767406B (en) * | 2021-02-02 | 2023-12-12 | 苏州大学 | Deep convolution neural network training method for corneal ulcer segmentation and segmentation method |
| CN113011465B (en) * | 2021-02-25 | 2021-09-03 | 浙江净禾智慧科技有限公司 | Household garbage throwing intelligent supervision method based on grouping multi-stage fusion |
| CN113011465A (en) * | 2021-02-25 | 2021-06-22 | 浙江净禾智慧科技有限公司 | Household garbage throwing intelligent supervision method based on grouping multi-stage fusion |
| CN113052173A (en) * | 2021-03-25 | 2021-06-29 | 北京百度网讯科技有限公司 | Sample data feature enhancement method and device |
| CN113486898A (en) * | 2021-07-08 | 2021-10-08 | 西安电子科技大学 | Radar signal RD image interference identification method and system based on improved ShuffleNet |
| CN113486898B (en) * | 2021-07-08 | 2024-05-31 | 西安电子科技大学 | Radar signal RD image interference identification method and system based on improvement ShuffleNet |
| CN113920099B (en) * | 2021-10-15 | 2022-08-30 | 深圳大学 | Polyp segmentation method based on non-local information extraction and related components |
| CN113920099A (en) * | 2021-10-15 | 2022-01-11 | 深圳大学 | Polyp segmentation method and device, computer equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110490813B (en) | 2021-12-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110490813A (en) | Characteristic pattern Enhancement Method, device, equipment and the medium of convolutional neural networks | |
| Ma et al. | Pconv: The missing but desirable sparsity in dnn weight pruning for real-time execution on mobile devices | |
| Liu et al. | Squeezedtext: A real-time scene text recognition by binary convolutional encoder-decoder network | |
| Shaik et al. | Multi-level attention network: application to brain tumor classification | |
| CN113222041B (en) | High-order association discovery fine-grained image identification method and device of graph structure representation | |
| Liu et al. | Flower classification via convolutional neural network | |
| Liu et al. | Deep learning based single sample face recognition: a survey | |
| Bi et al. | A survey on evolutionary computation for computer vision and image analysis: Past, present, and future trends | |
| CN110008853B (en) | Pedestrian detection network and model training method, detection method, medium and equipment | |
| CN113822209B (en) | Hyperspectral image recognition method and device, electronic equipment and readable storage medium | |
| Zeng et al. | Deep learning for scene classification: A survey | |
| CN106650781A (en) | Convolutional neural network image recognition method and device | |
| Al-Sabaawi et al. | Amended convolutional neural network with global average pooling for image classification | |
| Yu et al. | A lightweight and efficient model for surface tiny defect detection | |
| Wang et al. | Rapid detection of incomplete coal and gangue based on improved PSPNet | |
| Wang et al. | Multiscale densely connected attention network for hyperspectral image classification | |
| CN107967461A (en) | The training of SVM difference models and face verification method, apparatus, terminal and storage medium | |
| Kishorjit Singh et al. | Image classification using SLIC superpixel and FAAGKFCM image segmentation | |
| Hussain et al. | Additive deep feature optimization for semantic image retrieval | |
| Liu et al. | A novel image retrieval algorithm based on transfer learning and fusion features | |
| Xu et al. | Multi‐pyramid image spatial structure based on coarse‐to‐fine pyramid and scale space | |
| Ma et al. | Facial expression recognition method based on PSA—YOLO network | |
| Jin et al. | Research on image sentiment analysis technology based on sparse representation | |
| Zang et al. | A pooled Object Bank descriptor for image scene classification | |
| Liu et al. | Lightweight target detection algorithm based on YOLOv4 |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |