CN114494284A - Scene analysis model and method based on explicit supervision area relation - Google Patents
Scene analysis model and method based on explicit supervision area relation Download PDFInfo
- Publication number
- CN114494284A CN114494284A CN202111635864.7A CN202111635864A CN114494284A CN 114494284 A CN114494284 A CN 114494284A CN 202111635864 A CN202111635864 A CN 202111635864A CN 114494284 A CN114494284 A CN 114494284A
- Authority
- CN
- China
- Prior art keywords
- region
- layer
- scene
- module
- convolution
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000004458 analytical method Methods 0.000 title claims abstract description 96
- 238000000034 method Methods 0.000 title claims abstract description 23
- 238000012549 training Methods 0.000 claims abstract description 32
- 230000008569 process Effects 0.000 claims abstract description 10
- 238000004364 calculation method Methods 0.000 claims abstract description 5
- 238000011176 pooling Methods 0.000 claims description 23
- 238000007781 pre-processing Methods 0.000 claims description 17
- 239000011159 matrix material Substances 0.000 claims description 16
- 238000012937 correction Methods 0.000 claims description 13
- 238000000605 extraction Methods 0.000 claims description 9
- 230000004927 fusion Effects 0.000 claims description 9
- 238000010606 normalization Methods 0.000 claims description 8
- 238000012360 testing method Methods 0.000 claims description 6
- 230000006870 function Effects 0.000 claims description 5
- 238000012545 processing Methods 0.000 claims description 5
- 238000010276 construction Methods 0.000 claims description 4
- 238000005070 sampling Methods 0.000 claims description 4
- 238000011478 gradient descent method Methods 0.000 claims description 3
- 238000002407 reforming Methods 0.000 claims description 3
- 238000005215 recombination Methods 0.000 claims description 2
- 230000006798 recombination Effects 0.000 claims description 2
- 230000009466 transformation Effects 0.000 claims description 2
- 230000004438 eyesight Effects 0.000 abstract description 4
- 238000003709 image segmentation Methods 0.000 abstract description 2
- 230000004304 visual acuity Effects 0.000 abstract description 2
- 230000008901 benefit Effects 0.000 description 4
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000013527 convolutional neural network Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 238000003909 pattern recognition Methods 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 230000011218 segmentation Effects 0.000 description 2
- 238000013135 deep learning Methods 0.000 description 1
- 230000009977 dual effect Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/11—Region-based segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/243—Classification techniques relating to the number of classes
- G06F18/2431—Multiple classes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/048—Activation functions
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
- G06T2207/20221—Image fusion; Image merging
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Engineering & Computer Science (AREA)
- General Health & Medical Sciences (AREA)
- Software Systems (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Biophysics (AREA)
- Biomedical Technology (AREA)
- Mathematical Physics (AREA)
- Computational Linguistics (AREA)
- Health & Medical Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Evolutionary Biology (AREA)
- Image Analysis (AREA)
Abstract
The invention aims to provide a scene analysis model and a scene analysis method based on an explicit supervised region relation, and belongs to the technical field of image segmentation in computer vision. The scene analysis model disclosed by the invention respectively models the relations in the regions and between the regions in the scene picture, and gives the explicit supervision information corresponding to the relations between the two regions in the network training process, so that the intra-region relation module utilizes more accurate intra-region category prior information, and the discrimination capability of difficult categories is improved; the inter-region relation module captures more accurate inter-region category correlation, improves the resolving power of similar categories, and has the characteristics of high resolving accuracy, low extra calculation complexity and the like.
Description
Technical Field
The invention belongs to the technical field of image segmentation in computer vision, and particularly relates to a scene analysis system and method based on an explicit supervision region relation.
Background
In recent years, with the progress of deep learning technology, the field of computer vision has been developed vigorously, wherein scene analysis is one of the most challenging and representative tasks in the field. The main task of scene analysis is to assign a corresponding category label to each pixel in a scene picture according to a given input scene picture, so as to realize intelligent analysis and deep understanding of the scene. The task has wide application prospects in the aspects of robot navigation, automatic driving, virtual reality, accurate agriculture and the like, and is a research hotspot in the field of computer vision at home and abroad.
Most of the existing scene analysis methods are based on a Full Convolutional Network (FCN) structure, and benefit from the strong classification capability of a deep Convolutional Neural Network (CNN). The method based on the FCN structure has a good effect on pixel classification in simple scenes, but as the number of scene categories increases, the method based on the FCN structure is difficult to distinguish some similar categories, such as bookshelves and cabinets, and the resolution performance of complex scenes is reduced. Some recent studies[1][2]A self-attention mechanism is provided for modeling regional relations in a scene, and the problem of difficult analysis of similar categories is solved. However, the area relationships of these methods often lack explicit supervision information, and some unsupervised inter-area relationships captured by the self-attention mechanism may have negative effects on the network, so that similar category determination is easy to be confused, and the scene parsing performance is reduced. In addition, most of the existing methods focus on the relationship modeling between areas in scene pictures, and lack of relationship between areasModeling is not beneficial to solving the problem of difficult classification error in the region.
Therefore, how to solve the problem of the lack of explicit supervision on the intra-area and inter-area relationships in the existing scene analysis method becomes a research focus.
[1]Zhang,Hang,et al."Context encoding for semantic segmentation."Proceedings of the IEEE conference on Computer Vision and Pattern Recognition.2018.
[2]Fu,Jun,et al."Dual attention network for scene segmentation."Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2019.
Disclosure of Invention
Aiming at the problems in the background art, the invention aims to provide a scene analysis model and a scene analysis method based on an explicit supervision area relationship. The model respectively models the intra-area and inter-area relations in the scene picture, and gives the explicit supervision information corresponding to the two area relations in the network training process, so that the classification capability of the network on similar categories and difficult categories is improved, and the scene analysis performance is improved.
In order to achieve the purpose, the technical scheme of the invention is as follows:
a scene analysis model based on an explicit supervision region relation comprises a network preprocessing module, a deep expansion convolution feature extraction module, an expansion convolution space pooling pyramid module, a preliminary analysis module, an intra-region relation module, an inter-region relation module and a correction analysis module;
the network preprocessing module is used for preprocessing a scene picture to obtain a required scene image; the depth expansion convolution characteristic extraction module is used for extracting the image characteristics of the preprocessed scene image; the expanded convolution space pooling pyramid module is used for processing the extracted image features to obtain multi-scale enhanced image features;
the preliminary analysis module sequentially comprises a 3 x 3 convolutional layer, a batch normalization layer, a ReLU layer, a Dropout layer and a 1x1 convolutional layer, wherein the output dimension of the 1x1 convolutional layer is the number of scene type labels of the data set; after the multi-scale enhanced image features are input into a preliminary analysis module, a preliminary scene analysis result can be obtained;
the in-region relation module comprises a region dividing module and an in-region relation learning submodule which are sequentially connected; the region division submodule is a self-adaptive pooling layer, and the in-region relation learning submodule comprises a dynamic convolution layer, four continuous 3 x 3 convolution-batch normalization-ReLU layers and a sigmoid layer which are sequentially connected; the intra-region relation module divides input multi-scale enhanced image features into kxk regions through a region division submodule, then adaptively transforms different region features through a dynamic convolution layer, processes the transformed region features through four continuous 3 multiplied by 3 convolution-batch normalization-ReLU layers to obtain intra-region relation enhancement features, and obtains a predicted intra-region relation through a sigmoid layer;
the inter-region relation module sequentially comprises a region division submodule and a supervised Non-Local submodule, wherein the region division submodule is a self-adaptive pooling layer, and the supervised Non-Local submodule comprises three 1 multiplied by 1 convolutional layers; the inter-region relation module divides input multi-scale enhanced image features into kxk regions through a region division submodule, performs feature transformation and dimension recombination on the image features of different regions through a first 1 × 1 convolutional layer and a second 1 × 1 convolutional layer, performs matrix multiplication on a result obtained after dimension reforming of the 1 × 1 convolutional layer to obtain a predicted inter-region relation, wherein the relation is an inter-region category similarity matrix, and performs matrix multiplication on the predicted inter-region relation and the third 1 × 1 convolutional layer to obtain inter-region relation enhanced features;
the correction analysis module comprises a 3 x 3 convolution-batch normalization-ReLU layer, a Dropout layer, a 1x1 convolution layer and a linear interpolation up-sampling layer which are sequentially connected, wherein the output dimension of the 1x1 convolution layer is the number of scene type labels of the data set; the correction analysis module performs feature fusion on the intra-region relationship enhancement features and the inter-region relationship enhancement features through a 3 x 3 convolution-batch normalization-ReLU layer, corrects the fused features through a Dropout layer and a 1x1 convolution layer, linearly weights the corrected result and the initial scene analysis result to obtain a corrected analysis result, and upsamples the corrected analysis result to the original image resolution through a linear interpolation upsampling layer to obtain a final scene analysis result.
Further, the loss function of the scene parsing model is set as:
Ltotal＝Lseg+αLIntra+βLInter，
wherein L issegRepresents the cross-entropy loss, L, of the preliminary analysis moduleIntraRepresents the cross-entropy loss, L, of the relation blocks within a regionInterRepresenting the cross entropy loss of the relation module between the areas, wherein alpha and beta are preset loss coefficients;
wherein N is the number of image pixels, C is the number of scene categories, DKL(mi,ni) Is miAnd niThe KL distance between i and j are the pixels of the i and j columns, xi,jPredicting probability, y, for pixel classiA pixel class label; r is the number of image areas, miFor the predicted intra-area class distribution, niThe real intra-area category distribution is obtained; q. q.si,jFor predicted inter-region similarity, pi,jThe similarity between the real areas.
Further, the deep expansion convolution feature extraction module is a scaled-ResNet 101 network, and the network comprises an STEM module and four groups of residual error sub-networks; the STEM module comprises three continuous 3 multiplied by 3 convolution-batch normalization-ReLU layers and a maximum pooling layer with the step length of 2, and four groups of residual sub-networks respectively comprise 3, 4, 23 and 3 residual modules; and the residual modules in the third and fourth residual sub-networks respectively adopt expansion convolution with expansion coefficients of 2 and 4.
The expanded convolution space pooling pyramid module comprises four expanded convolution branches with different scales, a global branch and a feature fusion branch; the expansion convolution branch comprises a 3 x 3 convolution layer, a batch normalization layer and a ReLU layer, the expansion convolution coefficients of the four branches are 1, 12, 24 and 36, the global branch comprises a global pooling layer and a 3 x 3 convolution-batch normalization-ReLU layer, and the feature fusion branch comprises a feature cascade layer and a 3 x 3 convolution-batch normalization-ReLU layer.
Meanwhile, the invention also discloses a construction method for constructing the scene analysis model, which specifically comprises the following steps:
step 2, constructing a network structure of a scene analysis model based on the explicit supervision area relation;
step 3, training the network structure of the scene analysis model, and adding explicit supervision on the intra-region relation module and the inter-region relation module in the training process until the model network is converged and the network parameters are fixed;
and 4, verifying the scene analysis model with the network structure parameters on the test set to obtain the required scene analysis model.
Further, when the network structure is trained in the step 3, the predicted intra-area relationship obtained by the intra-area relationship module is subjected to calculation loss with the real intra-area category distribution during training, and is learned under the supervision of the calculated loss; the predicted inter-region relationship obtained by the inter-region relationship module is subjected to class similarity matrix calculation loss with the real inter-region during training, and is learned under the supervision of the matrix; two kinds of supervision information are directly calculated from the scene analysis category labels.
Further, the network structure hyper-parameters in step 3 include: and selecting a gradient descent method, and setting an updating mode of the learning rate, the maximum training iteration times, a loss function and the like.
In addition, the invention also discloses an analysis method based on the scene analysis model, which specifically comprises the following steps:
step 2, carrying out network training on the scene analysis model constructed by the invention based on the required scene image, and saving the trained scene analysis model as a scene analyzer when the network training meets the preset precision requirement;
and 3, carrying out image data preprocessing on the image to be analyzed, and then inputting the image to be analyzed into the scene analyzer obtained in the step 2 to obtain a scene analysis image result.
In summary, due to the adoption of the technical scheme, the invention has the beneficial effects that:
the scene analysis model disclosed by the invention utilizes more accurate intra-area category prior information through the intra-area relation module, so that the discrimination capability of difficult categories is improved; the inter-region relation module captures more accurate inter-region category correlation, improves the resolving power of similar categories, and has the characteristics of high resolving accuracy, low extra calculation complexity and the like. Compared with the existing scene analysis method, the model has more performance advantages in complex scenes.
Drawings
FIG. 1 is a schematic view of a scene parsing model processing flow according to the present invention.
FIG. 2 is a schematic diagram of an inter-domain relationship module in the scene parsing model according to the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention will be described in further detail with reference to the following embodiments and accompanying drawings.
In order to fully utilize scene inter-region information and intra-region information and more accurately complete scene analysis, the invention provides a scene analysis model and a scene analysis method based on an explicit supervision region relationship. The present invention will now be described more fully hereinafter with reference to the accompanying specific examples.
As shown in fig. 1, the scene analysis model based on the explicit supervised region relationship specifically includes: the system comprises a network preprocessing module, a depth expansion convolution characteristic extraction module, an expansion convolution space pooling pyramid module, a preliminary analysis module, an intra-region relation module, an inter-region relation module and a correction analysis module.
A construction method for constructing the scene analysis model specifically comprises the following steps:
the ADE20k data set comprises 20210 training set images and 2000 verification set images, and 150 scene category labels are contained in the data set; the Pascal-Context dataset includes 4998 training set images and 6105 test set images, for a total of 60 scene class labels, including 59 foreground object classes and a background class.
Step 2, constructing a scene analysis model based on the explicit supervision area relationship, which specifically comprises the following steps: the system comprises a network preprocessing module, a depth expansion convolution characteristic extraction module, an expansion convolution space pooling pyramid module, a preliminary analysis module, an intra-region relation module, an inter-region relation module and a correction analysis module.
The network preprocessing module is used for preprocessing the scene picture to obtain a required scene image; the preprocessing comprises the steps of carrying out mean value reduction and variance removal operation, random horizontal turning, image scaling and clipping and filling on the image to obtain a scene image with a fixed size of 512x 512.
The deep expansion convolution characteristic extraction module is a scaled-ResNet 101 network, and the network comprises an STEM module and four groups of residual error sub-networks; the STEM module comprises three continuous 3 multiplied by 3 convolution-batch normalization-ReLU layers and a maximum pooling layer with the step length of 2; the four groups of residual error sub-networks respectively comprise 3, 4, 23 and 3 residual error modules; and the residual modules in the third and fourth residual sub-networks respectively adopt expansion convolution with expansion coefficients of 2 and 4. The module inputs the pre-processed image and outputs the extracted image features.
The expanding convolution space pooling pyramid module comprises four expanding convolution branches with different scales, a global branch and a feature fusion branch; the expansion convolution branch comprises a 3 x 3 convolution layer, a batch normalization layer and a ReLU layer, expansion convolution coefficients of the four branches are 1, 12, 24 and 36, the global branch comprises a global pooling layer and a 3 x 3 convolution-batch normalization-ReLU layer, and the feature fusion branch comprises a feature cascade layer and a 3 x 3 convolution-batch normalization-ReLU layer. The module can perform multi-scale information enhancement on image features.
The preliminary analysis module sequentially comprises a 3 x 3 convolutional layer, a batch normalization layer, a ReLU layer, a Dropout layer and a 1x1 convolutional layer, wherein the output dimension of the 1x1 convolutional layer is the number of scene type labels of the data set; the module inputs the enhanced features of the multi-scale information and outputs a preliminary scene analysis result.
The intra-region relation module sequentially comprises a region division submodule, a self-adaptive pooling layer, a dynamic convolution layer, four continuous 3 multiplied by 3 convolution-batch normalization-ReLU layers and a sigmoid layer; the multi-scale enhanced image features are firstly input into a region dividing module, the image features are divided into kxk regions through a self-adaptive pooling layer, then a dynamic convolution layer is input, the region features are adaptively transformed according to different input images, then four continuous 3 x 3 convolution-batch normalization-ReLU layers are input to obtain more discriminative region relation enhancement features, and finally a sigmoid layer is input to obtain a predicted region relation, wherein the relation is the category distribution in the region.
The structural diagram of the inter-region relation module is shown in fig. 2, and the inter-region relation module sequentially comprises a region division submodule and a supervised Non-Local submodule, wherein the region division submodule is a self-adaptive pooling layer, and the supervised Non-Local submodule comprises three 1 × 1 convolution layers; inputting the image characteristics of multi-scale enhancement into a region division submodule firstly, dividing the image characteristics into kxk regions through a self-adaptive pooling layer, inputting a supervised Non-Local submodule, carrying out dimension reforming on the characteristics transformed by the first two 1x1 convolutional layers, and then carrying out matrix multiplication to obtain a predicted inter-region relationship, wherein the relationship is an inter-region category similarity matrix, and carrying out matrix multiplication on the relationship matrix and the last 1x1 convolutional layer to obtain a more discriminative inter-region relationship enhancement characteristic.
The correction analysis module sequentially comprises a 3 x 3 convolution-batch normalization-ReLU layer, a Dropout layer, a 1x1 convolution layer and a linear interpolation up-sampling layer, wherein the output dimension of the 1x1 convolution layer is the number of data set scene type labels; the input of the correction analysis module is the in-region relationship enhancement feature, the inter-region relationship enhancement feature and the preliminary analysis result. Firstly, inputting the intra-region relationship enhancement features and the inter-region relationship enhancement features into a 3 x 3 convolution-batch normalization-ReLU layer for feature fusion, then inputting a Dropout layer and a 1x1 convolution layer to obtain a correction result, linearly weighting the result and the primary analysis result to obtain a scene analysis result which is more refined after correction, and finally inputting a linear interpolation up-sampling layer to up-sample the analysis result to the original image resolution to obtain a final scene analysis result.
And 3, training a network structure of the scene analysis model, and adding explicit supervision on the intra-region relation module and the inter-region relation module in the training process until the model network is converged and the network parameters are fixed. The method selects a random gradient descent method to update network parameters, the initial learning rate is set to be 0.004, a Poly strategy is adopted to update the learning rate, the total number of training iterations is set to be 180 epochs, and the Batch Size (Batch Size) of training input is 16; the parameters of a Dropout layer in the preliminary analysis module and a Dropout layer in the correction analysis module are set to be 0.1;
the loss function of the scene parsing model is set as:
Ltotal＝Lseg+αLIntra+βLInter，
wherein L issegRepresents the cross-entropy loss, L, of the preliminary analysis moduleIntraRepresents the cross-entropy loss, L, of the relation blocks within a regionInterRepresenting the cross entropy loss of the inter-region relation module, wherein alpha and beta are preset loss coefficients, alpha is 0.5, and beta is 3.0;
wherein N is the number of image pixels, C is the number of scene categories, DKL(mi,ni) Is miAnd niThe KL distance between i and j are the pixels of the i and j columns, xi,jPredicting probability, y, for pixel classiA pixel class label; r is the number of image areas, miFor the predicted intra-area class distribution, niThe real intra-area category distribution is obtained; q. q ofi,jFor predicted inter-region similarity, pi,jIs the true inter-region similarity.
And the intra-area relation module monitors the intra-area category real distribution calculated from the scene analysis category label in the training process, enables the network to learn and predict the intra-area category distribution of the test picture through loss constraint, and is used as prior information of a network prediction scene analysis result in the test process to strengthen the discrimination capability of the network on the difficult category in the area.
The inter-region relation module network predicts the inter-region class similarity matrix through learning, so that the feature difference between different classes of scenes can be effectively enhanced, and the discrimination capability of the network on similar classes is improved. In the training process, the class similarity matrix between the supervised information regions of the module can be directly calculated from the scene analysis class label, and the prediction result of the class similarity matrix between the regions is restrained by using the binary cross entropy loss.
And 4, verifying the scene analysis model with the network structure parameters on the test set to obtain the required scene analysis model.
The comparison results of the scene analysis method based on the explicit supervised area relationship and the conventional scene analysis methods are shown in table 1. As can be seen from the table, on the two common scene parsing data sets ADE20k and Pascal-Contex, the parsing performance of the scene parsing model of the invention respectively reaches the average cross-over ratio (mIoU) of 46.9% and 54.9%, which is superior to that of other existing scene parsing models lacking explicit regional relation supervision. The method has the advantage of higher scene analysis result.
TABLE 1
While the invention has been described with reference to specific embodiments, any feature disclosed in this specification may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise; all of the disclosed features, or all of the method or process steps, may be combined in any combination, except mutually exclusive features and/or steps.
Claims (7)
1. A scene analysis model based on an explicit supervision region relation is characterized by comprising a network preprocessing module, a deep expansion convolution feature extraction module, an expansion convolution space pooling pyramid module, a primary analysis module, an intra-region relation module, an inter-region relation module and a correction analysis module;
the network preprocessing module is used for preprocessing a scene picture to obtain a required scene image; the depth expansion convolution characteristic extraction module is used for extracting the image characteristics of the preprocessed scene image; the expanded convolution space pooling pyramid module is used for processing the extracted image features to obtain multi-scale enhanced image features;
the preliminary analysis module comprises a 3 x 3 convolutional layer, a batch normalization layer, a ReLU layer, a Dropout layer and a 1x1 convolutional layer which are sequentially connected, wherein the output dimension of the 1x1 convolutional layer is the number of scene type labels of the data set; after the multi-scale enhanced image features are input into a preliminary analysis module, a preliminary scene analysis result can be obtained;
the in-region relation module comprises a region dividing module and an in-region relation learning submodule which are sequentially connected; the region division submodule is a self-adaptive pooling layer, and the in-region relation learning submodule comprises a dynamic convolution layer, four continuous 3 x 3 convolution-batch normalization-ReLU layers and a sigmoid layer which are sequentially connected; the intra-region relation module divides input multi-scale enhanced image features into kxk regions through a region division submodule, then adaptively transforms different region features through a dynamic convolution layer, processes the transformed region features through four continuous 3 multiplied by 3 convolution-batch normalization-ReLU layers to obtain intra-region relation enhancement features, and obtains a predicted intra-region relation through a sigmoid layer;
the inter-region relation module comprises a region division submodule and a supervised Non-Local submodule which are sequentially connected, wherein the region division submodule is a self-adaptive pooling layer, and the supervised Non-Local submodule comprises three 1 multiplied by 1 convolution layers; the inter-region relation module divides the input multi-scale enhanced image features into kxk regions through a region division submodule, performs feature transformation and dimension recombination on the image features of different regions through a first 1 × 1 convolutional layer and a second 1 × 1 convolutional layer, performs matrix multiplication on the result obtained after dimension reforming of the 1 × 1 convolutional layer to obtain a predicted inter-region relation, and performs matrix multiplication on the predicted inter-region relation and a third 1 × 1 convolutional layer to obtain inter-region relation enhanced features;
the correction analysis module comprises a 3 x 3 convolution-batch normalization-ReLU layer, a Dropout layer, a 1x1 convolution layer and a linear interpolation up-sampling layer which are sequentially connected, wherein the output dimension of the 1x1 convolution layer is the number of scene type labels of the data set; the correction analysis module performs feature fusion on the intra-region relationship enhancement features and the inter-region relationship enhancement features through a 3 x 3 convolution-batch normalization-ReLU layer, corrects the fused features through a Dropout layer and a 1x1 convolution layer, linearly weights the corrected result and the initial scene analysis result to obtain a corrected analysis result, and upsamples the corrected analysis result to the original image resolution through a linear interpolation upsampling layer to obtain a final scene analysis result.
2. The explicit supervised region relationship based scene resolution model of claim 1, wherein a penalty function of the scene resolution model is set to:
Ltotal＝Lseg+αLIntra+βLInter，
wherein L issegRepresents the cross-entropy loss, L, of the preliminary analysis moduleIntraRepresents the cross-entropy loss, L, of the relation blocks within a regionInterRepresenting the cross entropy loss of the relation module between the areas, wherein alpha and beta are preset loss coefficients;
wherein N is the number of image pixels, C is the number of scene categories, DKL(mi,ni) Is miAnd niThe KL distance between i and j are the pixels of the i and j columns, xi,jPredicting probability, y, for pixel classiA pixel class label; r is the number of image areas, miFor the predicted intra-area class distribution, niThe real intra-area category distribution is obtained; q. q.si,jFor predicted inter-region similarity, pi,jIs the true inter-region similarity.
3. The explicit supervised region relationship based scene resolution model of claim 1, wherein the deep extended convolution feature extraction module is a scaled-ResNet 101 network comprising a STEM module and four sets of residual subnetworks; the STEM module comprises three continuous 3 multiplied by 3 convolution-batch normalization-ReLU layers and a maximum pooling layer with the step length of 2, and four groups of residual sub-networks respectively comprise 3, 4, 23 and 3 residual modules; and the residual modules in the third and fourth residual sub-networks respectively adopt expansion convolution with expansion coefficients of 2 and 4.
The expanded convolution space pooling pyramid module comprises four expanded convolution branches with different scales, a global branch and a feature fusion branch; the expansion convolution branch comprises a 3 x 3 convolution layer, a batch normalization layer and a ReLU layer, the expansion convolution coefficients of the four branches are 1, 12, 24 and 36, the global branch comprises a global pooling layer and a 3 x 3 convolution-batch normalization-ReLU layer, and the feature fusion branch comprises a feature cascade layer and a 3 x 3 convolution-batch normalization-ReLU layer.
4. A method for constructing a scene analysis model is characterized by comprising the following steps:
step 1, selecting a training data set;
step 2, constructing a network structure of the scene analysis model based on the explicit supervised region relation as claimed in any one of claims 1 to 3;
step 3, training the network structure of the scene analysis model, and adding explicit supervision on the intra-region relation module and the inter-region relation module in the training process until the model network is converged and the network parameters are fixed;
and 4, verifying the scene analysis model with the network structure parameters on the test set to obtain the required scene analysis model.
5. The construction method according to claim 4, wherein, when the network structure is trained in step 3, the predicted intra-region relationship obtained by the intra-region relationship module is subjected to learning under the supervision of the calculated loss and the real intra-region class distribution during training; the predicted inter-region relationship obtained by the inter-region relationship module is subjected to class similarity matrix calculation loss with the real inter-region during training, and is learned under the supervision of the matrix; two kinds of supervision information are directly calculated from the scene analysis category labels.
6. The method of constructing as claimed in claim 4, wherein the network structure hyper-parameters in step 3 comprise: and selecting a gradient descent method, and setting an updating mode of the learning rate, the maximum training iteration times and a loss function.
7. A scene analysis method based on display supervision area relation is characterized by comprising the following steps:
step 1, acquiring a training data set, and performing image preprocessing on training data in the data set to generate and obtain a required scene image; the preprocessing comprises the steps of carrying out normalization processing, multi-scale random scaling, random overturning and the like on the image;
step 2, constructing a scene analysis model according to the construction method of claim 4, performing network training on the scene analysis model based on the required scene image, and saving the trained scene analysis model as a scene analyzer when the network training meets the preset precision requirement;
and 3, carrying out image data preprocessing on the image to be analyzed, and then inputting the image to be analyzed into the scene analyzer obtained in the step 2 to obtain a scene analysis image result.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111635864.7A CN114494284B (en) | 2021-12-29 | 2021-12-29 | Scene analysis model and method based on explicit supervision area relation |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111635864.7A CN114494284B (en) | 2021-12-29 | 2021-12-29 | Scene analysis model and method based on explicit supervision area relation |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114494284A true CN114494284A (en) | 2022-05-13 |
| CN114494284B CN114494284B (en) | 2023-04-18 |
Family
ID=81507834
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111635864.7A Active CN114494284B (en) | 2021-12-29 | 2021-12-29 | Scene analysis model and method based on explicit supervision area relation |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114494284B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117423020A (en) * | 2023-12-19 | 2024-01-19 | 临沂大学 | Dynamic characteristic and context enhancement method for detecting small target of unmanned aerial vehicle |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111062441A (en) * | 2019-12-18 | 2020-04-24 | 武汉大学 | Scene classification method and device based on self-supervision mechanism and regional suggestion network |
| CN113160246A (en) * | 2021-04-14 | 2021-07-23 | 中国科学院光电技术研究所 | Image semantic segmentation method based on depth supervision |
| US20210279950A1 (en) * | 2020-03-04 | 2021-09-09 | Magic Leap, Inc. | Systems and methods for efficient floorplan generation from 3d scans of indoor scenes |
-
2021
- 2021-12-29 CN CN202111635864.7A patent/CN114494284B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111062441A (en) * | 2019-12-18 | 2020-04-24 | 武汉大学 | Scene classification method and device based on self-supervision mechanism and regional suggestion network |
| US20210279950A1 (en) * | 2020-03-04 | 2021-09-09 | Magic Leap, Inc. | Systems and methods for efficient floorplan generation from 3d scans of indoor scenes |
| CN113160246A (en) * | 2021-04-14 | 2021-07-23 | 中国科学院光电技术研究所 | Image semantic segmentation method based on depth supervision |
Non-Patent Citations (3)
| Title |
|---|
| QINGBO WU 等: "Subjective and Objective De-Raining Quality Assessment Towards Authentic Rain Image" * |
| 张晋: "\"基于局部感知的场景图像识别 \"" * |
| 曹玉红 等: "基于深度学习的医学影像分割研究综述" * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117423020A (en) * | 2023-12-19 | 2024-01-19 | 临沂大学 | Dynamic characteristic and context enhancement method for detecting small target of unmanned aerial vehicle |
| CN117423020B (en) * | 2023-12-19 | 2024-02-27 | 临沂大学 | Dynamic characteristic and context enhancement method for detecting small target of unmanned aerial vehicle |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114494284B (en) | 2023-04-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110335290B (en) | Twin candidate region generation network target tracking method based on attention mechanism | |
| CN111626300B (en) | Image segmentation method and modeling method of image semantic segmentation model based on context perception | |
| CN113033570B (en) | Image semantic segmentation method for improving void convolution and multilevel characteristic information fusion | |
| CN108804397B (en) | Chinese character font conversion generation method based on small amount of target fonts | |
| CN111325165B (en) | Urban remote sensing image scene classification method considering spatial relationship information | |
| CN111091130A (en) | Real-time image semantic segmentation method and system based on lightweight convolutional neural network | |
| CN112381097A (en) | Scene semantic segmentation method based on deep learning | |
| CN111696110B (en) | Scene segmentation method and system | |
| CN113807355A (en) | Image semantic segmentation method based on coding and decoding structure | |
| CN111242844B (en) | Image processing method, device, server and storage medium | |
| CN114943963A (en) | Remote sensing image cloud and cloud shadow segmentation method based on double-branch fusion network | |
| CN113066089B (en) | Real-time image semantic segmentation method based on attention guide mechanism | |
| CN110852199A (en) | Foreground extraction method based on double-frame coding and decoding model | |
| CN113159067A (en) | Fine-grained image identification method and device based on multi-grained local feature soft association aggregation | |
| CN114973011A (en) | High-resolution remote sensing image building extraction method based on deep learning | |
| CN114821058A (en) | Image semantic segmentation method and device, electronic equipment and storage medium | |
| CN113516133A (en) | Multi-modal image classification method and system | |
| CN113240683A (en) | Attention mechanism-based lightweight semantic segmentation model construction method | |
| CN115545166A (en) | Improved ConvNeXt convolutional neural network and remote sensing image classification method thereof | |
| CN116863194A (en) | Foot ulcer image classification method, system, equipment and medium | |
| CN114494284B (en) | Scene analysis model and method based on explicit supervision area relation | |
| CN117058235A (en) | Visual positioning method crossing various indoor scenes | |
| CN113096133A (en) | Method for constructing semantic segmentation network based on attention mechanism | |
| CN117011655A (en) | Adaptive region selection feature fusion based method, target tracking method and system | |
| CN116311349A (en) | Human body key point detection method based on lightweight neural network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |