CN113435268A - Earthquake disaster area remote sensing image interpretation method based on graph transformation knowledge embedding algorithm - Google Patents
Earthquake disaster area remote sensing image interpretation method based on graph transformation knowledge embedding algorithm Download PDFInfo
- Publication number
- CN113435268A CN113435268A CN202110644598.8A CN202110644598A CN113435268A CN 113435268 A CN113435268 A CN 113435268A CN 202110644598 A CN202110644598 A CN 202110644598A CN 113435268 A CN113435268 A CN 113435268A
- Authority
- CN
- China
- Prior art keywords
- remote sensing
- sensing image
- sample
- training
- earthquake
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 51
- 238000004422 calculation algorithm Methods 0.000 title claims abstract description 22
- 230000009466 transformation Effects 0.000 title claims abstract description 19
- 230000011218 segmentation Effects 0.000 claims abstract description 90
- 238000012549 training Methods 0.000 claims abstract description 64
- 238000005520 cutting process Methods 0.000 claims abstract description 18
- 230000008569 process Effects 0.000 claims abstract description 13
- 238000011160 research Methods 0.000 claims abstract description 12
- 230000002776 aggregation Effects 0.000 claims abstract description 7
- 238000004220 aggregation Methods 0.000 claims abstract description 7
- 239000011159 matrix material Substances 0.000 claims description 38
- 238000012795 verification Methods 0.000 claims description 16
- 238000004364 calculation method Methods 0.000 claims description 12
- 238000013527 convolutional neural network Methods 0.000 claims description 12
- 238000004590 computer program Methods 0.000 claims description 9
- 238000007781 pre-processing Methods 0.000 claims description 9
- 238000012937 correction Methods 0.000 claims description 8
- 238000012545 processing Methods 0.000 claims description 7
- 238000003860 storage Methods 0.000 claims description 5
- 238000012896 Statistical algorithm Methods 0.000 claims description 4
- VWTINHYPRWEBQY-UHFFFAOYSA-N denatonium Chemical compound [O-]C(=O)C1=CC=CC=C1.C=1C=CC=CC=1C[N+](CC)(CC)CC(=O)NC1=C(C)C=CC=C1C VWTINHYPRWEBQY-UHFFFAOYSA-N 0.000 claims description 4
- 238000002372 labelling Methods 0.000 claims description 3
- 238000013528 artificial neural network Methods 0.000 description 8
- 238000010586 diagram Methods 0.000 description 6
- 238000013135 deep learning Methods 0.000 description 5
- 238000003062 neural network model Methods 0.000 description 4
- 238000006243 chemical reaction Methods 0.000 description 3
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 238000013461 design Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 238000004458 analytical method Methods 0.000 description 2
- 238000013473 artificial intelligence Methods 0.000 description 2
- 230000002457 bidirectional effect Effects 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 238000010276 construction Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000003595 spectral effect Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000013136 deep learning model Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000011176 pooling Methods 0.000 description 1
- 239000002689 soil Substances 0.000 description 1
- 238000001228 spectrum Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Health & Medical Sciences (AREA)
- Software Systems (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Biophysics (AREA)
- Biomedical Technology (AREA)
- Mathematical Physics (AREA)
- Computational Linguistics (AREA)
- Health & Medical Sciences (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Biology (AREA)
- Image Analysis (AREA)
Abstract
The invention relates to the technical field of earthquake disaster interpretation, in particular to an earthquake disaster area remote sensing image interpretation method based on a graph transformation knowledge embedding algorithm. Acquiring remote sensing image data of an earthquake-stricken area; inputting the remote sensing image data of the earthquake-stricken area into a remote sensing image semantic segmentation model obtained by pre-training; outputting an interpretation result; the pre-training process of the remote sensing image semantic segmentation model comprises the following steps: obtaining a remote sensing image of a research area; constructing an initial remote sensing image semantic segmentation model; training the initial remote sensing image semantic segmentation modelThe method for constructing the initial remote sensing image semantic segmentation model comprises the following steps: computing the original graph structure G of a single sampleori(ii) a Traversing and copying the nodes in the original graph based on the total category number, and obtaining the structure G of the original graphoriConverted into a knowledge hypergraph structure Gnew(ii) a Carrying out node feature aggregation in a GCN model; and cutting the knowledge hypergraph structure through the classifier to obtain an initial remote sensing image semantic segmentation model.
Description
Technical Field
The invention relates to the technical field of earthquake disaster interpretation, in particular to an earthquake disaster area remote sensing image interpretation method based on a graph transformation knowledge embedding algorithm.
Background
Semantic segmentation and object recognition of remote sensing images are always important research contents in the field of remote sensing, and the remote sensing technology is also an important research direction for analysis after earthquake disasters. In recent years, artificial intelligence technology is rapidly developed, and the technical method of artificial intelligence interacts with many fields, of course, the remote sensing field is also one of the fields. In the aspect of remote sensing application, many researchers use deep learning for interpretation of remote sensing images, and the application effect is very considerable. However, in the field of remote sensing, the traditional deep learning method is mostly to use the convolutional neural network CNN based on pixel level for remote sensing interpretation, and the interpretation of the method is inefficient and consumes a large amount of computing resources. With the development of another deep learning model, the graph neural network model can realize object-level-based semantic segmentation, which improves the efficiency of remote sensing image interpretation and greatly reduces consumed computing resources. The Graph Convolutional neural Network (GCN) is a Graph neural Network model used for semantic segmentation of remote sensing images, and is a main reference model for the research.
The conventional atlas neural network is used for interpretation of remote sensing image geographic objects and has the following problems:
spatial adjacency over geographic objects: based on the adjacency matrix of spatial relationships, the phenomenon of the first law of inverse geography occurs, leading to misinterpretation of geographic objects.
The geographic object receptive field remains limited: object-level based graph neural networks have greatly expanded the receptive field, but still do not obtain more global information, compared to pixel-level based convolutional neural networks.
Due to the problems, a method is urgently needed to improve the graph convolution neural network, and a more efficient and accurate remote sensing image interpretation result of the earthquake-stricken area is realized.
Disclosure of Invention
The invention aims to provide a graph transformation knowledge embedding algorithm-based earthquake-stricken area remote sensing image interpretation method aiming at the defects of the prior art, and the more efficient and accurate interpretation result of the earthquake-stricken area remote sensing image can be realized.
The scheme provides a graph transformation knowledge embedding algorithm-based earthquake disaster area remote sensing image interpretation method, which comprises the following steps:
acquiring remote sensing image data of the earthquake-stricken area;
inputting the remote sensing image data of the earthquake-stricken area into a remote sensing image semantic segmentation model obtained by pre-training;
outputting an interpretation result;
the pre-training process of the remote sensing image semantic segmentation model comprises the following steps:
obtaining a remote sensing image of a research area;
processing the remote sensing image to obtain sample image characteristics, sample object segmentation data, a sample GT, a sample space adjacency matrix and data set class co-occurrence probability matrix data;
constructing an initial remote sensing image semantic segmentation model;
training the initial remote sensing image semantic segmentation model based on sample image features, sample object segmentation data, a sample GT, a sample space adjacency matrix and a data set class co-occurrence probability matrix to obtain a remote sensing image semantic segmentation model;
the method for constructing the initial remote sensing image semantic segmentation model comprises the following steps:
computing the original graph structure G of a single sampleori；
Traversing nodes in an original graph based on total number of classesCopying the original graph structure GoriConverted into a knowledge hypergraph structure Gnew；
Carrying out node feature aggregation in a GCN model;
and cutting the knowledge hypergraph structure through the classifier to obtain an initial remote sensing image semantic segmentation model.
Preferably, the original graph structure GoriThe following formula is used for calculation:
Gori＝(Vori，Eori)
Vori＝{vi，i∈{1，…，N}}
Eori＝{eij，i，j∈{1，…，N}}
wherein, VoriIs a set of object nodes in a single sample generated by superpixel segmentation, EoriAs a set of contiguous relationships between object nodes, eij＝aij，aijIs the spatial adjacency relation of the node i and the node j obtained by the calculation of a sample spatial adjacency matrix, N is the number of objects in a single sample, viIs the superpixel object node i.
Preferably, the knowledge hypergraph structure GnewThe following formula is used for calculation:
Gnew＝(Vnew，Enew)
Vnew＝{viα，i∈{1，...，N}，α∈{1，...，C}}
Enew＝{e(iα，jβ)，i，j∈{1，...，N}，α，β∈{1，...，C}}
wherein, VnewIs the object node set of the knowledge hypergraph obtained by copying the nodes of the original graph for C times, wherein C is the total category number viαExpressed as inode as alpha class, EnewExpressed as a set of contiguous relationships between object nodes, e(iα，jβ)Is a node viαAnd node vjβV. of adjacent relationjβThe node denoted j is a β class.
Preferably, the processing the remote sensing image includes:
performing superpixel segmentation on the remote sensing image to obtain geographic segmentation object vector data;
GT labeling is carried out on the vector data of the geographic segmentation object according to a ground surface coverage classification system to obtain GT grid data;
and cutting the remote sensing image, the geographic segmentation object vector data and the GT raster data by using the sample pre-selection frame vector file to obtain a complete data set, wherein the complete data set comprises a plurality of samples, and each sample comprises a sample original image file, a sample object segmentation file and a sample GT file.
Preferably, the obtaining of the image characteristics of the sample includes:
constructing a VGG-19 convolutional neural network;
and inputting the sample original image file into a VGG-19 convolutional neural network to obtain sample image characteristics.
Preferably, the sample space adjacency matrix is obtained by calculating the adjacency distance of the sample geographic segmentation objects pairwise, and the data set class co-occurrence probability matrix is obtained by counting the sample GT of the complete data set through a class co-occurrence probability statistical algorithm.
Preferably, after the remote sensing image of the research area is obtained, the method further comprises preprocessing the remote sensing image, wherein the preprocessing comprises atmospheric correction, geometric correction, radiometric calibration and cutting operation.
Preferably, before the initial remote sensing image semantic segmentation model is trained, the method further comprises data set division, wherein the data set is divided into a training set and a verification set, and the ratio of the number of samples of the training set to the number of samples of the verification set is 3: 1.
Preferably, before the initial remote sensing image semantic segmentation model is trained, the initial remote sensing image semantic segmentation model further comprises model training parameter setting, the learning rate of model training is 0.0001, the batch _ size of single training is 1, the number of times of model training is 400, and the Dropout parameter is 0.6.
The present solution also provides a computer-readable storage medium storing a computer program which, when executed by a processor, implements the steps of the method described above.
The invention has the beneficial effects that: according to the scheme, all nodes in the graph are traversed, the condition that one node is any one of a plurality of classes is considered, the node in the original graph is copied and expressed for many times as the node is in a certain class, therefore, the node structure of the knowledge hypergraph is completed, and embedding of knowledge in the training process is further achieved. And geographic prior knowledge (category co-occurrence probability matrix) obtained by data set statistics is combined with a graph convolution neural network, so that the receptive field is expanded to the whole data set from a single sample, more global context information can be obtained, more efficient and accurate remote sensing interpretation results of earthquake-stricken areas can be realized, and conditions are provided for analyzing the influence of earthquake disasters on other ground objects. The relation of the nodes in the knowledge hypergraph is the combination of the spatial adjacency relation and the category co-occurrence probability, and then the construction of the knowledge hypergraph edge is completed, so that the conversion from the original graph to the knowledge hypergraph is realized. And then, carrying out feature aggregation on the nodes in the knowledge hypergraph based on a GCN model to realize the updating of the node features. And finally, cutting the knowledge hypergraph into the number of nodes of the original graph, thereby completing the transformation from the knowledge hypergraph to the original graph and realizing the class prediction of the nodes.
Drawings
FIG. 1 is a schematic flow diagram of the present invention;
FIG. 2 is a schematic diagram of the construction of a semantic segmentation model of a remote sensing image according to the present invention;
FIG. 3 is a schematic diagram of the method for constructing the initial remote sensing image semantic segmentation model based on the graph change knowledge embedding algorithm.
Fig. 4 is a structural diagram of the semantic segmentation model of the remote sensing image according to the present invention.
Detailed Description
In order to make the technical problems, technical solutions and advantageous effects to be solved by the present application clearer, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.
It will be understood that when an element is referred to as being "secured to" or "disposed on" another element, it can be directly on the other element or be indirectly on the other element. When an element is referred to as being "connected to" another element, it can be directly connected to the other element or be indirectly connected to the other element.
It will be understood that the terms "length," "width," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," and the like, refer to an orientation or positional relationship illustrated in the drawings for convenience in describing the present application and to simplify description, and do not indicate or imply that the referenced device or element must have a particular orientation, be constructed and operated in a particular orientation, and thus should not be construed as limiting the present application.
Furthermore, the terms "first", "second" and "first" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of the present application, "a plurality" means two or more unless specifically limited otherwise.
The invention designs a knowledge embedding algorithm graph convolution neural network model based on graph transformation, which realizes the combination of geographic prior knowledge and a graph convolution neural network to acquire global context information. Clustering is carried out on each sample according to spectral characteristics to realize superpixel segmentation so as to obtain a geographic object mask, a pre-trained VGG-19 convolutional neural network is used for extracting sample image characteristics, and the geographic object image characteristics are obtained through calculation of the geographic object mask and the sample image characteristics; then, calculating a category co-occurrence probability matrix of the data set according to a category co-occurrence probability algorithm, and then obtaining geographic prior knowledge; then, calculating a spatial adjacency relation between every two geographic object masks in a single sample to obtain a spatial adjacency matrix; and finally, constructing an original graph structure model of the single sample by using the image features of the geographic objects in the single sample and the spatial adjacency matrix, and inputting the constructed original graph structure and the geographic prior knowledge into a knowledge embedding graph convolution neural network model based on graph transformation.
Fig. 1 shows a schematic flow diagram of the present solution, detailed as follows:
step 1, acquiring remote sensing image data of an earthquake-stricken area;
step 2, inputting the remote sensing image data of the earthquake-stricken area into a remote sensing image semantic segmentation model obtained by pre-training;
and 3, outputting an interpretation result.
The pre-training process of the semantic segmentation model of the remote sensing image is shown in fig. 2 and comprises the following steps:
s201, data acquisition. Namely, the high-resolution remote sensing image of the research area of the method is collected.
And S202, preprocessing data. Aiming at the collected remote sensing image of the location of the research area, ENVI professional software is used for finishing a series of conventional remote sensing image preprocessing, including operations such as atmospheric correction, geometric correction, radiometric calibration, cutting and the like.
And S203, super-pixel segmentation. And performing superpixel segmentation on the preprocessed remote sensing image of the research area by using SuperSIAT software to obtain raster data of the geographic object segmentation result. And then, carrying out grid vector conversion operation on the grid data of the geographic object segmentation result by using ArcGIS to obtain geographic segmentation object vector data.
And S204, designing a ground surface coverage classification system. Because the invention is used for interpreting the remote sensing image of the earthquake-stricken area, the earth surface covering earth object system is determined by the earth object class after the earthquake disaster.
S205, GT mark. And marking GT in an attribute table of the geographic segmentation object vector data in ArcGIS software according to a designed earth surface coverage classification system, and then performing vector-to-grid operation on the marked geographic segmentation object vector data to obtain GT grid data.
And S206, cutting the sample. Based on the determined sample size, a sample pre-selection box is selected. And then, in ArcGIS software, cutting raster data such as remote sensing images, geographic object segmentation results, GT and the like by using a sample pre-selection frame vector file. A complete data set is obtained, including the sample original image, the sample object segmentation and the sample GT.
And S207, calculating a sample space adjacency matrix. And (3) calculating the adjacent distance of every two super-pixel segmentation results, namely sample geographic segmentation objects, wherein the spatial adjacent distance between the super-pixel segmentation objects is 1, the direct adjacent distance is 0.5, the indirect adjacent distance is 0.25, and finally the indirect adjacent distance is 0, so that the irrelevance is represented, and then obtaining a sample spatial adjacent matrix.
And S208, calculating the geographical prior knowledge of the data set. The invention obtains the category co-occurrence probability matrix of the data set based on the category co-occurrence probability statistical algorithm based on the geographical prior knowledge, namely the category co-occurrence probability.
S209, calculating the original graph structure G of a single sampleori。
Gori＝(Vori，Eori)
Vori＝{vi，i∈{1，…，N}}
Eori＝{eij，i，j∈{1，…，N}}
Wherein, VoriIs a set of object nodes in a single sample generated by superpixel segmentation, EoriAs a set of contiguous relationships between object nodes, eij＝aij，aijIs the spatial adjacency relation of the node i and the node j obtained by the calculation of a sample spatial adjacency matrix, N is the number of objects in a single sample, viIs the superpixel object node i.
S210, traversing and copying the nodes in the original graph based on the total category number, and converting the original graph structure into a knowledge hypergraph structure Gnew。
Gnew＝(Vnew，Enew)
Vnew＝{viα，i∈{1，...，N}，α∈{1，...，C}}
Enew＝{e(iα，jβ)，i，j∈{1，...，N}，α，β∈{1，...，C}}
Wherein, VnewIs made from the originalCopying C times to obtain object node set of knowledge hypergraph by graph nodes, wherein C is total category number viαExpressed as inode as alpha class, EnewExpressed as a set of contiguous relationships between object nodes, e(iα，jβ)Is a node viαAnd node vjβV. of adjacent relationjβThe node denoted j is a β class.
e(iα，jβ)Combining the spatial adjacency relation and the class co-occurrence probability, e is characterized in that the class co-occurrence probability comprises a forward co-occurrence probability and a reverse co-occurrence probability(iα，jβ)Is also a bidirectional edge, e(iα，jβ)＝a(iα，jβ)Or a'(iα，jβ) The calculation formula is as follows:
a(iα，jβ)＝aij×mαβ，mαβ∈M
a′(iα，jβ)＝aij×mβα，mβα∈M
wherein a is(iα，jβ)Denotes a forward side, a'(iα，jβ)The reverse edge is represented, and M is a co-occurrence probability matrix, which is generated by the step S8.
And S211, carrying out node feature aggregation in the GCN model.
S212, cutting the knowledge hypergraph structure through the classifier to obtain an initial remote sensing image semantic segmentation model.
And S213, dividing the data set. And dividing the whole data set into a training set and a verification set for training the network model, wherein the ratio of the number of samples of the training set to the number of samples of the verification set is 3: 1, and randomly dividing the whole data set into the training set and the verification set based on the ratio.
And S214, setting model training parameters. For the knowledge-embedded graph convolution neural network, the training parameters required to be set comprise the learning rate of model training, the batch _ size of single training and the number of times of model training, and in addition, a Dropout parameter is required to be set, so that the problem of overfitting in the model training process is solved. The convolution kernel size and number as well as the pooling size need to be set when setting up the VGG-19 pre-training model.
And S215, training a model. Input data is loaded, wherein the input data comprises sample image characteristics, sample object segmentation data, a sample GT, a sample space adjacency matrix and a data set class co-occurrence probability matrix. And training the initial remote sensing image semantic segmentation model to obtain a remote sensing image semantic segmentation model.
Wherein, the acquisition of sample image characteristics includes:
constructing a VGG-19 convolutional neural network;
and inputting the sample original image file into a VGG-19 convolutional neural network to obtain sample image characteristics.
Example one
And (4) training the semantic segmentation model of the remote sensing image obtained in advance according to the scheme. Taking Wenchuan county as an example, a preferred implementation mode is provided, and the process is as follows:
s201, data acquisition. The high-resolution remote sensing image of the research area in the method is acquired, and the Quickbird remote sensing image with the resolution of 0.5m in the area of 30 degrees 28 '41' -30 degrees 32 '29' north latitude and 114 degrees 22 '42' -114 degrees 28 '11' east longitude after earthquake disaster in Wenchun county of Sichuan province, which is shot in 7 months of 2008 is acquired in the example.
And S202, preprocessing data. Aiming at the collected remote sensing images in partial regions of Wenchuan county, ENVI professional software is used for finishing the remote sensing image preprocessing work such as atmospheric correction, geometric correction, radiometric calibration, cutting and the like, and the ENVI software is used for preprocessing more popular and has higher data processing precision.
And S203, super-pixel segmentation. And performing super-pixel segmentation operation on the preprocessed Wenchuan county remote sensing image by using SuperSIAT software to obtain raster data of a geographic object segmentation result. And then, carrying out grid vector conversion operation on the grid data of the geographic object segmentation result by using ArcGIS to finally obtain the vector data of the geographic segmentation object, and carrying out super-pixel segmentation to greatly improve the efficiency of model training.
And S204, designing a ground surface coverage classification system. Because the invention is used for interpreting the remote sensing image of the earthquake-stricken area, the earth surface covering earth object system is determined by the earth object class after the earthquake disaster. The earth surface coverage categories after the Wenchuan disaster are flat field, landslide, grassland, water body, village, highway, soil road, town, terrace, strip field, greening grassland, thinning forest and greening thinning forest. Therefore, the ground surface covering classification system takes the 13 types as the standard, and the designed classification system can cover most of the ground object types in the earthquake-stricken area and meet the requirement of identifying all the ground object types.
S205, GT (ground Truth) label. Establishing a class field for an attribute table of the geographic segmentation object vector data in ArcGIS software, marking a class number corresponding to an object in the class field, and marking GT according to the spectral characteristics, shapes and the like of various ground features in a classification system. And then, carrying out vector-to-grid operation on the marked geographic object segmentation vector data to obtain GT grid data. The GT labeling method is fast and accurate.
And S206, cutting the sample. The sample cut size determination is 224 x 224 size, and the sample pre-selection box is first selected. And then, in ArcGIS software, cutting raster data such as remote sensing images, geographic object segmentation results, GT and the like by using a sample pre-selection frame vector file. And finally, obtaining a complete data set which comprises a sample original image, sample object segmentation and a sample GT. The data set comprises 1680 samples, each sample comprises an original image file, an object segmentation file and a sample GT file, and the files are in TIF format. The sample cutting means is convenient and relatively easy to realize.
And S207, calculating a sample space adjacency matrix. And calculating the spatial adjacency distance of every two objects for each sample object segmentation file, wherein the spatial adjacency between the sample object segmentation file and the sample object segmentation file is 1, the direct adjacency is 0.5, the indirect adjacency is 0.25, and finally 0 represents no correlation, so that a sample spatial adjacency matrix is obtained and is stored as an npy file. The calculation method of the sample geographic object spatial adjacency matrix is simple and easy to understand and has high accuracy.
And S208, calculating the geographical prior knowledge of the data set. The invention obtains the category co-occurrence probability matrix of the data set based on the category co-occurrence probability statistical algorithm based on the geographical prior knowledge, namely the category co-occurrence probability. This is the result of statistics on the samples GT of the entire dataset, and the class co-occurrence probability matrix of the statistical dataset is stored as an npy file. The category co-occurrence probability calculation formula is as follows:
wherein n isαβRepresenting the number of samples simultaneously containing the alpha and beta types of ground objects; n isαRepresenting the number of samples containing the alpha category ground features; n isβRepresenting the number of samples containing the beta category feature. Wherein m isαβRepresenting the probability of the occurrence of the beta-type ground objects near the alpha-type ground objects, namely the forward co-occurrence probability; wherein m isβαThe probability of the occurrence of the alpha-type feature near the beta-type feature is expressed as a reverse co-occurrence probability. The way of representing the geographical prior knowledge by category co-occurrence probability is easily accepted and computationally efficient.
S209, calculating the original graph structure G of a single sampleori. For example, fig. 3 shows a process of converting an original graph into a knowledge hypergraph, performing node feature aggregation in a GCN network model, and cutting a knowledge hypergraph structure through a classifier to obtain an initial remote sensing image semantic segmentation model.
Gori＝(Vori，Eori)
Vori＝{vi，i∈{1，…，N}}
Eori＝{eij，i，j∈{1，…，N}}
Wherein, VoriIs a set of object nodes in a single sample generated by superpixel segmentation, EoriAs a set of contiguous relationships between object nodes, eij＝aij，aijIs the spatial adjacency relation of the node i and the node j obtained by the calculation of a sample spatial adjacency matrix, N is the number of objects in a single sample, viIs the superpixel object node i.
S210, traversing and copying the nodes in the original graph based on the total category numberConverting the original graph structure into a knowledge hypergraph structure Gnew。
Gnew＝(Vnew，Enew)
Vnew＝{viα，i∈{1，...，N}，α∈{1，...，C}}
Enew＝{e(iα，jβ)，i，j∈{1，...，N}，α，β∈{1，...，C}}
Wherein, VnewIs the object node set of the knowledge hypergraph obtained by copying the nodes of the original graph for C times, wherein C is the total category number viαExpressed as inode as alpha class, EnewExpressed as a set of contiguous relationships between object nodes, e(iα，jβ)Is a node viαAnd node vjβV. of adjacent relationjβThe node denoted j is a β class.
e(iα，jβ)The column combines the spatial adjacency and the class co-occurrence probability, since the class co-occurrence probability includes the forward co-occurrence probability and the reverse co-occurrence probability, e(iα，jβ)Is also a bidirectional edge, e(iα，jβ)＝a(iα，jβ)Or a'(iα，jβ)The calculation formula is as follows:
a(iα，jβ)＝aij×mαβ，mαβ∈M
a′(iα，jβ)＝aij×mβα，mβα∈M
wherein a is(iα，jβ)Denotes a forward side, a'(iα，jβ)Representing the reverse edge, M is the co-occurrence probability matrix, generated by step S208.
And S211, carrying out node feature aggregation in the GCN model.
S212, cutting the knowledge hypergraph structure through the classifier to obtain an initial remote sensing image semantic segmentation model. The knowledge embedding algorithm based on graph transformation follows a class-agnostic mechanism in a model training process, the design concept of the algorithm is novel, and symbiotic concepts are applied to model training. Aiming at the problem of foreign matter co-spectrum in the field of remote sensing, a solution algorithm is provided.
And S213, dividing the data set. Dividing the whole data set into a training set and a verification set for training a network model, wherein the number ratio of samples of the training set to the verification set is 3: 1, randomly dividing the whole data set into the training set and the verification set based on the ratio, and finally obtaining 1280 training set samples and 400 verification set samples. The data division proportion is popular, and accords with deep learning network model training.
And S214, setting model training parameters. For the knowledge-embedded graph convolution neural network, the learning rate is set to 0.0001, the batch _ size of a single training is set to 1, the number of model trainings is 400, and the Dropout parameter is set to 0.6. For the VGG-19 pre-training model, the convolution kernel size is set to 3 × 3, the feature map size is N × 32(N is the number of sample objects), the learning rate is set to 0.0001, the batch _ size of a single training is set to 1, and the number of times of model training is 400. The training parameter setting is obtained by carrying out multiple times of experimental comparison, and the deep learning network model training is in accordance with the patent.
And S215, training a model. Loading input data, wherein the input data comprises sample image data, sample object segmentation data, a sample GT, a sample object space adjacency matrix and a data set class co-occurrence probability matrix file, extracting sample characteristics by using a pre-trained VGG-19 model, and combining the sample characteristics with a sample object mask to obtain object characteristics. Then, the network is subjected to iterative training, the optimal model training weight, namely a model parameter file, is stored, and the optimal segmentation precision reaches 0.907. The training mode is popular and accords with deep learning network model training of the patent.
As shown in fig. 4, in the structure diagram of the remote sensing image semantic segmentation model of the present embodiment, H and W represent height and width of an input image, D represents feature dimensions, N represents number of objects, VGG-19 is a convolutional neural network model, X represents object features, a represents a spatial adjacency matrix, M represents a class co-occurrence probability matrix, WGCN represents a graph convolution weight, and X _ new represents updated object features. The network model combines symbiotic knowledge with the graph neural network, the idea is novel, and the model has high implementability.
After the model training is finished, the results of the model training are verified by using 400 verification set samples, the model verification precision is 0.907, and various confusion matrixes in the verification set are as follows:
TABLE 1 confusion matrix
As can be seen from Table 1, the precision of the landslide verification caused by the earthquake disaster reaches 0.934, and the precision of the verification of several types of land objects (plowed land types and buildings) affected by the earthquake is very high, which shows that the method has good effect and provides a scientific and effective method for researching the influence analysis of the earthquake disaster.
The present solution also provides a computer-readable storage medium storing a computer program which, when executed by a processor, implements the steps of the method described above.
It should be understood that, the sequence numbers of the steps in the foregoing embodiments do not imply an execution sequence, and the execution sequence of each process should be determined by its function and inherent logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.
It will be apparent to those skilled in the art that, for convenience and brevity of description, only the above-mentioned division of the functional units and modules is illustrated, and in practical applications, the above-mentioned function distribution may be performed by different functional units and modules according to needs, that is, the internal structure of the apparatus is divided into different functional units or modules to perform all or part of the above-mentioned functions. Each functional unit and module in the embodiments may be integrated in one processing unit, or each unit may exist alone physically, or two or more units are integrated in one unit, and the integrated unit may be implemented in a form of hardware, or in a form of software functional unit. In addition, specific names of the functional units and modules are only for convenience of distinguishing from each other, and are not used for limiting the protection scope of the present application. The specific working processes of the units and modules in the system may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
In the above embodiments, the descriptions of the respective embodiments have respective emphasis, and reference may be made to the related descriptions of other embodiments for parts that are not described or illustrated in a certain embodiment.
Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
In the embodiments provided in the present application, it should be understood that the disclosed apparatus/terminal device and method may be implemented in other ways. For example, the above-described embodiments of the apparatus/terminal device are merely illustrative, and for example, the division of the modules or units is only one logical division, and there may be other divisions when actually implemented, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The integrated modules/units, if implemented in the form of software functional units and sold or used as separate products, may be stored in a computer readable storage medium. Based on such understanding, all or part of the flow in the method of the embodiments described above can be realized by a computer program, which can be stored in a computer-readable storage medium and can realize the steps of the embodiments of the methods described above when the computer program is executed by a processor. . Wherein the computer program comprises computer program code, which may be in the form of source code, object code, an executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, usb disk, removable hard disk, magnetic disk, optical disk, computer Memory, Read-Only Memory (ROM), Random Access Memory (RAM), electrical carrier wave signals, telecommunications signals, software distribution medium, and the like. It should be noted that the computer readable medium may contain content that is subject to appropriate increase or decrease as required by legislation and patent practice in jurisdictions, for example, in some jurisdictions, computer readable media does not include electrical carrier signals and telecommunications signals as is required by legislation and patent practice.
The above-mentioned embodiments are only used for illustrating the technical solutions of the present application, and not for limiting the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; such modifications and substitutions do not substantially depart from the spirit and scope of the embodiments of the present application and are intended to be included within the scope of the present application.
Claims (10)
1. A seismic disaster area remote sensing image interpretation method based on a graph transformation knowledge embedding algorithm is characterized by comprising the following steps:
acquiring remote sensing image data of the earthquake-stricken area;
inputting the remote sensing image data of the earthquake-stricken area into a remote sensing image semantic segmentation model obtained by pre-training;
outputting an interpretation result;
the pre-training process of the remote sensing image semantic segmentation model comprises the following steps:
obtaining a remote sensing image of a research area;
processing the remote sensing image to obtain sample image characteristics, sample object segmentation data, a sample GT, a sample space adjacency matrix and data set class co-occurrence probability matrix data;
constructing an initial remote sensing image semantic segmentation model;
training the initial remote sensing image semantic segmentation model based on sample image features, sample object segmentation data, a sample GT, a sample space adjacency matrix and a data set class co-occurrence probability matrix to obtain a remote sensing image semantic segmentation model;
the method for constructing the initial remote sensing image semantic segmentation model comprises the following steps:
computing the original graph structure G of a single sampleori；
Traversing and copying the nodes in the original graph based on the total category number, and obtaining the structure G of the original graphoriConverted into a knowledge hypergraph structure Gnew；
Carrying out node feature aggregation in a GCN model;
and cutting the knowledge hypergraph structure through the classifier to obtain an initial remote sensing image semantic segmentation model.
2. The method for interpreting the remote sensing image of the earthquake-stricken area based on the graph transformation knowledge embedding algorithm as claimed in claim 1, wherein the original graph structure GoriThe following formula was used for calculationObtaining:
Gori＝(Vori，Eori)
Vori＝{vi，i∈{1，…，N}}
Eori＝{eij，i，j∈{1，…，N}}
wherein, VoriIs a set of object nodes in a single sample generated by superpixel segmentation, EoriAs a set of contiguous relationships between object nodes, eij＝aij，aijIs the spatial adjacency relation of the node i and the node j obtained by the calculation of a sample spatial adjacency matrix, N is the number of objects in a single sample, viIs the superpixel object node i.
3. The method for interpreting earthquake-stricken area remote sensing images based on graph transformation knowledge embedding algorithm as claimed in claim 1, wherein the knowledge hypergraph structure GnewThe following formula is used for calculation:
Gnew＝(Vnew，Enew)
Vnew＝{viα，i∈{1，...，N}，α∈{1，...，C}}
Enew＝{e(iα，jβ)，i，j∈{1，...，N}，α，β∈{1，...，C}}
wherein, VnewIs the object node set of the knowledge hypergraph obtained by copying the nodes of the original graph for C times, wherein C is the total category number viαExpressed as inode as alpha class, EnewExpressed as a set of contiguous relationships between object nodes, e(iα，jβ)Is a node viαAnd node vjβV. of adjacent relationjβThe node denoted j is a β class.
4. The method for interpreting the earthquake-stricken area remote sensing image based on the graph transformation knowledge embedding algorithm according to claim 1, wherein the processing the remote sensing image comprises the following steps:
performing superpixel segmentation on the remote sensing image to obtain geographic segmentation object vector data;
GT labeling is carried out on the vector data of the geographic segmentation object according to a ground surface coverage classification system to obtain GT grid data;
and cutting the remote sensing image, the geographic segmentation object vector data and the GT raster data by using the sample pre-selection frame vector file to obtain a complete data set, wherein the complete data set comprises a plurality of samples, and each sample comprises a sample original image file, a sample object segmentation file and a sample GT file.
5. The method for interpreting the earthquake-stricken area remote sensing image based on the graph transformation knowledge embedding algorithm as claimed in claim 4, wherein the obtaining of the sample image features comprises:
constructing a VGG-19 convolutional neural network;
and inputting the sample original image file into a VGG-19 convolutional neural network to obtain sample image characteristics.
6. The method for interpreting the remote sensing image in the earthquake-stricken area based on the graph transformation knowledge embedding algorithm as recited in claim 4, wherein the sample space adjacency matrix is obtained by calculating the adjacency distance of the sample geographic segmentation objects pairwise, and the data set category co-occurrence probability matrix is obtained by counting the sample GT of the complete data set through a category co-occurrence probability statistical algorithm.
7. The method for interpreting the earthquake-stricken area remote sensing image based on the graph transformation knowledge embedding algorithm according to claim 1, characterized in that after the remote sensing image of a research area is obtained, the method further comprises preprocessing the remote sensing image, wherein the preprocessing comprises atmospheric correction, geometric correction, radiometric calibration and cutting operation.
8. The method for interpreting the earthquake-stricken area remote sensing image based on the graph transformation knowledge embedding algorithm as claimed in claim 1, is characterized in that before training the initial remote sensing image semantic segmentation model, the method further comprises data set division, wherein the data set is divided into a training set and a verification set, and the ratio of the number of samples of the training set to the number of samples of the verification set is 3: 1.
9. the method for interpreting the earthquake-stricken area remote sensing image based on the graph transformation knowledge embedding algorithm according to claim 1, characterized by further comprising setting model training parameters before training the initial remote sensing image semantic segmentation model, wherein the learning rate of model training is 0.0001, the batch _ size of single training is 1, the number of times of model training is 400, and the Dropout parameter is 0.6.
10. A computer-readable storage medium storing a computer program, characterized in that: the computer program when executed by a processor implementing the steps of the method according to any one of claims 1 to 9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110644598.8A CN113435268A (en) | 2021-06-09 | 2021-06-09 | Earthquake disaster area remote sensing image interpretation method based on graph transformation knowledge embedding algorithm |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110644598.8A CN113435268A (en) | 2021-06-09 | 2021-06-09 | Earthquake disaster area remote sensing image interpretation method based on graph transformation knowledge embedding algorithm |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113435268A true CN113435268A (en) | 2021-09-24 |
Family
ID=77755523
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110644598.8A Pending CN113435268A (en) | 2021-06-09 | 2021-06-09 | Earthquake disaster area remote sensing image interpretation method based on graph transformation knowledge embedding algorithm |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113435268A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115294489A (en) * | 2022-06-22 | 2022-11-04 | 太原理工大学 | Semantic segmentation method and system for disaster video data |
| CN116305461A (en) * | 2023-03-13 | 2023-06-23 | 清华大学 | Structure response calculation method, device, electronic equipment and storage medium |
| CN116524369A (en) * | 2023-04-18 | 2023-08-01 | 中国地质大学（武汉） | Remote sensing image segmentation model construction method and device and remote sensing image interpretation method |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111461258A (en) * | 2020-04-26 | 2020-07-28 | 武汉大学 | Remote sensing image scene classification method of coupling convolution neural network and graph convolution network |
| WO2020188119A1 (en) * | 2019-03-21 | 2020-09-24 | Kepler Vision Technologies B.V. | A medical device for transcription of appearances in an image to text with machine learning |
| CN111738318A (en) * | 2020-06-11 | 2020-10-02 | 大连理工大学 | Super-large image classification method based on graph neural network |
| CN112241676A (en) * | 2020-07-07 | 2021-01-19 | 西北农林科技大学 | Method for automatically identifying terrain sundries |
| CN112347970A (en) * | 2020-11-18 | 2021-02-09 | 江苏海洋大学 | Remote sensing image ground object identification method based on graph convolution neural network |
| CN112766280A (en) * | 2021-01-16 | 2021-05-07 | 北京工业大学 | Remote sensing image road extraction method based on graph convolution |
-
2021
- 2021-06-09 CN CN202110644598.8A patent/CN113435268A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2020188119A1 (en) * | 2019-03-21 | 2020-09-24 | Kepler Vision Technologies B.V. | A medical device for transcription of appearances in an image to text with machine learning |
| CN111461258A (en) * | 2020-04-26 | 2020-07-28 | 武汉大学 | Remote sensing image scene classification method of coupling convolution neural network and graph convolution network |
| CN111738318A (en) * | 2020-06-11 | 2020-10-02 | 大连理工大学 | Super-large image classification method based on graph neural network |
| CN112241676A (en) * | 2020-07-07 | 2021-01-19 | 西北农林科技大学 | Method for automatically identifying terrain sundries |
| CN112347970A (en) * | 2020-11-18 | 2021-02-09 | 江苏海洋大学 | Remote sensing image ground object identification method based on graph convolution neural network |
| CN112766280A (en) * | 2021-01-16 | 2021-05-07 | 北京工业大学 | Remote sensing image road extraction method based on graph convolution |
Non-Patent Citations (1)
| Title |
|---|
| WEI CUI ET AL.: "Knowledge and Geo-Object Based Graph Convolutional Network for Remote Sensing Semantic Segmentation", 《SENSORS》 * |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115294489A (en) * | 2022-06-22 | 2022-11-04 | 太原理工大学 | Semantic segmentation method and system for disaster video data |
| CN115294489B (en) * | 2022-06-22 | 2023-06-09 | 太原理工大学 | Semantic segmentation method and system for disaster video data |
| CN116305461A (en) * | 2023-03-13 | 2023-06-23 | 清华大学 | Structure response calculation method, device, electronic equipment and storage medium |
| CN116305461B (en) * | 2023-03-13 | 2023-10-13 | 清华大学 | Structure response calculation method, device, electronic equipment and storage medium |
| CN116524369A (en) * | 2023-04-18 | 2023-08-01 | 中国地质大学（武汉） | Remote sensing image segmentation model construction method and device and remote sensing image interpretation method |
| CN116524369B (en) * | 2023-04-18 | 2023-11-17 | 中国地质大学（武汉） | Remote sensing image segmentation model construction method and device and remote sensing image interpretation method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Lv et al. | Spatial–spectral attention network guided with change magnitude image for land cover change detection using remote sensing images | |
| Lobry et al. | RSVQA: Visual question answering for remote sensing data | |
| CN109829399B (en) | Vehicle-mounted road scene point cloud automatic classification method based on deep learning | |
| CN108985238B (en) | Impervious surface extraction method and system combining deep learning and semantic probability | |
| Tehrany et al. | A comparative assessment between object and pixel-based classification approaches for land use/land cover mapping using SPOT 5 imagery | |
| Abdollahi et al. | Improving road semantic segmentation using generative adversarial network | |
| Conolly et al. | Geographical information systems in archaeology | |
| CN110263717B (en) | Method for determining land utilization category of street view image | |
| CN113435268A (en) | Earthquake disaster area remote sensing image interpretation method based on graph transformation knowledge embedding algorithm | |
| CN111626947B (en) | Map vectorization sample enhancement method and system based on generation of countermeasure network | |
| Li et al. | Geospatial technology for earth observation | |
| Wang et al. | Using GF-2 imagery and the conditional random field model for urban forest cover mapping | |
| Cai et al. | A comparison of object-based and contextual pixel-based classifications using high and medium spatial resolution images | |
| CN113223042B (en) | Intelligent acquisition method and equipment for remote sensing image deep learning sample | |
| CN113988147B (en) | Multi-label classification method and device for remote sensing image scene based on graph network, and multi-label retrieval method and device | |
| Guo et al. | Using multi-scale and hierarchical deep convolutional features for 3D semantic classification of TLS point clouds | |
| CN109657728B (en) | Sample production method and model training method | |
| Feng et al. | Impervious surface extraction based on different methods from multiple spatial resolution images: a comprehensive comparison | |
| Gervasoni et al. | Convolutional neural networks for disaggregated population mapping using open data | |
| CN113936214A (en) | Karst wetland vegetation community classification method based on fusion of aerospace remote sensing images | |
| Tarsha Kurdi et al. | Automatic evaluation and improvement of roof segments for modelling missing details using Lidar data | |
| Swetanisha et al. | Land use/land cover classification using machine learning models. | |
| Song et al. | Small UAV-based multi-temporal change detection for monitoring cultivated land cover changes in mountainous terrain | |
| Marvaniya et al. | Small, sparse, but substantial: techniques for segmenting small agricultural fields using sparse ground data | |
| Ruiz-Lendínez et al. | Deep learning methods applied to digital elevation models: state of the art |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210924 |
|
| RJ01 | Rejection of invention patent application after publication |