CN113469186B - Cross-domain migration image segmentation method based on small number of point labels - Google Patents
Cross-domain migration image segmentation method based on small number of point labels Download PDFInfo
- Publication number
- CN113469186B CN113469186B CN202110734847.2A CN202110734847A CN113469186B CN 113469186 B CN113469186 B CN 113469186B CN 202110734847 A CN202110734847 A CN 202110734847A CN 113469186 B CN113469186 B CN 113469186B
- Authority
- CN
- China
- Prior art keywords
- domain
- network
- image
- segmentation
- sub
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 50
- 238000013508 migration Methods 0.000 title claims abstract description 50
- 230000005012 migration Effects 0.000 title claims abstract description 50
- 238000003709 image segmentation Methods 0.000 title claims abstract description 45
- 230000011218 segmentation Effects 0.000 claims abstract description 81
- 230000006870 function Effects 0.000 claims description 35
- 238000012549 training Methods 0.000 claims description 33
- 238000013139 quantization Methods 0.000 claims description 30
- 238000002372 labelling Methods 0.000 claims description 13
- 238000005457 optimization Methods 0.000 claims description 12
- 210000003470 mitochondria Anatomy 0.000 claims description 11
- 230000003044 adaptive effect Effects 0.000 claims description 10
- 238000000638 solvent extraction Methods 0.000 claims description 8
- 238000000605 extraction Methods 0.000 claims description 5
- 239000003550 marker Substances 0.000 claims description 4
- 238000011176 pooling Methods 0.000 claims description 4
- 238000010586 diagram Methods 0.000 claims description 2
- 230000000694 effects Effects 0.000 abstract description 10
- 238000012545 processing Methods 0.000 abstract description 4
- 230000008485 antagonism Effects 0.000 abstract 1
- 230000002860 competitive effect Effects 0.000 abstract 1
- 230000002438 mitochondrial effect Effects 0.000 description 8
- 238000012360 testing method Methods 0.000 description 6
- 238000007781 pre-processing Methods 0.000 description 4
- 241000255581 Drosophila <fruit fly, genus> Species 0.000 description 3
- 238000013461 design Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000000971 hippocampal effect Effects 0.000 description 3
- 238000005070 sampling Methods 0.000 description 3
- 238000012795 verification Methods 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 2
- 238000000354 decomposition reaction Methods 0.000 description 2
- 238000013135 deep learning Methods 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 230000009466 transformation Effects 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 description 1
- 230000006978 adaptation Effects 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 210000004027 cell Anatomy 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 230000000052 comparative effect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 238000000635 electron micrograph Methods 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 210000001320 hippocampus Anatomy 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000005192 partition Methods 0.000 description 1
- 238000011002 quantification Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000012216 screening Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N10/00—Quantum computing, i.e. information processing based on quantum-mechanical phenomena
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Computing Systems (AREA)
- Evolutionary Computation (AREA)
- Data Mining & Analysis (AREA)
- Artificial Intelligence (AREA)
- Software Systems (AREA)
- Mathematical Physics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computational Mathematics (AREA)
- Pure & Applied Mathematics (AREA)
- Mathematical Optimization (AREA)
- Mathematical Analysis (AREA)
- Condensed Matter Physics & Semiconductors (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a cross-domain migration image segmentation method based on a small number of point labels, and belongs to the technical field of image processing. In the multi-target segmentation task of the image, only a small amount of target center points in target domain data are required to be marked, then under the guidance of a model trained by source domain data, the target domain data marked by a small amount of points are subjected to segmentation prediction task, space counting task and quantized counting task learning, and the model is enabled to learn to be represented by the characteristic of target domain discrimination by utilizing an antagonism network on an output space, so that the cross-domain segmentation effect of the target domain is improved, and an image automatic segmentation model with competitiveness compared with an unsupervised model is obtained. The method disclosed by the invention has the advantages that on the new application scene data, only the existing marking data and a small amount of point marks on the new application scene data are needed, so that the labor cost is greatly reduced, the competitive segmentation effect is obtained, and the method can be applied to multi-object segmentation in the fields of natural scene images, medical images and the like.
Description
Technical Field
The invention belongs to the technical field of image processing, relates to a method for dividing multiple targets in an image, and particularly relates to a cross-domain migration image automatic dividing method based on a small number of point labels.
Background
Semantic segmentation of a large number of small objects in an image is one of the widely applied technologies in the field of image processing, and extraction of semantic information of the image is achieved by classifying the image pixel by pixel. With the development of the deep learning technology, the performance of the full-supervision-based deep image segmentation technology is greatly improved. But the fully supervised approach based on deep learning requires a large number of pixel level labels. However, in the actual application scene, the pixel-level labeling of the targets one by one requires a great deal of manpower and material resources, and the labeling cost is high.
For such problems, researchers have proposed an unsupervised domain adaptive segmentation method, which applies model migration trained by the marked data domain in the existing application scene to the new unmarked data domain. For example, the patent application number is CN 111402257A, named as "automatic segmentation method of medical image based on multi-task system cross-domain migration", the geometric clues of the label domain and the visual clues of the image domain are integrated to guide the domain self-adaptation to obtain good segmentation effect, but the segmentation performance of the cross-domain segmentation model learned by the method is limited due to the large data distribution difference of different data domains and complete lack of new data domain marking information. In addition to the high-cost pixel-level marking, in the research work of some weak supervision segmentation methods, marking information such as center point marking, bounding box marking, graffiti marking and the like can greatly reduce marking cost while obtaining good segmentation effect. However, when the number of the targets in the massive image data and the images is too large, the weak labeling of each target is still a low labeling cost, and the labeling of a small number of targets can greatly reduce the labeling cost on the basis of providing a small amount of accurate labeling information. Therefore, a small amount of weak marks on a new target data domain are utilized to provide real discrimination information for the target domain data, the segmentation model trained by the source domain data with the marks can be guided to be better migrated and generalized to the target domain, and the segmentation precision of the cross-domain segmentation model can be improved under the condition of a small amount of marking cost.
Disclosure of Invention
In order to solve some problems of the cross-domain image semantic segmentation method mentioned in the background art, the invention provides a cross-domain migration image segmentation method based on a small amount of point labels, which only carries out center point labels on partial targets on a target domain data set with multiple targets, and then realizes semantic segmentation on target domain image data by utilizing a training model which is similar to a target domain image but from different scenes and has full labels, namely a source domain. The invention designs the space counting sub-network and the quantization counting sub-network by considering general source domain data knowledge, such as position and quantity information, and the like, positions the split targets and makes constraint on the quantity of the split targets, and on the output space, based on the idea of countermeasure learning, the domain discriminator is utilized to assist the split network to have better split effect on the target domain data.
The invention adopts the following technical scheme:
a cross-domain migration image segmentation method based on a small number of point labels comprises the following steps:
s1, pre-training a cross-domain migration image segmentation model by using source domain data, wherein the method comprises the following steps:
s11, preprocessing source domain image data;
s12, constructing a cross-domain migration image segmentation model;
preferably, the cross-domain migration image segmentation model is divided into a semantic segmentation sub-network, a space counting sub-network and a quantization counting sub-network, and the quantization counting sub-network parameters are not optimized on the source domain data;
s13, designing a loss function;
mapping source domainThe image is denoted as x s Its corresponding pixel level label is y s The corresponding point mark graph is r s The number of points is recorded as T s 。p s For source domain image x s Segmentation prediction result through semantic segmentation sub-network, q s For source domain image x s Through the prediction results of the space counting sub-network,
count the prediction of the subnetwork for quantification, +.>
And c represents the category, and K is the total number of image pixel points. The semantic segmentation sub-network optimization objective loss function under the source domain data is as follows:
the space counting sub-network optimization objective loss function under the source domain data is as follows:
wherein,,
for variance sigma 1 G is a Gaussian function of (g) s Marking the graph r for points s Corresponding Gaussian point marker map, +.>
Representing a pixel value corresponding to a source domain Gaussian point marked image pixel i; weight map->
For variance sigma 2 Is a Gaussian function of->
Representing a weight map beta s A weight value corresponding to the middle pixel i; />
Representing a network prediction output result corresponding to the source domain image pixel i; λ is a weight parameter, and K is the total number of image pixels;
s14, pre-training a cross-domain migration image segmentation model by using the data set and the loss function;
and S15, saving the cross-domain migration image segmentation model parameters for parameter initialization of a training model on the subsequent target domain data.
S2, training a quantitative counting model.
Preferably, the quantitative counting model is initialized by using the characteristic extraction network parameters of the cross-domain migration image segmentation model;
s21, preprocessing a source domain image dataset;
s22, constructing a quantitative counting model;
preferably, the quantization counting model has the same structure as the semantic segmentation sub-network, and the last single-channel prediction graph obtained by up-sampling is subjected to a self-adaptive average pooling layer to obtain the last prediction target number;
s23, designing a loss function;
the mean square error is used as an optimization objective function for the target number prediction output as follows:
s24, training a quantization count model by utilizing the data set and the corresponding loss function;
and S25, storing the network parameter model for estimating the mitochondrial quantity of the target domain.
S3, training a cross-domain migration image segmentation model based on a small number of point marks on the target domain image dataset with the small number of point marks.
Preferably, the cross-domain migration image segmentation model based on a small number of point labels simultaneously uses a source domain image data set, a source domain image complete label, a target domain image data set and a small number of point labels of target domain data to perform parameter optimization.
S31, preprocessing a source domain data set and a target domain data set;
s32, designing a cross-domain migration image segmentation model based on a small number of point labels;
s33, designing corresponding loss functions aiming at different sub-networks;
s34, training and optimizing a cross-domain migration image segmentation model by using the data set and the loss function;
s35, saving the generated cross-domain migration image segmentation model for segmentation prediction on a target domain;
further, the cross-domain migration image segmentation model of step S32 includes three sub-networks sharing a feature extraction network; acquiring semantic information of an image through a semantic segmentation sub-network, and outputting a prediction graph with the same resolution as that of the input of two channels; the space counting sub-network and the semantic segmentation sub-network have the same network structure, and output single-channel predictive pictures with the same resolution as input are used for positioning the space position of a target; a quantization count sub-network, which takes the output of the space count sub-network as input to predict the number of targets; further, although the source domain data and the target domain data are different from each other, they have a large similarity in the tag space, and thus it is considered to use a domain arbiter on the output space to make a weak shape constraint on the output space.
Preferably, the present invention constructs a domain arbiter having inputs from the source domain split prediction tag and the target domain split prediction tag, the output image being the same size as the input, each pixel of the output image representing whether the pixel is from the source domain prediction tag or the target domain prediction tag.
Further, corresponding loss functions are designed for different sub-networks.
The source domain and the target domain data are simultaneously used in the cross-domain migration image segmentation model training, and a loss function adopted for the output of the source domain data through the network is the same as that in the step S13.
Considering that only a small amount of point labeling information is on the target domain, the space count loss function on the target domain is defined as follows:
preferably, the loss function introduces a weight map w, each value w of the weight map i The definition is as follows:
wherein h is t In addition, as the target domain label has a small amount of foreground labeling information, background information, m, needs to be gradually added from the segmentation prediction graph according to the iteration times by a method of setting a soft threshold value t A representation of a middle pixel value of 0 is a background pixel; the soft threshold value changes with the iteration number of the model, m t Each value of (3)
The definition is as follows:
ρ is defined as follows:
where ε is the total number of iterations of the model and ε is the current number of iterations of the model.
Preferably, the loss of the arbiter employs cross entropy loss as follows:
wherein D is pred Representation domain arbiter, p s And p t Respectively source domain images x s And a target domain image x t And dividing the prediction result through the semantic dividing sub-network.
Preferably, the arbiter network parameters are fixed to optimize the semantic segmentation sub-network by minimizing the following loss function, as shown below:
preferably, the number of mitochondria in the target domain is constrained by a source domain pre-trained quantitative count model. Predicting mitochondrial numbers of target domains using a quantized count model trained on source domain data
Because the source domain and the target domain have certain domain differences, the quantization counting model trained on the source domain has certain deviation. Taking the error as epsilon, obtaining the target domain mitochondrial prediction quantity +.>
Is constrained within a certain range, as shown in the following formula:
this loss may place a constraint on the number of mitochondria in the target domain.
In summary, the cross-domain semantic segmentation sub-network is obtained by minimizing the following objective function, namely
Wherein lambda is pred Representing a preset first non-negative super parameter; lambda (lambda) count Representing a preset second non-negative super-parameter.
As can be seen from the above description of the present invention, compared with the prior art, the present invention has the following advantages:
(1) According to the method, only part of target center points are required to be marked on each picture with multiple targets in the target domain, so that marking work becomes efficient and simple;
(2) Based on general domain knowledge of data to be segmented, such as spatial position information and quantity information of targets, the invention provides a spatial counting sub-network and a quantitative counting sub-network, wherein the spatial counting sub-network assists a model to better position the targets, the quantitative counting sub-network performs quantity constraint on the targets predicted by the model, and the two sub-networks are combined to optimize the parameters of the segmentation prediction network and assist in better segmentation of the cross-domain model;
(3) According to the invention, a domain discriminator is introduced into the output space, weak shape constraint is carried out on the predicted image of the target domain, so that the model is helped to learn the characteristic representation with discrimination on the target domain, and the segmentation effect is further improved;
drawings
FIG. 1 is a block diagram of a cross-domain migration image segmentation method based on a small number of point labels;
FIG. 2 is a flow chart of the method of the present invention;
FIG. 3 is a graph of segmentation results of the method of the present invention and other segmentation methods; (a) is a target domain image and a standard segmentation map thereof; (b) a domain-free adaptive segmentation result graph; (c) dividing a result graph for the DAMT-Net method; (d) is a segmentation result graph of the present invention.
Detailed Description
The invention is further described below by means of specific embodiments. It should be noted that the specific examples described herein are for convenience of description and explanation of the specific embodiments of the present invention and are not intended to limit the present invention.
The method provided by the invention can be applied to the segmentation of multiple targets, such as the segmentation of cells and mitochondria in the medical field. The disclosed medical image mitochondrial dataset is used below as a specific example. Specifically, the source domain dataset is a third age drosophila dataset and the target domain dataset is a mouse hippocampal dataset. The third age drosophila dataset is composed of 20 1024×1024 pictures, and has 389 mitochondria in total, 10 are randomly selected as training sets during training, and the rest 10 are test sets; the mouse hippocampal dataset included a training set consisting of 165 images of 1024×768 for 2511 mitochondria and a test set consisting of 165 images of 1024×768 for 2623 mitochondria, and 15% of the mitochondria of the mouse hippocampal dataset were used as label information in the following specific examples, namely, 377 mitochondrial center markers were randomly selected.
Referring to fig. 1 and fig. 2, the invention discloses a cross-domain migration image segmentation method based on a small number of point labels, which specifically comprises the following steps:
and step 10, preprocessing data.
Randomly cutting image data of training data and corresponding label data into 512×512 size, normalizing the image data, inputting the normalized image data as network, setting foreground pixel value of the label data as 1, and setting background pixel value as 0; performing Gaussian blur on a point tag image on a tag data set of a training set to obtain a pseudo tag as a real tag of a training space counting branch, wherein the Gaussian blur parameter sigma 1 Set to 61;
and step 20, initializing parameters.
Random seeds during fixed network training ensure that training results of the same experimental model are consistent all the time; setting the iteration number of training loops to 10 ten thousand times, using an SGD (generalized discrete Fourier transform) optimizer for training a cross-domain model, and setting the initial learning rate to 10 -4 Momentum parameter of 0.9 and weight attenuation coefficient of 5×10 -5 The method comprises the steps of carrying out a first treatment on the surface of the Training a discriminant model, using an Adam optimizer, setting an initial learning rate to 5×10 -5 The polynomial attenuation coefficient is 0.9;
step 30, training a cross-domain migration image segmentation network on the source domain data.
Step 301, defining a network structure. The defined split prediction network uses coding and decoding branches of CS-Net, see in particular (Peng J, luo Z. CS-Net: instance-aware cellular segmentation with hierarchical dimension-decomposed convolutions and slice-attentive learning [ J ]. ArXiv preprint arXiv:2101.02877,2021.). Other similar networks may be used in implementations. The CS-Net uses a hierarchical dimension decomposition convolution module to extract multi-level upper and lower level information, so that characteristics can be better extracted and the model is lighter. The invention designs and adds the space counting sub-network and the quantization counting sub-network on the basis of the space counting sub-network, the three sub-networks share the characteristic extraction network, the output of the space counting sub-network in the fourth up-sampling of CS-Net is used as input, and the space counting structure is the same as the last up-sampling network structure. Finally, a convolution with a channel of 1 multiplied by 1 is used to obtain the output with the same resolution as the original image of the final single channel; the quantization counting sub-network output layer uses two hierarchical dimension decomposition convolution modules and a full connection layer, takes the output of a space counting branch as the input of the space counting sub-network output layer, and outputs the predicted number;
step 302, inputting the preprocessed source domain training data into a network, performing online data enhancement on the image data and the corresponding tag data in order to reduce the overfitting, such as random inversion, random brightness transformation, motion blur, and the like, wherein the motion blur and the brightness transformation are applied to the image data, are not applied to the tag data, and undergo semantic divisionDividing sub-network and space counting sub-network to obtain dividing prediction output map of source domain data s Space count map q s ；
Step 303, calculating a segmentation prediction map p output by the semantic segmentation sub-network s With source domain pixel level label map y s The calculation formula is as follows:
step 304, calculating the output q from the space counting sub-network s Gaussian point marker graph g of source domain s The spatial counting sub-network optimization objective loss function is as follows:
empirically, λ is set to 3;
step 305 by minimizing
Optimizing network parameters; empirically, lambda count Setting equal to 0.1;
step 306, saving the best parameters of the cross-domain migration image segmentation network model according to the test result of the verification set; specifically, the verification set is verified every 1000 iterations during model training, and network model parameters are saved according to the best JAC (Jaccard coefficient) result;
step 40, training a quantization count model on the source domain data, specifically comprising the following steps:
step 401, defining a quantitative counting network model; the quantization counting model has the same network structure as the semantic segmentation sub-network, the last single channel output of the model passes through an adaptive average pooling layer, the output size of the adaptive average pooling layer is set to be 1, and the final scalar value output is obtained
Step 402, inputting the processed source domain training data into a quantization count model to obtain a target predicted value of the source domain data
The optimization objective function is as follows:
step 403, back propagation update parameter size;
step 404, storing the best quantization count model parameters according to the test result of the verification set every 1000 iterations;
step 50, training a cross-domain migration image segmentation model based on a small number of point marks on a target domain image dataset with the small number of point marks, wherein the specific steps are as follows:
step 501, defining a cross-domain migration image segmentation model, wherein the model structure is as described in step 301; defining a quantized count model, the network structure being as described in step 401;
step 502, initializing parameters of the cross-domain migration image segmentation model by using the network parameters stored in step 306, initializing the quantization count model by using the network parameters stored in step 404, and fixing the parameters;
step 503, fixing the parameters of the arbiter network when the network starts training;
step 504, initializing the original domain label of the target domain data to be 0, and initializing the original domain label of the source domain data to be 1;
step 505, inputting the source domain data into the semantic segmentation sub-network to obtain a segmentation prediction output p s ；
Step 506, calculating a segmentation loss by the following formula;
step 507, inputting the source domain data into a space counting sub-network to obtain a space counting prediction output q s ；
Step 508, calculating a mean square error loss according to the following formula;
where λ is a weight parameter, empirically, λ=3 is set;
step 509, inputting the target domain data into the semantic segmentation sub-network to obtain a segmentation prediction output p t ；
Since there is no pixel level tag in the target domain, the loss cannot be directly calculated, but the resulting p t May be used to pick background pixels;
step 510, inputting the target domain data into the space counting sub-network to obtain space counting prediction output q t ；
Step 511, calculating the loss according to the following function;
w is a weight map varying with the number of iterations, from the partition prediction map p t Screening background pixels according to a threshold value rho upper Setting to 0.7, as described in step S33, by giving a larger loss weight to the area with a small number of marked points, λ=3 is set in this embodiment, that is, the loss weight of the area within the radius of 11 pixels with the marked points is 6, the loss weight of the annular area with the radius of 31 pixels with the marked points is 3, the background area is set to 1, and the loss weights of other pixels are set to 0, and no loss calculation is performed;
step 512, constraining the number of mitochondria of the target domain according to a quantization count model trained on the source domain data using a loss function as follows:
where ε is the disturbance parameter, we will
Constrained at T t Is empirically set to 3, T t Target domain mitochondrial number predicted for quantitative count model trained on source domain data,/for the target domain mitochondrial number predicted for quantitative count model trained on source domain data>
Counting the number of mitochondrial predictions obtained by the branches for quantization of the cross-domain model;
step 513, training the domain discriminator and updating the parameters of the domain discriminator;
step 514, fixing domain arbiter network parameters;
step 515, partitioning the source domain data into prediction labels p s Target domain data division prediction tag p t Inputting the source domain data semantic domain identification label and the target domain data semantic domain identification label into a semantic domain identifier;
in step 516, the domain arbiter is implemented by minimizing the following objective functions:
wherein the method comprises the steps of
Representing mathematical expectations, where D pred Representation domain arbiter, p s And p t Respectively source domain images x s And a target domain image x t Segmentation prediction results through a semantic segmentation sub-network;
step 517, calculating and minimizing the following loss function values, and bringing the data feature distribution of the target domain data to the data feature distribution of the source data in the output space;
wherein the method comprises the steps of
Representing mathematical expectations, D pred Representing a domain-dividing arbiter, p t For the target domain image x t Segmentation prediction results through a semantic segmentation sub-network;
in combination with the above, the cross-domain migration image segmentation model is obtained by minimizing the objective function, i.e.
Specifically lambda count ＝0.1，λ pred ＝0.001；
Step 518, repeating steps 505 to 517, and alternately optimizing
And updating parameters by adopting a back propagation method to obtain a final cross-domain migration image segmentation model.
In order to illustrate the effectiveness of the method provided by the invention, corresponding performance evaluation is performed, a DSC (price coefficient) and JAC (Jaccard coefficient) index is adopted to compare the segmentation quality of a segmentation prediction result P of a network on a target domain test set with that of a real label G, and the better the DSC and JAC segmentation effects are, the definition of DSC and JAC is as follows:
as shown in table 1 and fig. 3, a comparative test result is obtained by using 15% point labeling and non-adaptive method NoAdapt (model trained on source domain is directly applied on target domain), representative non-supervised domain adaptive method DAMT-Net (domain adaptation is guided by integrating geometric cues of label domain and visual cues of image domain together in consideration of target domain non-label).
Table 1 comparison of the method of the invention and baseline method, unsupervised domain adaptive representative method
The DAMT-Net is described in detail in the literature "Peng J, yi J, yuan Z.ensupervised mitochondria segmentation in EM images via domain adaptive multi-task learning [ J ]. IEEE Journal of Selected Topics in Signal Processing,2020,14 (6): 1199-1209".
As can be seen from Table 1, the comparison of the JAC index shows that the JAC index of NoAdpt is only 54.4%, which means that the model trained on the third-age drosophila dataset cannot be well applied to the mouse hippocampus dataset, the JAC index of the representative unsupervised domain adaptive method DAMT-Net is only 60.8%, the JAC index of the method provided by the invention reaches 77.6%, only 15% of the center point mark is utilized to obtain an effect improvement of 23.2% compared with the NoAdpt, and the DAMT-Net method is utilized to obtain an effect improvement of 16.8%, which means that the segmentation effect can be greatly improved by introducing a small number of mark points, and the effectiveness of the method is also proved.
While the invention has been described with reference to a specific embodiment, the design concept of the invention is not limited thereto, and any insubstantial modification of the invention by this concept should be construed as infringement to the scope of the invention.
Claims (10)
1. A cross-domain migration image segmentation method based on a small amount of point labeling is characterized in that the method utilizes knowledge of an existing completely labeled source domain image and small amount of point labeling information of a target domain image to be segmented to realize high-performance segmentation of the target domain image, and specifically comprises the following steps:
s1, training a cross-domain migration image segmentation model on a source domain image data set with pixel-level marks;
s2, training a quantization counting model on a source domain image data set with pixel-level marks;
s3, training a cross-domain migration image segmentation model based on a small number of point labels by combining the source domain and the target domain image data sets, wherein the cross-domain migration image segmentation model is specifically as follows:
s31, inputting a small amount of point marked target domain images and complete pixel marked source domain images into a cross-domain migration image segmentation model pre-trained on source domain data to obtain corresponding segmentation output as input of a domain discriminator, discriminating whether a predicted image comes from a source domain or a target domain based on the idea of countermeasure learning, and optimizing cross-domain migration image segmentation model parameters by using the domain discriminator of an output space;
s32, inputting a small amount of point marked target domain images into a pre-trained cross-domain migration image segmentation model on source domain data to obtain corresponding space counting output, picking out reliable pseudo background information by utilizing the output of a semantic segmentation sub-network, and combining a small amount of target center point pixels to learn a target domain space counting task;
s33, learning a quantization counting task on a target domain by using a quantization counting model of a source domain; fixing network parameters of a pre-trained quantitative counting model on source domain data, inputting a target domain image into a network to obtain target estimated quantity in the target domain image, and utilizing the quantitative counting model in the step S2 to make constraint on quantitative counting prediction on the target domain so as to learn quantitative counting tasks of the target domain;
and S34, saving model parameters of the cross-domain migration image segmentation model for segmentation prediction of the new target domain image.
2. The method for partitioning a cross-domain migration image based on a small number of point labels as claimed in claim 1, wherein the small number of point labels refer to labeling a center point of a small number of targets in a target domain image when performing multi-target partitioning in the image.
3. The cross-domain migration image segmentation method based on the small number of point labels according to claim 2, wherein the semantic segmentation sub-network, the space counting sub-network and the quantization counting sub-network form a cross-domain migration image segmentation model based on the small number of point labels; the semantic segmentation sub-network and the space counting sub-network share characteristics to extract network parameters; the semantic segmentation sub-network, the space counting sub-network and the quantization counting sub-network all adopt decoder-encoder structures, the semantic segmentation sub-network and the space counting sub-network respectively conduct image segmentation result prediction and space counting prediction, and the quantization counting sub-network conducts quantization counting task learning on space counting prediction output.
4. The method for partitioning a cross-domain migration image based on a small number of point labels according to claim 3, wherein the semantic partitioning sub-network, the space counting sub-network and the quantization counting sub-network are pre-trained by using a source domain image with pixel-level labels;
the semantic segmentation sub-network optimization objective loss function is as follows:
wherein,,
representing mathematical expectation, x s Representing a source domain image, y s For the pixel-level label corresponding to the source domain image, c represents the category, < >>
A pixel level label representing a class of source domain image c; />
The method comprises the steps that a source domain image is represented to obtain a c-class segmentation prediction result through a semantic segmentation sub-network;
the space counting sub-network optimization objective loss function is as follows:
wherein,,
for variance sigma 1 G is a Gaussian function of (g) s Marking the graph r for points s Corresponding Gaussian point marker map, +.>
Representing a pixel value corresponding to a source domain Gaussian point marked image pixel i; weight map->
For variance sigma 2 Is a Gaussian function of->
Representing a weight map beta s A weight value corresponding to the middle pixel i; />
Representing a network prediction output result corresponding to the source domain image pixel i; λ is a weight parameter, and K is the total number of image pixels;
the quantization count network optimization objective loss function is as follows:
wherein,,
representing mathematical expectation, T s Representing the true number of mitochondria,/->
Representing the predicted result of the source domain image passing through the quantization count network.
5. The cross-domain migration image segmentation method based on a small number of point labels according to claim 4, wherein the quantized count model trained in the source domain has the same network structure as the semantic segmentation sub-network, the feature extraction network shared by the semantic segmentation sub-network and the space count sub-network is used as an initialization parameter, the final output is obtained through an adaptive average pooling layer, and the final output is trained by using a corresponding target loss function.
6. The cross-domain migration image segmentation method based on a small number of point labels according to claim 5, wherein the semantic segmentation sub-network and the space counting sub-network use a pre-training model initialization parameter of a source domain, and then use a target domain image of a small number of point labels and a pixel-level label source domain image to train at the same time; the quantization count sub-network is trained using only the target domain image, and the source domain pre-trained quantization count model fixes its parameters as a separate number estimation model.
7. The cross-domain migration image segmentation method based on a small number of point marks as claimed in claim 6, wherein the target domain image with a small number of point marks and the source domain image with a complete pixel level mark are input into a cross-domain migration image segmentation model initialized by source domain pre-training to obtain segmentation predictions of the source domain and the target domain respectively, and input into a domain discriminatorIn (a) and (b); optimizing the semantic segmentation sub-network and the domain arbiter based on the countermeasure learning; representing a target domain image as x t ，p t For the target domain image x t The optimization target loss of the cross-domain migration image segmentation model is as follows through the segmentation prediction result of the semantic segmentation sub-network:
the domain arbiter optimization penalty is as follows:
wherein D is pred Representation domain arbiter, p s And p t Respectively source domain images x s And a target domain image x t Segmentation prediction results through a semantic segmentation sub-network; d (D) pred (p t ) The prediction output of the target domain segmentation prediction result passing through the discriminator is represented; d (D) pred (p s ) The prediction output of the source domain segmentation prediction result through the discriminator is shown.
8. The cross-domain migration image segmentation method based on a small number of point labels according to claim 7, wherein a target domain image of the small number of point labels is input into a source domain pre-trained space counting sub-network, a pseudo background and a real sparse point label are selected to learn a space counting task by combining a prediction output in a semantic segmentation sub-network, and a space counting loss function on the target domain is as follows:
wherein, gaussian point mark map
For variance sigma 1 Is a Gaussian function of r t Marking the graph for the center point, ">
G representing a Gaussian point marker graph t A pixel value corresponding to pixel i; weight map->
For variance sigma 2 Is a Gaussian function of->
Representing a weight map beta t A weight value corresponding to the pixel i; w is a weight diagram with balanced pseudo background and sparse foreground, and w is i Representing a weight value corresponding to a pixel i in the balance weight map w; />
A prediction output representing a target domain spatial count output image pixel i; k represents the total number of image pixel points; λ is the weight parameter.
9. The method for partitioning a cross-domain migration image based on a small number of point labels according to claim 8, wherein the quantitative estimation of the number of images in the target domain using the quantitative counting model in the source domain is denoted as T t The quantized count sub-network prediction output of the target domain is noted as
Will T t Reference estimate pair as target field +.>
Number of runsConstraint, quantization counting task learning of the target domain is carried out, and the optimization target loss function is as follows:
wherein ε represents the disturbance parameter, we will
Constrained at T t Is within the disturbance range of (2).
10. The method for partitioning a cross-domain migration image based on a small number of point labels according to claim 9, wherein said semantic partitioning sub-network is obtained by minimizing an objective function of
Wherein lambda is pred Representing a preset first non-negative super parameter; lambda (lambda) count Representing a preset second non-negative super-parameter.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110734847.2A CN113469186B (en) | 2021-06-30 | 2021-06-30 | Cross-domain migration image segmentation method based on small number of point labels |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110734847.2A CN113469186B (en) | 2021-06-30 | 2021-06-30 | Cross-domain migration image segmentation method based on small number of point labels |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113469186A CN113469186A (en) | 2021-10-01 |
| CN113469186B true CN113469186B (en) | 2023-06-23 |
Family
ID=77874346
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110734847.2A Active CN113469186B (en) | 2021-06-30 | 2021-06-30 | Cross-domain migration image segmentation method based on small number of point labels |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113469186B (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114240955B (en) * | 2021-12-22 | 2023-04-07 | 电子科技大学 | Semi-supervised cross-domain self-adaptive image segmentation method |
| CN114239753B (en) * | 2022-02-23 | 2022-07-22 | 山东力聚机器人科技股份有限公司 | Migratable image identification method and device |
| CN114638964A (en) * | 2022-03-07 | 2022-06-17 | 厦门大学 | Cross-domain three-dimensional point cloud segmentation method based on deep learning and storage medium |
| CN114445413B (en) * | 2022-04-07 | 2022-09-06 | 宁波康达凯能医疗科技有限公司 | Inter-frame image semantic segmentation method and system based on domain self-adaptation |
| CN117893988B (en) * | 2024-01-19 | 2024-06-18 | 元橡科技（北京）有限公司 | All-terrain scene pavement recognition method and training method |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111275713A (en) * | 2020-02-03 | 2020-06-12 | 武汉大学 | Cross-domain semantic segmentation method based on countermeasure self-integration network |
| CN111402257A (en) * | 2020-03-11 | 2020-07-10 | 华侨大学 | Medical image automatic segmentation method based on multi-task collaborative cross-domain migration |
| AU2020103905A4 (en) * | 2020-12-04 | 2021-02-11 | Chongqing Normal University | Unsupervised cross-domain self-adaptive medical image segmentation method based on deep adversarial learning |
| CN112819873A (en) * | 2021-02-05 | 2021-05-18 | 四川大学 | High-generalization cross-domain road scene semantic segmentation method and system |
-
2021
- 2021-06-30 CN CN202110734847.2A patent/CN113469186B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111275713A (en) * | 2020-02-03 | 2020-06-12 | 武汉大学 | Cross-domain semantic segmentation method based on countermeasure self-integration network |
| CN111402257A (en) * | 2020-03-11 | 2020-07-10 | 华侨大学 | Medical image automatic segmentation method based on multi-task collaborative cross-domain migration |
| AU2020103905A4 (en) * | 2020-12-04 | 2021-02-11 | Chongqing Normal University | Unsupervised cross-domain self-adaptive medical image segmentation method based on deep adversarial learning |
| CN112819873A (en) * | 2021-02-05 | 2021-05-18 | 四川大学 | High-generalization cross-domain road scene semantic segmentation method and system |
Non-Patent Citations (3)
| Title |
|---|
| 基于序列间先验约束和多视角信息融合的肝脏CT图像分割;彭佳林;《电子与信息学报》;全文 * |
| 基于深度学习的遥感影像去雾与舰船目标检测;候伟宁;《中国优秀硕士学位论文全文数据库 (工程科技Ⅱ辑)》;全文 * |
| 基于类别相关的领域自适应交通图像语义分割方法;贾颖霞;郎丛妍;冯松鹤;;计算机研究与发展(04);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113469186A (en) | 2021-10-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113469186B (en) | Cross-domain migration image segmentation method based on small number of point labels | |
| CN108256561B (en) | Multi-source domain adaptive migration method and system based on counterstudy | |
| CN111814854B (en) | Target re-identification method without supervision domain adaptation | |
| Fan et al. | Watching a small portion could be as good as watching all: Towards efficient video classification | |
| CN113326731B (en) | Cross-domain pedestrian re-identification method based on momentum network guidance | |
| CN110956185A (en) | Method for detecting image salient object | |
| CN111402257B (en) | Automatic medical image segmentation method based on multi-task collaborative cross-domain migration | |
| CN113128478B (en) | Model training method, pedestrian analysis method, device, equipment and storage medium | |
| CN112132014B (en) | Target re-identification method and system based on non-supervised pyramid similarity learning | |
| Zhukov et al. | Learning actionness via long-range temporal order verification | |
| CN114139676A (en) | Training method of domain adaptive neural network | |
| CN112052818A (en) | Unsupervised domain adaptive pedestrian detection method, unsupervised domain adaptive pedestrian detection system and storage medium | |
| CN114863091A (en) | Target detection training method based on pseudo label | |
| CN115439685A (en) | Small sample image data set dividing method and computer readable storage medium | |
| US11829442B2 (en) | Methods and systems for efficient batch active learning of a deep neural network | |
| Sood et al. | Neunets: An automated synthesis engine for neural network design | |
| CN116563602A (en) | Fine granularity image classification model training method based on category-level soft target supervision | |
| CN114495114B (en) | Text sequence recognition model calibration method based on CTC decoder | |
| CN112766354B (en) | Knowledge-graph-based small sample picture identification method and system | |
| Zhu et al. | Boosted cross-domain dictionary learning for visual categorization | |
| Germi et al. | Enhanced data-recalibration: utilizing validation data to mitigate instance-dependent noise in classification | |
| CN117648623B (en) | Network classification algorithm based on pooling comparison learning | |
| Samplawski et al. | Learning Graph-Based Priors for Generalized Zero-Shot Learning | |
| CN114187510B (en) | Small sample remote sensing scene classification method based on metanuclear network | |
| CN113688229B (en) | Text recommendation method, system, storage medium and equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |