Disclosure of Invention
In view of the above, the present application provides an image segmentation method and apparatus, so as to facilitate the segmentation of a corresponding object in an image to be segmented by a small amount of object data, with fast speed, good robustness and strong adaptability.
The embodiment of the application provides an image segmentation method, which comprises the following steps:
encoding the query image and the indication image corresponding to the query image by using the same encoding mode, wherein the indication image and the query image contain the same kind of target objects;
extracting first image features of the coded query image and extracting second image features of the coded indication image;
based on the first image features and the second image features, performing feature comparison on the query image and the indication image, and determining a feature distribution matrix of a target object in the indication image in the query image;
and decoding according to the characteristic distribution matrix and the first image characteristic to obtain a segmented image of the target object segmented from the query image.
Further, the determining the feature distribution matrix of the target object in the indication image in the query image based on the first image feature and the second image feature by comparing the features of the query image and the indication image includes:
determining a feature distribution map of the query image based on the first image features;
determining a feature vector of the indication image based on the second image feature;
and determining a feature distribution matrix of the indication image based on the feature distribution map of the query image and the image feature vector of the indication image.
Further, the determining, based on the second image feature, a feature vector of the indication image includes:
generating a feature distribution map of the indication image based on the second image feature;
and carrying out maximum pooling treatment on the characteristic distribution diagram of the indication image to obtain the characteristic vector of the indication image.
Further, the determining the feature distribution matrix of the indication image based on the feature distribution map of the query image and the feature vector of the indication image includes:
performing inner product processing on the feature vector of the indication image and the feature vector at each position in the feature distribution diagram of the query image;
and normalizing the result of the inner product processing to obtain the characteristic distribution matrix of the indication image.
Further, the decoding process according to the feature distribution matrix and the first image feature, to obtain a segmented image of the target object segmented from the query image, includes:
performing product processing on the characteristic distribution matrix and the second image characteristic;
the result after the product processing is connected in parallel with the first image feature and then is decoded;
and determining a segmented image of the target object obtained by carrying out image segmentation on the query image based on the result after the decoding processing.
An embodiment of the present application provides an image segmentation apparatus including:
the coding module is used for respectively coding the query image and the indication image corresponding to the query image in the same coding mode, wherein the indication image and the query image contain the same type of target objects;
the extraction module is used for extracting the first image characteristics of the coded query image and extracting the second image characteristics of the coded indication image;
the comparison module is used for comparing the features of the query image and the indication image based on the first image features and the second image features, and determining a feature distribution matrix of a target object in the indication image in the query image;
and the decoding module is used for carrying out decoding processing according to the characteristic distribution matrix and the first image characteristic to obtain a segmented image of the target object segmented from the query image.
Further, the comparison module includes:
a first determining unit configured to determine a feature distribution map of the query image based on the first image feature;
a second determining unit configured to determine a feature vector of the instruction image based on the second image feature;
and a third determining unit, configured to determine a feature distribution matrix of the indication image based on a feature distribution map of the query image and an image feature vector of the indication image.
Further, the second determining unit is specifically configured to:
generating a feature distribution map of the indication image based on the second image feature;
and carrying out maximum pooling treatment on the characteristic distribution diagram of the indication image to obtain the characteristic vector of the indication image.
Further, the third determining unit is specifically configured to:
performing inner product processing on the feature vector of the indication image and the feature vector at each position in the feature distribution diagram of the query image;
and normalizing the result of the inner product processing to obtain the characteristic distribution matrix of the indication image.
Further, the decoding module includes:
the first processing unit is used for performing product processing on the characteristic distribution matrix and the second image characteristic;
the second processing unit is used for parallelly connecting the result after the product processing with the first image characteristic and then carrying out decoding processing;
and a fourth determining unit configured to determine a segmented image of the target object obtained by image segmentation of the query image based on a result of the decoding process.
The embodiment of the application also provides electronic equipment, which comprises: a processor, a memory and a bus, the memory storing machine-readable instructions executable by the processor, the processor and the memory in communication via the bus when the electronic device is running, the machine-readable instructions when executed by the processor performing the steps of the image segmentation method as described above.
The embodiments of the present application also provide a computer readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of the image segmentation method as described above.
According to the image segmentation method and device provided by the embodiment of the application, the query image and the indication image corresponding to the query image are respectively encoded in the same encoding mode, and the indication image and the query image contain the same type of target objects; extracting first image features of the coded query image and extracting second image features of the coded indication image; based on the first image features and the second image features, performing feature comparison on the query image and the indication image, and determining a feature distribution matrix of a target object in the indication image in the query image; and decoding according to the characteristic distribution matrix and the first image characteristic to obtain a segmented image of the target object segmented from the query image. Compared with the image segmentation method in the prior art, the method has the advantages that the indication image containing the target object and the query image to be segmented are encoded, the image features extracted after encoding are used for comparison, and then the obtained feature distribution matrix is matched with the extracted image features for decoding, so that the segmented image of the target object is segmented from the query image, normalization or size constraint of the image to be segmented is not needed, the corresponding target object can be segmented from the query image to be segmented by using a small amount of data through comparison of the two images, the speed is high, the robustness is good, and the adaptability is strong.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present application more apparent, the technical solutions of the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application, and it is apparent that the described embodiments are only some embodiments of the present application, not all embodiments. The components of the embodiments of the present application generally described and illustrated in the figures herein may be arranged and designed in a wide variety of different configurations. Thus, the following detailed description of the embodiments of the application, as presented in the figures, is not intended to limit the scope of the application, as claimed, but is merely representative of selected embodiments of the application. All other embodiments, which can be made by a person skilled in the art without making any inventive effort, are intended to be within the scope of the present application.
First, an application scenario to which the present application is applicable will be described. The method can be applied to the technical field of image segmentation, and can segment the target object image from a given image, and has the advantages of high speed, good robustness and strong adaptability. Referring to fig. 1, fig. 1 is a system diagram in the application scenario. As shown in fig. 1, the system includes an image segmentation device, an indication image and a query image, the image segmentation device may acquire the indication image and the query image from the indication image and the query image, respectively, and image-segment the query image using the indication image.
According to research, the traditional image segmentation technology generally utilizes a neural network, such as a U-net neural network, to perform end-to-end semantic segmentation on the image, namely, the neural network is enabled to learn the mapping from RGB image space to segmented images, the method can effectively segment the image, but a large amount of labeling data is needed to be performed on the image to play a role, the data amount in the processing process is large, the cost for acquiring the labeling data is high, the time and the labor are consumed, and in practical application, a large amount of data labeling is not easy to be performed on some images, such as the medical field, the safety field and the like.
Based on the above, the embodiment of the application provides an image segmentation method and device, which can find out similar areas by comparing the image characteristics of a query image and an indication image so as to realize the segmentation of a target object image, so that a small amount of data can be used for segmenting a corresponding target object in the query image to be segmented, and the method and device are high in speed, good in robustness and strong in adaptability.
Referring to fig. 2, fig. 2 is a flowchart of an image segmentation method according to an embodiment of the application. As shown in fig. 2, the image segmentation method provided by the embodiment of the application includes:
step 201, encoding the query image and the indication image corresponding to the query image by using the same encoding mode, wherein the indication image and the query image contain the same kind of target objects.
In this step, when the image segmentation device for segmenting the image needs to segment the target object from the query image to be segmented, the image segmentation device may first obtain the query image and the instruction image corresponding to the query image, and encode the query image and the instruction image by using the same encoding method.
The indication image and the query image contain the same kind of target objects, and specifically, the indication image may be a target object indication image of the same kind of target objects that need to be segmented from the query image.
The query image and the indication image are encoded, and a deep learning segmentation neural network model may be used for encoding, for example, a U-net segmentation network model, a deconvolution neural network model, and the like, where the segmentation neural network model may include an encoding network and a decoding network.
In this way, the query image and the indication image are encoded by using the same encoding mode, parameters of the encoding network are shared, and data structures, attributes and the like can be unified, so that subsequent feature extraction, comparison and data processing are facilitated.
Step 202, extracting first image features of the coded query image, and extracting second image features of the coded indication image.
In this step, after encoding the query image and the instruction image, the image segmentation apparatus may further perform feature extraction processing on the encoded query image and the encoded instruction image to extract a first image feature of the query image and a second image feature of the instruction image.
In this way, since the same encoding method is used for encoding, the same feature extraction mode is used for extracting the features, and the workload of the image segmentation apparatus can be reduced.
And 203, comparing the features of the query image and the indication image based on the first image features and the second image features, and determining a feature distribution matrix of the target object in the indication image in the query image.
In this step, after the image segmentation device extracts the first image feature and the second image feature, the image segmentation device may perform feature comparison on the query image and the indication image according to the first image feature and the second image feature, so as to implement a process of fusing information of the query image and the indication image, thereby determining a feature distribution matrix of a target object in the indication image in the query image.
Therefore, by comparing the image characteristics of the query image and the indication image and determining the characteristic distribution matrix, the information fusion of the query image and the indication image can be realized, and the similar area of the query image and the indication image can be quickly found.
And 204, performing decoding processing according to the feature distribution matrix and the first image features to obtain a segmented image of the target object segmented from the query image.
In this step, after the feature distribution matrix is determined, the image segmentation apparatus may perform decoding processing on the feature distribution matrix and the first image feature according to the feature distribution matrix and the first image feature, so as to obtain a segmented image of the target object segmented from the query image.
Thus, after the similar areas of the query image and the indication image are found, the target image can be effectively segmented through decoding processing.
According to the image segmentation method provided by the embodiment of the application, the query image and the indication image corresponding to the query image are respectively encoded in the same encoding mode, and the indication image and the query image contain the same type of target objects; extracting first image features of the coded query image and extracting second image features of the coded indication image; based on the first image features and the second image features, performing feature comparison on the query image and the indication image, and determining a feature distribution matrix of a target object in the indication image in the query image; and decoding according to the characteristic distribution matrix and the first image characteristic to obtain a segmented image of the target object segmented from the query image.
Compared with the image segmentation method in the prior art, the method has the advantages that the indication image containing the target object and the query image to be segmented are encoded, the image features extracted after encoding are used for comparison, and then the obtained feature distribution matrix is matched with the extracted image features for decoding, so that the segmented image of the target object is segmented from the query image, normalization or size constraint of the image to be segmented is not needed, the corresponding target object can be segmented from the query image to be segmented by using a small amount of data through comparison of the two images, the speed is high, the robustness is good, and the adaptability is strong.
Referring to fig. 3, fig. 3 is a flowchart of an image segmentation method according to another embodiment of the present application. As shown in fig. 3, the image segmentation method provided by the embodiment of the application includes:
step 301, encoding a query image and an indication image corresponding to the query image by using the same encoding mode, wherein the indication image and the query image contain the same kind of target objects.
Step 302, extracting a first image feature of the coded query image, and extracting a second image feature of the coded indication image.
Step 303, comparing features of the query image and the indication image based on the first image feature and the second image feature, and determining a feature distribution matrix of the target object in the indication image in the query image.
And 304, multiplying the characteristic distribution matrix and the second image characteristic.
In this step, after the image segmentation apparatus determines the feature distribution matrix and the second image feature, the image segmentation apparatus may prepare to decode the feature and the feature distribution after the comparison, specifically, may first perform product processing on the feature distribution matrix and the second image feature, so as to multiply elements in the feature distribution matrix with the second image feature of the indication image one by one, and perform association.
And 305, parallelly connecting the result after the product processing with the first image feature, and then performing decoding processing.
In this step, the image segmentation apparatus may perform a product process on the feature distribution matrix and the second image feature, and then input a result of the product process and the first image feature in parallel to a decoding network of the segmented neural network model to perform a decoding process, so as to obtain a decoded processing result.
And 306, determining a segmented image of the target object obtained by performing image segmentation on the query image based on the decoded result.
In this step, the image segmentation device performs decoding processing after the result of the product processing is connected in parallel with the first image feature, and after the result of the decoding processing is obtained, the segmented image of the target object segmented after the image segmentation of the query image can be determined according to the result.
The descriptions of steps 301 to 303 may refer to the descriptions of steps 201 to 203, which are not described herein.
Further, step 303 includes:
determining a feature distribution map of the query image based on the first image features; determining a feature vector of the indication image based on the second image feature; and determining a feature distribution matrix of the indication image based on the feature distribution map of the query image and the image feature vector of the indication image.
In this step, the image segmentation apparatus may determine a query image feature distribution map of the query image using the first image feature after the first image feature is extracted, and may determine a feature vector of the indication image using the second image feature, and the image segmentation apparatus may determine a feature distribution matrix of the indication image by feature comparison based on the first image feature and the second image feature after the first image feature and the second image feature are determined.
Wherein each element in the feature distribution matrix reflects the distribution of the target objects contained in the indication image on the query image.
Specifically, in some specific embodiments, the determining, based on the second image feature, a feature vector of the indication image includes:
generating a feature distribution map of the indication image based on the second image feature; and carrying out maximum pooling treatment on the characteristic distribution diagram of the indication image to obtain the characteristic vector of the indication image.
In this step, the image segmentation apparatus may generate a feature distribution map of the instruction image by feature tracing or the like based on the second image feature, and then perform a maximum pooling process on the feature distribution map of the instruction image, so as to obtain a feature vector of the instruction image.
Wherein, the feature vector of the indication image may be a feature vector with a spatial resolution of 1×1.
Specifically, in some specific embodiments, the determining the feature distribution matrix of the indication image based on the feature distribution map of the query image and the feature vector of the indication image includes:
performing inner product processing on the feature vector of the indication image and the feature vector at each position in the feature distribution diagram of the query image; and normalizing the result of the inner product processing to obtain the characteristic distribution matrix of the indication image.
In this step, after determining the feature distribution map of the query image and the feature vector of the indication image, the image segmentation apparatus may correlate the feature vector of the indication image with the feature distribution map of the query image, perform inner product processing, and then normalize the result of the inner product processing by using a normalization exponential function, such as a softmax function, so as to obtain a feature distribution matrix of the indication image.
Wherein the feature distribution matrix dimension is a matrix of S x S dimensions.
For example, referring to fig. 4 and 5 as well, fig. 4 is a schematic diagram of an image segmentation model, and fig. 5 is a schematic diagram of feature alignment shown in fig. 4. As shown in fig. 4 and fig. 5, when the image segmentation apparatus needs to segment the target object 11 in the query image 10, an indication image 20 containing the target object 11 may be obtained, a first image feature of the query image 10 and a second image feature of the indication image 20 are extracted by means of feature extraction or the like, then a deep learning segmentation neural network model is used, for example, a typical U-net segmentation network structure may be used, and the query image 10 and the indication image 20 may be encoded by using the same encoding method, specifically, the first image feature and the second image feature may be encoded, and then multiple convolution computation is performed on the encoded first image feature and the encoded second image feature by using the segmentation neural network model, so as to obtain a first feature map 12 of the query image 10 and a second feature map 21 of the indication image 20 successively.
The first feature map 12 and the second feature map 21 may be represented by using feature vectors with spatial resolution of s×s×c, and may represent a distribution of features on the query image 10 according to the first feature map 12 of the query image 10.
Wherein S is the spatial resolution and C is the channel number.
In the feature comparison stage of the query image 10 and the indication image 20, the second feature map 21 of the indication image 20 may be subjected to global maximum pooling processing to obtain feature vectors 22 with a spatial resolution of 1×1, then the feature vectors 22 are respectively used to make inner products with feature vectors at each position in the first feature map 12 of the query image 10, and normalized by a softmax function to obtain a feature distribution matrix 23 of the indication image 20 in s×s dimension, each element in the feature distribution matrix 23 reflects the distribution condition of the target object contained in the indication image 20 on the query image 10, finally the elements in the feature distribution matrix 23 are multiplied by the feature vectors 22 of the indication image 20 one by one, the multiplied result is parallel connected with the first feature map 12 of the query image 10 and then is input to a decoding network as decoding features for decoding processing, for example, the segmented image 30 containing the target object may be output by a calculation mode such as deconvolution.
In the present embodiment, four convolution calculations are taken as an example, but the present application is not limited thereto, and in other embodiments, the number of convolution calculations may be set according to actual needs.
According to the image segmentation method provided by the embodiment of the application, the query image and the indication image corresponding to the query image are respectively encoded in the same encoding mode, and the indication image and the query image contain the same type of target objects; extracting first image features of the coded query image and extracting second image features of the coded indication image; based on the first image features and the second image features, performing feature comparison on the query image and the indication image, and determining a feature distribution matrix of a target object in the indication image in the query image; performing product processing on the characteristic distribution matrix and the second image characteristic; the result after the product processing is connected in parallel with the first image feature and then is decoded; and determining a segmented image of the target object obtained by carrying out image segmentation on the query image based on the result after the decoding processing.
Compared with the image segmentation method in the prior art, the method has the advantages that the indication image containing the target object and the query image to be segmented are encoded, the image features extracted after encoding are used for comparison, and the obtained feature distribution matrix is matched with the extracted image features for carrying out multiplication and decoding processing, so that the segmented image of the target object is determined from the query image, therefore, the normalization or the size constraint of the image to be segmented is not needed, the corresponding target object can be segmented in the query image to be segmented by using a small amount of data through comparison of the two images, the speed is high, the robustness is good, and the adaptability is strong.
Referring to fig. 6, fig. 6 is a block diagram of an image segmentation apparatus according to an embodiment of the present application, fig. 7 is a block diagram of the comparison module shown in fig. 6, and fig. 8 is a block diagram of the decoding module shown in fig. 6. As shown in fig. 6, the image segmentation apparatus 600 includes:
the encoding module 610 is configured to encode an inquiry image and an indication image corresponding to the inquiry image by using the same encoding mode, where the indication image and the inquiry image include the same type of target object.
An extraction module 620, configured to extract a first image feature of the encoded query image and extract a second image feature of the encoded indication image.
And a comparison module 630, configured to perform feature comparison on the query image and the indication image based on the first image feature and the second image feature, and determine a feature distribution matrix of the target object in the indication image in the query image.
And the decoding module 640 is configured to perform decoding processing according to the feature distribution matrix and the first image feature, so as to obtain a segmented image of the target object segmented from the query image.
Further, as shown in fig. 7, the comparison module 630 includes:
a first determining unit 631 for determining a feature distribution map of the query image based on the first image feature.
A second determining unit 632 is configured to determine a feature vector of the indication image based on the second image feature.
A third determining unit 633, configured to determine a feature distribution matrix of the indication image based on a feature distribution map of the query image and an image feature vector of the indication image.
Further, the second determining unit 632 is specifically configured to:
generating a feature distribution map of the indication image based on the second image feature; and carrying out maximum pooling treatment on the characteristic distribution diagram of the indication image to obtain the characteristic vector of the indication image.
Further, the third determining unit 633 is specifically configured to:
performing inner product processing on the feature vector of the indication image and the feature vector at each position in the feature distribution diagram of the query image; and normalizing the result of the inner product processing to obtain the characteristic distribution matrix of the indication image.
Further, as shown in fig. 8, the decoding module 640 includes:
the first processing unit 641 is configured to perform product processing on the feature distribution matrix and the second image feature.
And a second processing unit 642, configured to perform decoding processing after connecting the result after the product processing with the first image feature in parallel.
A fourth determining unit 643, configured to determine a segmented image of the target object obtained by image segmentation of the query image based on a result of the decoding process.
The image segmentation apparatus 600 in this embodiment may implement all the method steps of the image segmentation method in the embodiment shown in fig. 2 and 3, and may achieve the same effects, which are not described herein.
The image segmentation device provided by the embodiment of the application respectively carries out coding processing on the query image and the indication image corresponding to the query image by using the same coding mode, wherein the indication image and the query image contain the same type of target objects; extracting first image features of the coded query image and extracting second image features of the coded indication image; based on the first image features and the second image features, performing feature comparison on the query image and the indication image, and determining a feature distribution matrix of a target object in the indication image in the query image; and decoding according to the characteristic distribution matrix and the first image characteristic to obtain a segmented image of the target object segmented from the query image.
Compared with the image segmentation method in the prior art, the method has the advantages that the indication image containing the target object and the query image to be segmented are encoded, the image features extracted after encoding are used for comparison, and then the obtained feature distribution matrix is matched with the extracted image features for decoding, so that the segmented image of the target object is segmented from the query image, normalization or size constraint of the image to be segmented is not needed, the corresponding target object can be segmented from the query image to be segmented by using a small amount of data through comparison of the two images, the speed is high, the robustness is good, and the adaptability is strong.
Referring to fig. 9, fig. 9 is a block diagram of an electronic device according to an embodiment of the application. As shown in fig. 9, the electronic device 900 includes a processor 910, a memory 920, and a bus 930.
The memory 920 stores machine-readable instructions executable by the processor 910, when the electronic device 900 is running, the processor 910 communicates with the memory 920 through the bus 930, and when the machine-readable instructions are executed by the processor 910, the steps of the image segmentation method in the method embodiments shown in fig. 2 and fig. 3 may be executed, and a specific implementation may refer to a method embodiment and will not be described herein.
The embodiment of the present application further provides a computer readable storage medium, where a computer program is stored, and when the computer program is executed by a processor, the steps of the image segmentation method in the method embodiments shown in fig. 2 and fig. 3 may be executed, and a specific implementation manner may refer to the method embodiment and will not be described herein.
It will be clear to those skilled in the art that, for convenience and brevity of description, specific working procedures of the above-described systems, apparatuses and units may refer to corresponding procedures in the foregoing method embodiments, and are not repeated herein.
In the several embodiments provided by the present application, it should be understood that the disclosed systems, devices, and methods may be implemented in other manners. The above-described apparatus embodiments are merely illustrative, for example, the division of the units is merely a logical function division, and there may be other manners of division in actual implementation, and for example, multiple units or components may be combined or integrated into another system, or some features may be omitted, or not performed. In addition, each functional unit in the embodiments of the present application may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit.
The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a non-volatile computer readable storage medium executable by a processor. Based on this understanding, the technical solution of the present application may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution, in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a personal computer, a server, a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
Finally, it should be noted that: the above examples are only specific embodiments of the present application, and are not intended to limit the scope of the present application, but it should be understood by those skilled in the art that the present application is not limited thereto, and that the present application is described in detail with reference to the foregoing examples: any person skilled in the art may modify or easily conceive of the technical solution described in the foregoing embodiments, or perform equivalent substitution of some of the technical features, while remaining within the technical scope of the present disclosure; such modifications, changes or substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present application, and are intended to be included in the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.