CN113222890A - Small target detection method and device, electronic equipment and storage medium - Google Patents
Small target detection method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN113222890A CN113222890A CN202110345419.0A CN202110345419A CN113222890A CN 113222890 A CN113222890 A CN 113222890A CN 202110345419 A CN202110345419 A CN 202110345419A CN 113222890 A CN113222890 A CN 113222890A
- Authority
- CN
- China
- Prior art keywords
- image
- segmentation
- target object
- region
- object detection
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0012—Biomedical image inspection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/11—Region-based segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10072—Tomographic images
- G06T2207/10088—Magnetic resonance imaging [MRI]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30004—Biomedical image processing
- G06T2207/30096—Tumor; Lesion
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Landscapes
- Engineering & Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
- Radiology & Medical Imaging (AREA)
- Quality & Reliability (AREA)
- Image Analysis (AREA)
Abstract
The invention relates to an artificial intelligence technology, and discloses a small target detection method, which comprises the following steps: the method comprises the steps of obtaining an original image, labeling a target object of the original image to obtain a labeled image, carrying out image segmentation on the labeled image by utilizing a pre-constructed segmentation algorithm to obtain a segmented image, training a pre-constructed segmentation network by utilizing the segmented image to obtain an image segmentation model, carrying out target object detection on an image to be detected by utilizing the image segmentation model to obtain a candidate target object detection result, and carrying out corrosion and expansion processing on the candidate target object detection result to obtain a target object detection result. In addition, the invention also relates to a block chain technology, and the target object detection result can be stored in a node of the block chain. The invention also provides a small target object detection device, electronic equipment and a computer readable storage medium. The invention can solve the problem of low detection accuracy of small target objects.
Description
Technical Field
The invention relates to the technical field of artificial intelligence, in particular to a small target detection method and device, electronic equipment and a computer readable storage medium.
Background
The detection of the target object is an important application of artificial intelligence, namely the identification and detection of the object in the image. For example, the detection and positioning of tumors are important applications of artificial intelligence in the field of clinical medicine, and imaging physicians and clinicians usually perform accurate diagnosis of tumors through a large number of interpretation training based on medical images such as CT and MRI, but at the present time, the distribution of medical resources in china is extremely unbalanced, and doctors in remote areas often cannot accept a large number of training, and meanwhile, the tumors have large differences in position and size, and the quality of images generated by different scanners is uneven, which brings great difficulty to doctors, and is also an important reason for the application of artificial intelligence.
Currently, in existing tumor detection algorithms and products, positioning is usually performed based on a general detection network or a segmentation network, such as common Unet segmentation, fast RCNN detection, and the like. These networks have achieved good overall results on natural or medical images, however, they have not fully considered the needs of the physician at the beginning of the design. The trained tumor detection algorithm can successfully locate most significant tumors, and for small tumors or tumors with low contrast, the detection model is low in training efficiency due to difficult labeling, so that the detection accuracy is low.
Disclosure of Invention
The invention provides a small target detection method, a small target detection device and a computer readable storage medium, and mainly aims to solve the problem of low small target detection accuracy.
In order to achieve the above object, the present invention provides a small target detection method, including:
acquiring an original image, and carrying out target object annotation on the original image to obtain an annotated image;
carrying out image segmentation on the marked image by utilizing a pre-constructed segmentation algorithm to obtain a segmented image;
training a pre-constructed segmentation network by using the segmentation image to obtain an image segmentation model;
carrying out target object detection on an image to be detected by using the image segmentation model to obtain a candidate target object detection result;
and carrying out corrosion and expansion treatment on the detection result of the candidate target object to obtain the detection result of the target object.
Optionally, the performing target object labeling on the original image to obtain a labeled image includes:
generating a circumscribed ellipse based on the original label in the original image;
and generating an external matrix parallel to or perpendicular to the edge of the original image based on the external ellipse, and generating a labeled image containing the external matrix by taking the external matrix as a label.
Optionally, the image segmentation is performed on the annotated image by using a pre-constructed segmentation algorithm to obtain a segmented image, and the image segmentation includes:
setting a region outside the external matrix in the labeled image as a background region, and setting a region in the original label in the external matrix as a foreground region;
and constructing a segmentation function based on the background region and the foreground region, and performing image segmentation on the labeled image by using the segmentation function to obtain the segmented image, wherein the segmented image comprises a segmentation region.
Optionally, the training of the pre-constructed segmentation network by using the segmentation image to obtain the image segmentation model includes:
taking the segmentation areas as pseudo labels, and predicting the probability of the position of each segmentation area in the segmentation image and the label of the position of each segmentation area by using the segmentation network;
constructing a total loss function by using the probability of each segmentation region position and the label of each segmentation region position;
and performing gradient optimization on the total loss function by using a pre-constructed optimizer until the output value of the total loss function meets a preset threshold value, and generating the image segmentation model.
Optionally, the total loss function includes:
Loss＝LTversky+Ldis
wherein L isTverskyFor a Tewosky loss, LdisFor distance Loss, Loss is the total Loss function, pkAnd ykRespectively representing the probability of each predicted position in the output and the label corresponding to each predicted position, dkExpressing the distance from each predicted position to the nearest boundary of the divided region in the label, wherein omega expresses the number of all pixel points, and lambda expresses the distance between each predicted position and the nearest boundary of the divided region in the label1、λ2Is a hyper-parameter.
Optionally, the performing, by using the image segmentation model, target detection on the image to be detected to obtain a candidate target detection result includes:
carrying out target object segmentation on an image to be detected by using the image segmentation model to obtain a target object segmentation area;
and summarizing all the target object segmentation areas to obtain a target object segmentation set, and taking the target object segmentation set as a candidate target object detection result.
Optionally, the performing corrosion and expansion processing on the candidate target detection result to obtain a target detection result includes:
sequentially carrying out region selection on the image to be detected by utilizing a rectangular region with a preset size to obtain a corrosion region;
sequentially setting the target object segmentation area in the corrosion area as a background to obtain a corrosion image;
sequentially carrying out region selection on the corrosion image by utilizing the rectangular region to obtain an expansion region;
setting background pixel points in the expansion area as target object detection areas in sequence, and selecting the target object detection area meeting the preset size as the target object detection result.
In order to solve the above problems, the present invention also provides a small object detection device, including:
the image labeling module is used for acquiring an original image and labeling a target object on the original image to obtain a labeled image;
the image segmentation module is used for carrying out image segmentation on the annotated image by utilizing a pre-constructed segmentation algorithm to obtain a segmented image;
the model training module is used for training a pre-constructed segmentation network by utilizing the segmentation image to obtain an image segmentation model;
the target object detection module is used for detecting a target object of an image to be detected by utilizing the image segmentation model to obtain a candidate target object detection result;
and the target object selection module is used for carrying out corrosion and expansion treatment on the detection result of the candidate target object to obtain the detection result of the target object.
In order to solve the above problem, the present invention also provides an electronic device, including:
a memory storing at least one instruction; and
and the processor executes the instructions stored in the memory to realize the small target detection method.
In order to solve the above problem, the present invention further provides a computer-readable storage medium, which stores at least one instruction, where the at least one instruction is executed by a processor in an electronic device to implement the small object detection method described above.
According to the embodiment of the invention, the target object labeling is carried out on the original image to obtain the labeled image, and the labeling of pixel points one by one is not needed, so that the training efficiency of the model is improved. And the segmented image after segmentation is used as supervision to train the pre-constructed segmentation network, so that the model can focus on the image characteristics in the segmented image, and the accuracy of the model is further improved. Compared with the traditional algorithm in which the segmentation network is firstly utilized for segmentation and then the detection network is utilized for detection, the method only needs to use the segmentation network for detection, so that the waste of computing resources is greatly reduced, meanwhile, the detection result of the candidate target object is subjected to corrosion and expansion treatment, a smaller area can be completely eliminated in the process of multiple corrosion, and a larger area is basically kept unchanged in shape and size through two complementary operations of corrosion and expansion, so that the detection result of the target object is more accurate. Therefore, the small target detection method, the small target detection device, the electronic equipment and the computer readable storage medium provided by the invention can solve the problem of low small target detection accuracy.
Drawings
Fig. 1 is a schematic flow chart of a small target detection method according to an embodiment of the present invention;
FIG. 2 is a schematic flow chart showing a detailed implementation of one of the steps in FIG. 1;
FIG. 3 is a schematic flow chart showing another step of FIG. 1;
FIG. 4 is a schematic flow chart showing another step of FIG. 1;
FIG. 5 is a schematic flow chart showing another step in FIG. 1;
FIG. 6 is a functional block diagram of a small target detection device according to an embodiment of the present invention;
fig. 7 is a schematic structural diagram of an electronic device for implementing the small target detection method according to an embodiment of the present invention.
The implementation, functional features and advantages of the objects of the present invention will be further explained with reference to the accompanying drawings.
Detailed Description
It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
The embodiment of the application provides a small target detection method. The execution subject of the small object detection method includes, but is not limited to, at least one of electronic devices such as a server and a terminal, which can be configured to execute the method provided by the embodiments of the present application. In other words, the small object detection method may be performed by software or hardware installed in a terminal device or a server device, and the software may be a block chain platform. The server includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, and the like.
Fig. 1 is a schematic flow chart of a small target detection method according to an embodiment of the present invention. In this embodiment, the small target detection method includes:
and S1, acquiring an original image, and labeling the original image with a target object to obtain a labeled image.
In an embodiment of the present invention, the original image may be a medical image in a medical field, for example, in a clinical medical field, the original image may be an MRI (magnetic resonance imaging) image or a CT image of a tumor, and the MRI (magnetic resonance imaging) image includes a plurality of sequences (e.g., T2W, T1W, etc.). The original image includes an original label, which may be a RECIST (solid tumor response evaluation criterion) label labeled by a doctor, and the RECIST (solid tumor response evaluation criterion) label is a tumor labeling criterion.
Specifically, referring to fig. 2, the labeling the target object on the original image to obtain a labeled image includes:
s10, generating a circumscribed ellipse based on the original label in the original image;
and S11, generating an external matrix parallel or perpendicular to the edge of the original image based on the external ellipse, and generating a labeled image containing the external matrix by taking the external matrix as a label.
In the embodiment of the invention, for example, according to RECIST labeling of a doctor, an external ellipse is generated based on four vertexes of the RECIST, then an external rectangle of the external ellipse is obtained, four sides of the external rectangle are parallel or perpendicular to the sides of an original image, and an external matrix is used as a label in a detection task, namely a bounding box (bounding box).
According to the embodiment of the invention, only the original marks in the original image need to be further marked, and each pixel point in the original image does not need to be marked, so that the training efficiency of the model is greatly improved.
And S2, carrying out image segmentation on the annotated image by using a pre-constructed segmentation algorithm to obtain a segmented image.
In the embodiment of the present invention, the image segmentation refers to dividing an image into a plurality of non-overlapping regions (such as a foreground and a background) according to features such as gray scale, color, texture, and shape of an annotated image, and making the features present similarities in the same region and obvious differences between different regions. The pre-constructed segmentation algorithm may be a Grab-Cut segmentation algorithm. And the Grab-Cut segmentation algorithm is used for outputting different objects in the marked image as a foreground and a background respectively. For example, for a tumor image, the tumor in the bounding box is taken as the foreground, and the non-tumor in the bounding box is taken as the background.
Specifically, referring to fig. 3, the image segmentation on the labeled image by using the pre-constructed segmentation algorithm to obtain a segmented image includes:
s20, setting the area outside the external matrix in the labeled image as a background area, and setting the area in the original label in the external matrix as a foreground area;
s21, constructing a segmentation function based on the background area and the foreground area, and performing image segmentation on the annotated image by using the segmentation function to obtain the segmented image, wherein the segmented image comprises segmentation areas.
In an optional embodiment of the present invention, the segmentation function may be a function related to a GrabCut algorithm provided by OpenCV: grabCut (img, mask, rect, bgdModel, fgdModel, iterCount, mode), where img represents a segmented image; the mask represents a mask image generated according to a foreground region and a background region, when segmentation is executed, the set foreground region and the set background region can be stored in the mask, then a grabCut function is input, after processing is finished, the result can be stored in the mask, and the mask can only adopt the following four conditions: foreground, background, possible foreground and possible background; rect is used to define the image range that needs to be segmented (for example, it may be a bounding box, and only the image part in the bounding box is processed); the bgdModel represents a background model, and if the bgdModel is None, a bgdModel (background model) is automatically created inside the function; the fgdModel represents a foreground model, and if the foreground model is None, an fgdModel (foreground model) is automatically created inside the function; iterCount represents the number of iterations; mode is used to instruct the grabCut function what to do, such as: initializing GrabCut with a bounding box (bounding box) or initializing GrabCut with a mask image (mask).
In an optional embodiment of the present invention, taking a medical image of a tumor as an example, the image segmentation is performed on the labeled image by using a pre-constructed segmentation algorithm, so as to obtain a tumor region.
In the embodiment of the invention, the pre-constructed segmentation algorithm is used for carrying out image segmentation on the labeled image, so that a more accurate segmentation result can be obtained, and the accuracy of target object detection is improved.
And S3, training a pre-constructed segmentation network by using the segmentation image to obtain an image segmentation model.
In the embodiment of the present invention, the pre-constructed split network may be an Unet split network. The Unet segmentation network comprises a feature extraction layer and an upper sampling layer, wherein the feature extraction layer comprises a plurality of pooling layers, and the pooling layers are used for segmenting a segmentation image into a plurality of segmentation images with different sizes and extracting image features in the segmentation images. The up-sampling layer is used for fusing the segmentation image and the image characteristics corresponding to the segmentation image.
In detail, referring to fig. 4, the training of the pre-constructed segmentation network by using the segmented image to obtain the image segmentation model includes:
s30, taking the segmentation areas as pseudo labels, and predicting the probability of each segmentation area position and the label of each segmentation area position in the segmentation image by using the segmentation network;
s31, constructing a total loss function by using the probability of each segmentation region position and the label of each segmentation region position;
and S32, performing gradient optimization on the total loss function by using a pre-constructed optimizer, and generating the image segmentation model until the output value of the total loss function meets a preset threshold value.
In an embodiment of the present invention, the total loss function includes:
Loss＝LTversky+Ldis
wherein L isTverskyFor a Tewosky loss, LdisFor distance Loss, Loss is the total Loss function, pkAnd ykRespectively representing the probability of each predicted position in the output and the label corresponding to each predicted position, dkExpressing the distance from each predicted position to the nearest boundary of the divided region in the label, wherein omega expresses the number of all pixel points, and lambda expresses the distance between each predicted position and the nearest boundary of the divided region in the label1、λ2Is a hyper-parameter.
In an embodiment of the present invention, the pre-constructed optimizer may be an adaptive moment estimation (Adam) optimizer. The adaptive moment estimation (Adam) optimizer is used for calculating the gradient of a total loss function in each iteration, updating the parameters of the segmentation network by using the gradient until the total loss function meets a preset threshold value, storing the parameters of the segmentation network, and generating a final segmentation model based on the parameters.
In the embodiment of the invention, the pseudo-labeling supervised training segmentation network is utilized, so that the model training efficiency can be improved, and meanwhile, the model can be controlled to be biased to higher recall rate or higher accuracy based on the probability of each segmentation region position and the total loss function constructed by the label of each segmentation region position, so that more small targets can be captured, and the applicability is stronger.
And S4, carrying out target object detection on the image to be detected by using the image segmentation model to obtain a candidate target object detection result.
In an alternative embodiment of the present invention, in the medical imaging field, in order to improve the accuracy of the image segmentation model, for a plurality of sequences (e.g., T2W, T1W, etc.) of MRI (magnetic resonance imaging) images, an image segmentation model may be trained for each sequence.
In the embodiment of the present invention, referring to fig. 5, the performing target object detection on an image to be detected by using the image segmentation model to obtain a candidate target object detection result includes:
s40, carrying out target object segmentation on the image to be detected by using the image segmentation model to obtain a target object segmentation area;
and S41, summarizing all the target object segmentation areas to obtain a target object segmentation set, and taking the target object segmentation set as a candidate target object detection result.
In the embodiment of the invention, the outline of the micro tumor in the MRI (magnetic resonance imaging) image can be marked by utilizing the image segmentation model, so that more micro tumors can be captured.
And S5, carrying out corrosion and expansion treatment on the detection result of the candidate target object to obtain the detection result of the target object.
In an embodiment of the present invention, the performing corrosion and expansion processing on the detection result of the candidate target object to obtain the detection result of the target object includes:
sequentially carrying out region selection on the image to be detected by utilizing a rectangular region with a preset size to obtain a corrosion region;
sequentially setting the target object segmentation area in the corrosion area as a background to obtain a corrosion image;
sequentially carrying out region selection on the corrosion image by utilizing the rectangular region to obtain an expansion region;
setting background pixel points in the expansion area as target object detection areas in sequence, and selecting the target object detection area meeting the preset size as the target object detection result.
In the embodiment of the present invention, for example, in the detection of a micro tumor, the erosion processing refers to moving along the extension of the target object segmentation region by using a rectangular region (which may be a 3 × 3 rectangular region) with a preset size, setting the target object segmentation region covered by the rectangular region as a background, and completely eliminating a smaller region in the erosion process; the expansion processing is that a rectangular region (which can also be a 3-by-3 rectangular region) with a preset size is used for extending and moving along the target object segmentation region, background pixel points covered by the rectangular region in the moving process are set as tumor regions, the region eliminated in the corrosion processing can not appear in the expansion step, the shape and the size of a larger region are basically kept unchanged through two complementary operations of corrosion first and expansion later, and therefore a small false positive region can be effectively removed; meanwhile, if a plurality of tumor regions are predicted, the plurality of tumor regions can be distinguished in a mode of solving a connected domain, and finally a final detection result is screened out according to the size of the tumor regions, for example, the tumor region with the diameter within 3mm can be selected as a target object detection result.
According to the embodiment of the invention, the target object labeling is carried out on the original image to obtain the labeled image, and the labeling of pixel points one by one is not needed, so that the training efficiency of the model is improved. And the segmented image after segmentation is used as supervision to train the pre-constructed segmentation network, so that the model can focus on the image characteristics in the segmented image, and the accuracy of the model is further improved. Compared with the traditional algorithm in which the segmentation network is firstly utilized for segmentation and then the detection network is utilized for detection, the method only needs to use the segmentation network for detection, so that the waste of computing resources is greatly reduced, meanwhile, the detection result of the candidate target object is subjected to corrosion and expansion treatment, a smaller area can be completely eliminated in the process of multiple corrosion, and a larger area is basically kept unchanged in shape and size through two complementary operations of corrosion and expansion, so that the detection result of the target object is more accurate. Therefore, the embodiment of the invention can solve the problem of low detection accuracy of the small target.
Fig. 6 is a functional block diagram of a small target detection device according to an embodiment of the present invention.
The small object detection apparatus 100 of the present invention can be installed in an electronic device. According to the realized functions, the small target object detection device 100 may include an image labeling module 101, an image segmentation module 102, a model training module 103, a target object detection module 104, and a target object selection module 105. The module of the present invention, which may also be referred to as a unit, refers to a series of computer program segments that can be executed by a processor of an electronic device and that can perform a fixed function, and that are stored in a memory of the electronic device.
In the present embodiment, the functions regarding the respective modules/units are as follows:
the image labeling module 101 is configured to obtain an original image, and perform target object labeling on the original image to obtain a labeled image.
In an embodiment of the present invention, the original image may be a medical image in a medical field, for example, in a clinical medical field, the original image may be an MRI (magnetic resonance imaging) image or a CT image of a tumor, and the MRI (magnetic resonance imaging) image includes a plurality of sequences (e.g., T2W, T1W, etc.). The original image includes an original label, which may be a RECIST (solid tumor response evaluation criterion) label labeled by a doctor, and the RECIST (solid tumor response evaluation criterion) label is a tumor labeling criterion.
Specifically, the image annotation module 101 obtains an annotated image by the following operations:
generating a circumscribed ellipse based on the original label in the original image;
and generating an external matrix parallel to or perpendicular to the edge of the original image based on the external ellipse, and generating a labeled image containing the external matrix by taking the external matrix as a label.
In the embodiment of the invention, for example, according to RECIST labeling of a doctor, an external ellipse is generated based on four vertexes of the RECIST, then an external rectangle of the external ellipse is obtained, four sides of the external rectangle are parallel or perpendicular to the sides of an original image, and an external matrix is used as a label in a detection task, namely a bounding box (bounding box).
According to the embodiment of the invention, only the original marks in the original image need to be further marked, and each pixel point in the original image does not need to be marked, so that the training efficiency of the model is greatly improved.
The image segmentation module 102 is configured to perform image segmentation on the annotated image by using a pre-constructed segmentation algorithm to obtain a segmented image.
In the embodiment of the present invention, the image segmentation refers to dividing an image into a plurality of non-overlapping regions (such as a foreground and a background) according to features such as gray scale, color, texture, and shape of an annotated image, and making the features present similarities in the same region and obvious differences between different regions. The pre-constructed segmentation algorithm may be a Grab-Cut segmentation algorithm. And the Grab-Cut segmentation algorithm is used for outputting different objects in the marked image as a foreground and a background respectively. For example, for a tumor image, the tumor in the bounding box is taken as the foreground, and the non-tumor in the bounding box is taken as the background.
Specifically, the image segmentation module 102 obtains a segmented image by:
setting a region outside the external matrix in the labeled image as a background region, and setting a region in the original label in the external matrix as a foreground region;
and constructing a segmentation function based on the background region and the foreground region, and performing image segmentation on the labeled image by using the segmentation function to obtain the segmented image, wherein the segmented image comprises a segmentation region.
In an optional embodiment of the present invention, the segmentation function may be a function related to a GrabCut algorithm provided by OpenCV: grabCut (img, mask, rect, bgdModel, fgdModel, iterCount, mode), where img represents a segmented image; the mask represents a mask image generated according to a foreground region and a background region, when segmentation is executed, the set foreground region and the set background region can be stored in the mask, then a grabCut function is input, after processing is finished, the result can be stored in the mask, and the mask can only adopt the following four conditions: foreground, background, possible foreground and possible background; rect is used to define the image range that needs to be segmented (for example, it may be a bounding box, and only the image part in the bounding box is processed); the bgdModel represents a background model, and if the bgdModel is None, a bgdModel (background model) is automatically created inside the function; the fgdModel represents a foreground model, and if the foreground model is None, an fgdModel (foreground model) is automatically created inside the function; iterCount represents the number of iterations; mode is used to instruct the grabCut function what to do, such as: initializing GrabCut with a bounding box (bounding box) or initializing GrabCut with a mask image (mask).
In an optional embodiment of the present invention, taking a medical image of a tumor as an example, the image segmentation is performed on the labeled image by using a pre-constructed segmentation algorithm, so as to obtain a tumor region.
In the embodiment of the invention, the pre-constructed segmentation algorithm is used for carrying out image segmentation on the labeled image, so that a more accurate segmentation result can be obtained, and the accuracy of target object detection is improved.
The model training module 103 is configured to train a pre-constructed segmentation network with the segmentation images to obtain an image segmentation model.
In the embodiment of the present invention, the pre-constructed split network may be an Unet split network. The Unet segmentation network comprises a feature extraction layer and an upper sampling layer, wherein the feature extraction layer comprises a plurality of pooling layers, and the pooling layers are used for segmenting a segmentation image into a plurality of segmentation images with different sizes and extracting image features in the segmentation images. The up-sampling layer is used for fusing the segmentation image and the image characteristics corresponding to the segmentation image.
In detail, the model training module 103 obtains an image segmentation model by:
taking the segmentation areas as pseudo labels, and predicting the probability of the position of each segmentation area in the segmentation image and the label of the position of each segmentation area by using the segmentation network;
constructing a total loss function by using the probability of each segmentation region position and the label of each segmentation region position;
and performing gradient optimization on the total loss function by using a pre-constructed optimizer until the output value of the total loss function meets a preset threshold value, and generating the image segmentation model.
In an embodiment of the present invention, the total loss function includes:
Loss＝LTversky+Ldis
wherein L isTverskyFor a Tewosky loss, LdisFor distance Loss, Loss is the total Loss function, pkAnd ykRespectively representing the probability of each predicted position in the output and the label corresponding to each predicted position, dkExpressing the distance from each predicted position to the nearest boundary of the divided region in the label, wherein omega expresses the number of all pixel points, and lambda expresses the distance between each predicted position and the nearest boundary of the divided region in the label1、λ2Is a hyper-parameter.
In an embodiment of the present invention, the pre-constructed optimizer may be an adaptive moment estimation (Adam) optimizer. The adaptive moment estimation (Adam) optimizer is used for calculating the gradient of a total loss function in each iteration, updating the parameters of the segmentation network by using the gradient until the total loss function meets a preset threshold value, storing the parameters of the segmentation network, and generating a final segmentation model based on the parameters.
In the embodiment of the invention, the pseudo-labeling supervised training segmentation network is utilized, so that the model training efficiency can be improved, and meanwhile, the model can be controlled to be biased to higher recall rate or higher accuracy based on the probability of each segmentation region position and the total loss function constructed by the label of each segmentation region position, so that more small targets can be captured, and the applicability is stronger.
The target detection module 104 is configured to perform target detection on the image to be detected by using the image segmentation model to obtain a candidate target detection result.
In an alternative embodiment of the present invention, in the medical imaging field, in order to improve the accuracy of the image segmentation model, for a plurality of sequences (e.g., T2W, T1W, etc.) of MRI (magnetic resonance imaging) images, an image segmentation model may be trained for each sequence.
In the embodiment of the present invention, the target detection module 104 obtains a candidate target detection result by:
carrying out target object segmentation on an image to be detected by using the image segmentation model to obtain a target object segmentation area;
and summarizing all the target object segmentation areas to obtain a target object segmentation set, and taking the target object segmentation set as a candidate target object detection result.
In the embodiment of the invention, the outline of the micro tumor in the MRI (magnetic resonance imaging) image can be marked by utilizing the image segmentation model, so that more micro tumors can be captured.
And the target object selection module 105 is configured to perform corrosion and expansion processing on the candidate target object detection result to obtain a target object detection result.
In the embodiment of the present invention, the target selection module 105 obtains a target detection result through the following operations:
sequentially carrying out region selection on the image to be detected by utilizing a rectangular region with a preset size to obtain a corrosion region;
sequentially setting the target object segmentation area in the corrosion area as a background to obtain a corrosion image;
sequentially carrying out region selection on the corrosion image by utilizing the rectangular region to obtain an expansion region;
setting background pixel points in the expansion area as target object detection areas in sequence, and selecting the target object detection area meeting the preset size as the target object detection result.
In the embodiment of the present invention, for example, in the detection of a micro tumor, the erosion processing refers to moving along the extension of the target object segmentation region by using a rectangular region (which may be a 3 × 3 rectangular region) with a preset size, setting the target object segmentation region covered by the rectangular region as a background, and completely eliminating a smaller region in the erosion process; the expansion processing is that a rectangular region (which can also be a 3-by-3 rectangular region) with a preset size is used for extending and moving along the target object segmentation region, background pixel points covered by the rectangular region in the moving process are set as tumor regions, the region eliminated in the corrosion processing can not appear in the expansion step, the shape and the size of a larger region are basically kept unchanged through two complementary operations of corrosion first and expansion later, and therefore a small false positive region can be effectively removed; meanwhile, if a plurality of tumor regions are predicted, the plurality of tumor regions can be distinguished in a mode of solving a connected domain, and finally a final detection result is screened out according to the size of the tumor regions, for example, the tumor region with the diameter within 3mm can be selected as a target object detection result.
Fig. 7 is a schematic structural diagram of an electronic device for implementing a small target detection method according to an embodiment of the present invention.
The electronic device 1 may comprise a processor 10, a memory 11 and a bus, and may further comprise a computer program, such as a small object detection program 12, stored in the memory 11 and executable on the processor 10.
The memory 11 includes at least one type of readable storage medium, which includes flash memory, removable hard disk, multimedia card, card-type memory (e.g., SD or DX memory, etc.), magnetic memory, magnetic disk, optical disk, etc. The memory 11 may in some embodiments be an internal storage unit of the electronic device 1, such as a removable hard disk of the electronic device 1. The memory 11 may also be an external storage device of the electronic device 1 in other embodiments, such as a plug-in mobile hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like, which are provided on the electronic device 1. Further, the memory 11 may also include both an internal storage unit and an external storage device of the electronic device 1. The memory 11 may be used not only to store application software installed in the electronic device 1 and various types of data, such as codes of the small object detection program 12, but also to temporarily store data that has been output or is to be output.
The processor 10 may be composed of an integrated circuit in some embodiments, for example, a single packaged integrated circuit, or may be composed of a plurality of integrated circuits packaged with the same or different functions, including one or more Central Processing Units (CPUs), microprocessors, digital Processing chips, graphics processors, and combinations of various control chips. The processor 10 is a Control Unit (Control Unit) of the electronic device, connects various components of the electronic device by using various interfaces and lines, and executes various functions and processes data of the electronic device 1 by running or executing programs or modules (e.g., small object detection programs, etc.) stored in the memory 11 and calling data stored in the memory 11.
The bus may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. The bus is arranged to enable connection communication between the memory 11 and at least one processor 10 or the like.
Fig. 7 only shows an electronic device with components, and it will be understood by a person skilled in the art that the structure shown in fig. 7 does not constitute a limitation of the electronic device 1, and may comprise fewer or more components than shown, or a combination of certain components, or a different arrangement of components.
For example, although not shown, the electronic device 1 may further include a power supply (such as a battery) for supplying power to each component, and preferably, the power supply may be logically connected to the at least one processor 10 through a power management device, so as to implement functions of charge management, discharge management, power consumption management, and the like through the power management device. The power supply may also include any component of one or more dc or ac power sources, recharging devices, power failure detection circuitry, power converters or inverters, power status indicators, and the like. The electronic device 1 may further include various sensors, a bluetooth module, a Wi-Fi module, and the like, which are not described herein again.
Further, the electronic device 1 may further include a network interface, and optionally, the network interface may include a wired interface and/or a wireless interface (such as a WI-FI interface, a bluetooth interface, etc.), which are generally used for establishing a communication connection between the electronic device 1 and other electronic devices.
Optionally, the electronic device 1 may further comprise a user interface, which may be a Display (Display), an input unit (such as a Keyboard), and optionally a standard wired interface, a wireless interface. Alternatively, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, or the like. The display, which may also be referred to as a display screen or display unit, is suitable for displaying information processed in the electronic device 1 and for displaying a visualized user interface, among other things.
It is to be understood that the described embodiments are for purposes of illustration only and that the scope of the appended claims is not limited to such structures.
The small object detection program 12 stored in the memory 11 of the electronic device 1 is a combination of instructions that, when executed in the processor 10, may implement:
acquiring an original image, and carrying out target object annotation on the original image to obtain an annotated image;
carrying out image segmentation on the marked image by utilizing a pre-constructed segmentation algorithm to obtain a segmented image;
training a pre-constructed segmentation network by using the segmentation image to obtain an image segmentation model;
carrying out target object detection on an image to be detected by using the image segmentation model to obtain a candidate target object detection result;
and carrying out corrosion and expansion treatment on the detection result of the candidate target object to obtain the detection result of the target object.
Specifically, the specific implementation method of the processor 10 for the instruction may refer to the description of the relevant steps in the embodiments corresponding to fig. 1 to fig. 5, which is not repeated herein.
Further, the integrated modules/units of the electronic device 1, if implemented in the form of software functional units and sold or used as separate products, may be stored in a computer readable storage medium. The computer readable storage medium may be volatile or non-volatile. For example, the computer-readable medium may include: any entity or device capable of carrying said computer program code, recording medium, U-disk, removable hard disk, magnetic disk, optical disk, computer Memory, Read-Only Memory (ROM).
The present invention also provides a computer-readable storage medium, storing a computer program which, when executed by a processor of an electronic device, may implement:
acquiring an original image, and carrying out target object annotation on the original image to obtain an annotated image;
carrying out image segmentation on the marked image by utilizing a pre-constructed segmentation algorithm to obtain a segmented image;
training a pre-constructed segmentation network by using the segmentation image to obtain an image segmentation model;
carrying out target object detection on an image to be detected by using the image segmentation model to obtain a candidate target object detection result;
and carrying out corrosion and expansion treatment on the detection result of the candidate target object to obtain the detection result of the target object.
In the embodiments provided in the present invention, it should be understood that the disclosed apparatus, device and method can be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the modules is only one logical functional division, and other divisions may be realized in practice.
The modules described as separate parts may or may not be physically separate, and parts displayed as modules may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment.
In addition, functional modules in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, or in a form of hardware plus a software functional module.
It will be evident to those skilled in the art that the invention is not limited to the details of the foregoing illustrative embodiments, and that the present invention may be embodied in other specific forms without departing from the spirit or essential attributes thereof.
The present embodiments are therefore to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference signs in the claims shall not be construed as limiting the claim concerned.
The block chain is a novel application mode of computer technologies such as distributed data storage, point-to-point transmission, a consensus mechanism, an encryption algorithm and the like. A block chain (Blockchain), which is essentially a decentralized database, is a series of data blocks associated by using a cryptographic method, and each data block contains information of a batch of network transactions, so as to verify the validity (anti-counterfeiting) of the information and generate a next block. The blockchain may include a blockchain underlying platform, a platform product service layer, an application service layer, and the like.
Furthermore, it is obvious that the word "comprising" does not exclude other elements or steps, and the singular does not exclude the plural. A plurality of units or means recited in the system claims may also be implemented by one unit or means in software or hardware. The terms second, etc. are used to denote names, but not any particular order.
Finally, it should be noted that the above embodiments are only for illustrating the technical solutions of the present invention and not for limiting, and although the present invention is described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that modifications or equivalent substitutions may be made on the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.
Claims (10)
1. A method of small object detection, the method comprising:
acquiring an original image, and carrying out target object annotation on the original image to obtain an annotated image;
carrying out image segmentation on the marked image by utilizing a pre-constructed segmentation algorithm to obtain a segmented image;
training a pre-constructed segmentation network by using the segmentation image to obtain an image segmentation model;
carrying out target object detection on an image to be detected by using the image segmentation model to obtain a candidate target object detection result;
and carrying out corrosion and expansion treatment on the detection result of the candidate target object to obtain the detection result of the target object.
2. The small object detection method of claim 1, wherein the performing object labeling on the original image to obtain a labeled image comprises:
generating a circumscribed ellipse based on the original label in the original image;
and generating an external matrix parallel to or perpendicular to the edge of the original image based on the external ellipse, and generating a labeled image containing the external matrix by taking the external matrix as a label.
3. The method for detecting a small target object according to claim 2, wherein the image segmentation of the labeled image by using a pre-constructed segmentation algorithm to obtain a segmented image comprises:
setting a region outside the external matrix in the labeled image as a background region, and setting a region in the original label in the external matrix as a foreground region;
and constructing a segmentation function based on the background region and the foreground region, and performing image segmentation on the labeled image by using the segmentation function to obtain the segmented image, wherein the segmented image comprises a segmentation region.
4. The small object detection method of claim 3, wherein the training of the pre-constructed segmentation network with the segmented image to obtain the image segmentation model comprises:
taking the segmentation areas as pseudo labels, and predicting the probability of the position of each segmentation area in the segmentation image and the label of the position of each segmentation area by using the segmentation network;
constructing a total loss function by using the probability of each segmentation region position and the label of each segmentation region position;
and performing gradient optimization on the total loss function by using a pre-constructed optimizer until the output value of the total loss function meets a preset threshold value, and generating the image segmentation model.
5. The small object detection method of claim 4, wherein the total loss function comprises:
Loss＝LTversky+Ldis
wherein L isTverskyFor a Tewosky loss, LdisFor distance Loss, Loss is the total Loss function, pkAnd ykRespectively representing the probability of each predicted position in the output and the label corresponding to each predicted position, dkExpressing the distance from each predicted position to the nearest boundary of the divided region in the label, wherein omega expresses the number of all pixel points, and lambda expresses the distance between each predicted position and the nearest boundary of the divided region in the label1、λ2Is a hyper-parameter.
6. The method for detecting small objects according to any one of claims 1 to 5, wherein the detecting the object of the image to be detected by using the image segmentation model to obtain the detection result of the candidate object comprises:
carrying out target object segmentation on an image to be detected by using the image segmentation model to obtain a target object segmentation area;
and summarizing all the target object segmentation areas to obtain a target object segmentation set, and taking the target object segmentation set as a candidate target object detection result.
7. The small object detection method of claim 6, wherein the erosion and expansion of the candidate object detection results to obtain object detection results comprises:
sequentially carrying out region selection on the image to be detected by utilizing a rectangular region with a preset size to obtain a corrosion region;
sequentially setting the target object segmentation area in the corrosion area as a background to obtain a corrosion image;
sequentially carrying out region selection on the corrosion image by utilizing the rectangular region to obtain an expansion region;
setting background pixel points in the expansion area as target object detection areas in sequence, and selecting the target object detection area meeting the preset size as the target object detection result.
8. A small object detection device, characterized in that the device comprises:
the image labeling module is used for acquiring an original image and labeling a target object on the original image to obtain a labeled image;
the image segmentation module is used for carrying out image segmentation on the annotated image by utilizing a pre-constructed segmentation algorithm to obtain a segmented image;
the model training module is used for training a pre-constructed segmentation network by utilizing the segmentation image to obtain an image segmentation model;
the target object detection module is used for detecting a target object of an image to be detected by utilizing the image segmentation model to obtain a candidate target object detection result;
and the target object selection module is used for carrying out corrosion and expansion treatment on the detection result of the candidate target object to obtain the detection result of the target object.
9. An electronic device, characterized in that the electronic device comprises:
at least one processor; and the number of the first and second groups,
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform a small object detection method as claimed in any one of claims 1 to 7.
10. A computer-readable storage medium, storing a computer program, wherein the computer program, when executed by a processor, implements a small object detection method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110345419.0A CN113222890B (en) | 2021-03-30 | 2021-03-30 | Small target object detection method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110345419.0A CN113222890B (en) | 2021-03-30 | 2021-03-30 | Small target object detection method and device, electronic equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113222890A true CN113222890A (en) | 2021-08-06 |
| CN113222890B CN113222890B (en) | 2023-09-15 |
Family
ID=77086122
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110345419.0A Active CN113222890B (en) | 2021-03-30 | 2021-03-30 | Small target object detection method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113222890B (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111862096A (en) * | 2020-09-23 | 2020-10-30 | 平安科技（深圳）有限公司 | Image segmentation method and device, electronic equipment and storage medium |
| US20200349712A1 (en) * | 2019-04-06 | 2020-11-05 | Kardiolytics Inc. | Method and system for machine learning based segmentation of contrast filled coronary artery vessels on medical images |
| CN111932547A (en) * | 2020-09-24 | 2020-11-13 | 平安科技（深圳）有限公司 | Method and device for segmenting target object in image, electronic device and storage medium |
| CN111932482A (en) * | 2020-09-25 | 2020-11-13 | 平安科技（深圳）有限公司 | Method and device for detecting target object in image, electronic equipment and storage medium |
| CN112184714A (en) * | 2020-11-10 | 2021-01-05 | 平安科技（深圳）有限公司 | Image segmentation method, image segmentation device, electronic device, and medium |
| CN112465060A (en) * | 2020-12-10 | 2021-03-09 | 平安科技（深圳）有限公司 | Method and device for detecting target object in image, electronic equipment and readable storage medium |
-
2021
- 2021-03-30 CN CN202110345419.0A patent/CN113222890B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20200349712A1 (en) * | 2019-04-06 | 2020-11-05 | Kardiolytics Inc. | Method and system for machine learning based segmentation of contrast filled coronary artery vessels on medical images |
| CN111862096A (en) * | 2020-09-23 | 2020-10-30 | 平安科技（深圳）有限公司 | Image segmentation method and device, electronic equipment and storage medium |
| CN111932547A (en) * | 2020-09-24 | 2020-11-13 | 平安科技（深圳）有限公司 | Method and device for segmenting target object in image, electronic device and storage medium |
| CN111932482A (en) * | 2020-09-25 | 2020-11-13 | 平安科技（深圳）有限公司 | Method and device for detecting target object in image, electronic equipment and storage medium |
| CN112184714A (en) * | 2020-11-10 | 2021-01-05 | 平安科技（深圳）有限公司 | Image segmentation method, image segmentation device, electronic device, and medium |
| CN112465060A (en) * | 2020-12-10 | 2021-03-09 | 平安科技（深圳）有限公司 | Method and device for detecting target object in image, electronic equipment and readable storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113222890B (en) | 2023-09-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111932482B (en) | Method and device for detecting target object in image, electronic equipment and storage medium | |
| CN110059697B (en) | Automatic lung nodule segmentation method based on deep learning | |
| Shen et al. | Unsupervised domain adaptation with adversarial learning for mass detection in mammogram | |
| WO2021217851A1 (en) | Abnormal cell automatic labeling method and apparatus, electronic device, and storage medium | |
| CN111932562B (en) | Image identification method and device based on CT sequence, electronic equipment and medium | |
| CN111932547B (en) | Method and device for segmenting target object in image, electronic device and storage medium | |
| CN111932534B (en) | Medical image picture analysis method and device, electronic equipment and readable storage medium | |
| CN112465060A (en) | Method and device for detecting target object in image, electronic equipment and readable storage medium | |
| CN113298159B (en) | Target detection method, target detection device, electronic equipment and storage medium | |
| CN113283446A (en) | Method and device for identifying target object in image, electronic equipment and storage medium | |
| CN111862096B (en) | Image segmentation method and device, electronic equipment and storage medium | |
| CN112446544A (en) | Traffic flow prediction model training method and device, electronic equipment and storage medium | |
| CN112699775A (en) | Certificate identification method, device and equipment based on deep learning and storage medium | |
| CN110969623B (en) | Lung CT multi-symptom automatic detection method, system, terminal and storage medium | |
| CN111414916A (en) | Method and device for extracting and generating text content in image and readable storage medium | |
| EP3535685A1 (en) | Systems and methods for encoding image features of high-resolution digital images of biological specimens | |
| CN112137591A (en) | Target object position detection method, device, equipment and medium based on video stream | |
| CN112990374A (en) | Image classification method, device, electronic equipment and medium | |
| CN111932595A (en) | Image registration method and device, electronic equipment and storage medium | |
| CN115205225A (en) | Training method, device and equipment of medical image recognition model and storage medium | |
| CN115294426B (en) | Method, device and equipment for tracking interventional medical equipment and storage medium | |
| CN108765399B (en) | Lesion site recognition device, computer device, and readable storage medium | |
| CN113222890B (en) | Small target object detection method and device, electronic equipment and storage medium | |
| CN114511569B (en) | Tumor marker-based medical image identification method, device, equipment and medium | |
| CN115100103A (en) | Tumor prediction method and device based on bacterial data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |