CN109117837B

CN109117837B - Region-of-interest determination method and apparatus

Info

Publication number: CN109117837B
Application number: CN201810836098.2A
Authority: CN
Inventors: 吕梁; 李舒磊; 熊健皓; 赵昕
Original assignee: Shanghai Eaglevision Medical Technology Co Ltd
Current assignee: Shanghai Eaglevision Medical Technology Co Ltd
Priority date: 2018-07-26
Filing date: 2018-07-26
Publication date: 2021-12-07
Anticipated expiration: 2038-07-26
Also published as: CN109117837A

Abstract

The invention provides a method and equipment for determining a region of interest, wherein the method comprises the following steps: acquiring an initial area selected by a user in an image; extracting a plurality of first seed points in the initial region; performing region growing based on the plurality of first seed points to determine at least one growing region; determining a region of interest from the at least one growth region and the initial region.

Description

Region-of-interest determination method and apparatus

Technical Field

The invention relates to the field of image processing, in particular to a method and equipment for determining a region of interest.

Background

Recognizing images using machine learning algorithms and models is an efficient way and also the underlying technology in many fields such as autopilot, smart camera, robotics, etc.

Before image recognition using a machine learning model (e.g., a neural network), the model is first trained using sample images, typically: 1. marking an interested target in the image manually, and generating target area identification information; 2. inputting the identification information generated by the annotation and the image into the deep neural network; 3. and training the deep neural network until the deep neural network converges. The trained machine learning model may then be used to identify and label objects of interest from the images.

In the process of manually marking the target of interest in the image, the annotator needs to manually draw the target of interest according to the position, shape and the like of the target of interest. Fig. 1 shows a fundus image in the medical field, which contains a lesion region whose outline is an irregular polygon. The annotator needs to mark the lesion area in the image, the current image annotation means is usually based on labeling tools such as Labelimg and eidolon annotation assistants, the lesion features of the fundus image are marked by using polygons such as circles and rectangles, and a large error exists between the contour of the marked area and the contour of the target.

The labeling result can only meet the requirements of image identification and detection, but cannot meet higher-level requirements, such as the requirement of segmenting an interested target. When higher-level requirements are met, the contour of the target of interest is required to be labeled more accurately, and if manual labeling is used, a large amount of labor and time cost is consumed.

Disclosure of Invention

In view of this, the present invention provides a method for determining a region of interest, including:

acquiring an initial area selected by a user in an image;

extracting a plurality of first seed points in the initial region;

performing region growing based on the plurality of first seed points to determine at least one growing region;

determining a region of interest from the at least one growth region and the initial region.

Preferably, the acquiring of the initial area selected by the user in the image includes:

acquiring a delineation track of a user in an image;

and determining the initial area according to the delineation track.

Preferably, the extracting a plurality of first seed points in the initial region includes:

acquiring all pixel points in the initial region;

and corroding all the pixel points to screen out a plurality of discrete pixel points as first seed points.

Preferably, the performing region growing based on the plurality of first seed points to determine at least one growing region includes:

respectively extracting characteristic values of the plurality of first seed points;

determining a contour in the image;

and respectively taking the plurality of first seed points as starting points, and performing region growth by using a first condition and a second condition to obtain at least one growth region, wherein the first condition is whether the difference of the characteristic values of adjacent pixel points is smaller than a preset threshold value, and the second condition is whether the pixel points are pixel points on the contour.

Preferably, all of the growth region and the initial region are included in the region of interest.

Preferably, said determining a region of interest from said at least one growth region and said initial region comprises:

traversing pixel points in the image, and respectively judging whether each pixel point belongs to the growth region and/or the initial region;

and determining an interested area according to the pixel points in the growth area and/or the initial area.

Preferably, the determining a region of interest according to pixel points in the growth region and/or the initial region includes:

marking pixel points which belong to the initial region and do not belong to the growth region;

selecting a plurality of second seed points from the marked pixel points;

performing region growing based on the selected second seed point to determine a supplementary region;

and determining a region composed of the pixel points in the supplementary region, the growing region and the initial region as the region of interest.

Preferably, the selecting a plurality of second seed points from the marked pixel points includes:

acquiring all marked pixel points;

and corroding all marked pixel points to screen out a plurality of discrete pixel points as second seed points.

Preferably, the performing region growing based on the selected second seed point to determine a supplementary region includes:

respectively extracting characteristic values of the plurality of second seed points;

determining a contour in the image;

and respectively taking the plurality of second seed points as starting points, and performing region growth by using a first condition, a second condition and a third condition to obtain at least one supplementary region, wherein the first condition is whether the difference of the characteristic values of adjacent pixel points is smaller than a preset threshold, the second condition is whether the pixel points are pixel points on the contour, the third condition is whether the difference of the characteristic values of the pixel points and the average characteristic value is smaller than the preset threshold, and the average characteristic value is the average value of the characteristic values of all marked pixel points.

The present invention is also directed to an electronic device, comprising: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the one processor to cause the at least one processor to perform the above-described region of interest determination method.

According to the method and the device for determining the region of interest, provided by the invention, a user can roughly give the position of the target of interest in an image, namely an initial region is drawn, then the seed point is automatically extracted from the initial region, the region is grown based on the seed point to determine the growing region, and finally the region of interest is determined through the growing region and the initial region, so that the obtained outline of the region of interest is closer to the actual outline of the target of interest, and even the same effect is achieved, thereby reducing the operation of the user, enabling the marking of the target of interest to be more accurate, and achieving the purpose of saving manpower and time cost.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts.

FIG. 1 is a fundus image with a diseased region;

FIG. 2 is a flow chart of a region of interest determination method provided by the present invention;

FIG. 3 is an exemplary image of a process of determining a region of interest;

FIG. 4 is a fundus image to be processed in one embodiment of the present invention;

FIG. 5 is a flow chart of a region of interest determination method in an exemplary embodiment of the invention;

FIG. 6 is a resulting image of the user manually smearing the image shown in FIG. 4;

FIG. 7 is a resulting image after performing sub-region growing based on the contents shown in FIG. 6;

fig. 8 is a region-of-interest result image determined based on the contents shown in fig. 6 and 7.

Detailed Description

The technical solutions of the present invention will be described clearly and completely with reference to the accompanying drawings, and it should be understood that the described embodiments are some, but not all embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

In addition, the technical features involved in the different embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

The embodiment of the invention provides a method for determining a region of interest, which can be executed by electronic equipment such as a server or a personal computer and the like. As shown in fig. 2, the method comprises the steps of:

s1, an initial region selected by the user in the image is obtained, the image can be any image used for training the machine learning model, the image contains some interested target, and the target can be any shape. Fig. 3 shows an exemplary image comprising an object of interest 31 with an irregular contour, which object has a color value that is distinguishable from the background. First, the user can manually mark the original image, for example, draw a line or paint a region within the target, i.e., select the initial region 32, and the initial region 32 can be of any shape without having to paint or outline the target of interest.

S2, extracting a plurality of first seed points in the initial region, where there are a plurality of ways to extract seed points in the region, for example, calculation may be performed according to the shape of the initial region, the RGB values and HSV values of the pixels in the region, and some pixels with outstanding characteristics may be screened out as seed points. The seed points can be distributed discretely or continuously, and the specific distribution mode depends on the extraction algorithm. After this step, 4 first seed points 33 can be determined in the image shown in fig. 3.

S3, performing region growing based on the plurality of first seed points to determine at least one growing region. The region growing algorithm includes multiple kinds, and the region growing condition, the related threshold and the parameter may be preset according to the actual condition of the image, for example, the region growing may be performed according to the relationship between the difference between the RGB value and HSV value of the adjacent pixel and the set threshold, and the edge in the image. Each first seed point is used as a starting point for growth, and the obtained growth areas may be coincident or partially coincident or non-coincident. For example, the boundaries of the growth regions corresponding to the plurality of first seed points may overlap with each other, that is, the plurality of growth regions are connected to form a larger growth region, so that the finally obtained growth regions may be one or more, the number of the growth regions does not have a specified correspondence with the number of the first seed points, and the similarity between the features of the pixel points in the growth regions and the corresponding first seed points is higher. Still taking fig. 3 as an example, two growth areas 34 may be obtained based on 4 first seed points 33 in the image.

S4, determining the region of interest according to the growth region and the initial region, wherein the positional relationship between the growth region and the initial region may present various situations, for example, all the growth region is in the initial region, or one part is in the initial region and the other part is outside the initial region, i.e. there may be completely contained and partially overlapped situations. Since the first seed point is necessarily within the initial region, the growth region does not completely coincide with the initial region. In the case of full inclusion and partial overlap, different processing methods can be adopted to obtain the final region of interest.

In practical applications, a case that the sum of all the growing regions contains and is larger than the initial region may occur, in which case the sum of all the growing regions may be determined as the region of interest; another situation that may occur in comparison is shown in fig. 3, where the growth area 34 contains most of the initial area 32, but the two areas are still not overlapped, i.e. between the two growth areas 34 in the figure, in this case, this vacant area needs to be considered, for example, the sum of the vacant area and the growth area may be used as the region of interest, or further area growth may be performed on the vacant area, and the final region of interest is determined according to the growth result and the growth area determined before. The resulting contour of the region of interest is thus as much as possible identical to the actual object of interest.

According to the method for determining the region of interest provided by the embodiment of the invention, a user can roughly give the position of the object of interest in an image, namely an initial region is drawn, then the method automatically extracts seed points in the initial region, performs region growth based on the seed points to determine a growth region, and finally determines the region of interest through the growth region and the initial region, so that the obtained contour of the region of interest is closer to the actual contour of the object of interest, and even completely the same effect is achieved, thereby reducing the operation of the user, enabling the marking of the object of interest to be more accurate, and achieving the purpose of saving manpower and time cost.

The region of interest determination method provided by the present invention can be applied to the labeling of medical images, and fig. 4 shows a fundus image, which is actually a colored image, taken by a specific medical device. In order to train a neural network model capable of identifying the optic disc, a large number of eye fundus images marked with the optic disc and corresponding label information are used as model training data, and the method can be used for marking the eye fundus images. As shown in fig. 5, an embodiment of the present invention provides a method for determining a region of interest, that is, an automatic image annotation method, where the method includes the following steps:

and S11, acquiring a delineation track of the user in the image, and determining an initial area according to the delineation track. As shown in fig. 6, the user manually applies the fundus image shown in fig. 4, and the manually applied region 61 is an initial region.

S12, all pixel points in the initial region are obtained, and all pixel points are corroded to screen out a plurality of discrete pixel points to serve as first seed points. Specifically, the pixel points included in the smearing region 61 can be set as a set, the region is reduced to a plurality of unconnected pixel points through a corrosion algorithm, and the points are used as first seed points for region growth.

And S13, determining contours in the image, wherein the lines in the image naturally form contours, namely edges of various actual objects, and the contours serve as stop conditions in the subsequent region growing process. The image edges can be obtained by methods such as gaussian blurring or frequency filtering. The Gaussian blur can reduce the picture noise and reduce the detail level, so that the edges existing in a large range can be extracted more easily. Contour edges present in the image are marked in the graph that has been denoised, for example edges can be marked using gradient operators such as Sobel operators.

S14, respectively extracting the characteristic values of the first seed points, wherein the RGB values of the pixel points can be extracted firstly, then the RGB values are converted into HSV values, namely the characteristic values, and the color difference is calculated in the HSV mode subsequently;

and S15, respectively taking the plurality of first seed points as starting points, and performing region growth by using a first condition and a second condition to obtain at least one growth region, wherein the first condition is whether the difference of the characteristic values of adjacent pixel points is smaller than a preset threshold value, and the second condition is whether the pixel points are the pixel points on the contour. The first condition and the second condition are applied, when the growth touches the outline in the growth process, the growth is stopped; or stopping growing when the color difference of the adjacent pixel areas exceeds a preset threshold value in the growing process.

This is the first region growing process in this embodiment, and in this process, the first seed point may be marked as 1, and the remaining pixel points are marked as 0. The first seed point is traversed breadth-first, traversing 8 points (points in 8 neighborhoods) around the point marked as 1, and according to the HSV color space model, the difference of colors can be represented by a space distance in the model, namely a Euclidean distance. If the color difference between a point in the 8 neighborhood and the center point is less than a preset threshold, it is marked as 1. When an edge is detected in the 8 neighborhood, the current region growing is stopped. The region consisting of the points marked 1 is the growth region. After processing, two growth regions 71 are obtained as shown in fig. 7, and the two growth regions 71 are not completely overlapped with the initial region 61, and a vacant region 72 is present in the middle.

S16, traversing pixel points in the image, and respectively judging whether each pixel point belongs to a growth region and/or an initial region;

and S17, determining the region of interest according to the pixel points in the growth region and/or the initial region.

The above processing procedure can be regarded as performing an or operation on the smearing region 61 and the growing region 71, which means traversing each pixel point of the whole image, and if the current pixel point is marked in the manual smearing region 61 or the region growing process, updating and marking the pixel point in the image; if this pixel is not marked in both operations, it is unchanged. This is done to ensure that during the region growing operation, the manually marked region is not automatically erased, and the region of interest 81 shown in fig. 8 is finally obtained.

Further, step S17 may include the following steps:

s171, marking the pixel points which belong to the initial region and do not belong to the growing region, and marking the pixel points as 2 if the user paints the pixel points and the region growing does not reach the pixel points in the OR operation. This is done to extract the vulnerability area of the algorithm for further optimization.

S172, selecting a plurality of second seed points from the marked pixel points, and performing region growing based on the selected second seed points to determine the supplemental region 72. After the area growth is finished, the area marked as 2 is reduced to a plurality of single pixel points through corrosion, the second area growth processing in the embodiment is performed by taking the points as seed points, the process is similar to the steps S12-S15, the seed points are determined by adopting a corrosion processing algorithm, and then the area growth is performed based on the set growth conditions, so that the area which is not reached by the first area growth is supplemented.

S173, determining a region composed of pixel points in the supplement region, the growth region, and the initial region as a region of interest 81.

The above process of the second region growing has some differences compared to the first region growing, that is, there may be three growing conditions when the second region growing is performed, specifically, step S172 may include the following steps:

s1721, respectively extracting characteristic values of a plurality of second seed points, namely acquiring RGB values of pixel points, and then converting the RGB values into HSV values;

s1722, respectively taking the plurality of second seed points as starting points, and performing region growing under a first condition, a second condition and a third condition to obtain at least one supplemental region, where the first condition and the second condition can be referred to in step S15. The third condition is whether the difference between the characteristic value of the pixel point and the average characteristic value is smaller than a preset threshold value, and the average characteristic value is the average value of the characteristic values of all marked pixel points.

Specifically, when the smear region 61 is acquired, the values of the hsv model of the point marked 2 may be averaged for each channel, which is referred to as an average color. When the second area growth is carried out, comparing the color difference value between the point of the 8 neighborhoods and the central point, if the first condition is met but the second condition is not met (when the difference value is larger than the preset threshold), also comparing the color difference value between the neighborhoods and the average color, if the difference value is smaller than the preset difference value, marking the point as 1 to continue to grow, and stopping growing until all the conditions are not met.

The embodiment of the invention also provides electronic equipment, which comprises at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to cause the at least one processor to perform the region of interest determination method.

As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

It should be understood that the above examples are only for clarity of illustration and are not intended to limit the embodiments. Other variations and modifications will be apparent to persons skilled in the art in light of the above description. And are neither required nor exhaustive of all embodiments. And obvious variations or modifications therefrom are within the scope of the invention.

Claims

1. A region-of-interest determination method, comprising:

acquiring an initial area selected by a user in an image;

extracting a plurality of first seed points in the initial region;

determining a contour in the image;

respectively taking the plurality of first seed points as starting points, and performing region growth by using a first condition and a second condition to obtain at least one growth region, wherein the first condition is whether the difference of the characteristic values of adjacent pixel points is smaller than a preset threshold value, and the second condition is whether the pixel points are pixel points on the contour;

selecting a plurality of second seed points from the marked pixel points;

2. The method of claim 1, wherein the obtaining of the user selected initial region in the image comprises:

acquiring a delineation track of a user in an image;

and determining the initial area according to the delineation track.

3. The method of claim 1, wherein said extracting a plurality of first seed points in the initial region comprises:

acquiring all pixel points in the initial region;

4. The method of claim 1, wherein selecting a plurality of second seed points from the marked pixel points comprises:

acquiring all marked pixel points;

5. The method of claim 1, wherein the region growing based on the selected second seed point to determine a supplemental region comprises:

determining a contour in the image;

6. An electronic device, comprising: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to cause the at least one processor to perform the region of interest determination method of any one of claims 1-5.