CN104835146A - Salient object segmenting method in stereo image based on depth information and image cutting - Google Patents

Abstract

The invention discloses a salient object segmenting method in a stereo image based on depth information and image cutting. The method comprises: inputting and segmenting an original image and a depth map in order to acquire a plurality of areas; computing a saliency map of the original image in combination with area-class depths, colors, and spatial domain information; acquiring a threshold value of the saliency map obtained through computation in order to complete initial segmentation of the original image, thereby acquiring an object/background seed point; constructing a map by using the depth map, the saliency map obtained through computation, and a salient weighted histogram, and designing a cost function; and completing salient object segmentation at one time by using a maximum flow minimum cut algorithm. The method reasonably utilizes the depth information and the saliency map, and more accurately and automatically segment the salient object in the stereo image.

Description

Based on object segmentation methods remarkable in the stereo-picture that depth information and figure cut

Technical field

The present invention relates to communication technical field, particularly relate to a kind of based on depth information and remarkable object segmentation methods in the stereo-picture of figure cutting.The method is mainly considered from Appropriate application depth information and Saliency maps to improve this angle of remarkable object segmentation, be intended to utilize the degree of depth, color and spatial information (si) to generate Saliency maps, histogram again in conjunction with depth map and significance weighted carrys out design of graphics and designs cost function, improves the result of remarkable Object Segmentation by figure cutting method.

Background technology

Remarkable Object Segmentation technology refers to and is realized being separated with background on pixel level by the interested object of user in video or image.In existing cutting techniques, the information such as color, direction, texture mainly can be utilized to build conspicuousness model, and utilize the Saliency maps generated to carry out remarkable Object Segmentation by figure cutting method.But result of study also exists many shortcomings, versatility and the accuracy of algorithm have much room for improvement.

Along with the development of three-dimensional imaging and display technique and universal, the depth information utilizing stereo camera or Kinect sensor to obtain can promote the performance of conspicuousness model and remarkable object segmentation methods to a certain extent.A conspicuousness model in conjunction with depth information is proposed in " marking area of stereo-picture detects " that the 19th Digital Signal Processing (DSP) meeting that the people such as Fan held in 2014 in Hong Kong is delivered, this model efficiently utilizes depth information, substantially increase conspicuousness performance, namely the present invention adopts the Saliency maps of this model generation as one of input picture of dividing method.

Summary of the invention

The object of the invention is to Appropriate application depth information and Saliency maps, and then improve remarkable object segmentation, propose a kind of based on depth information and remarkable object segmentation methods in the stereo-picture of figure cutting, the method is compared with the remarkable object method of Saliency maps with traditional color that only utilizes, introduce depth information, improve remarkable Object Segmentation performance.

In order to achieve the above object, the scheme of the present invention's employing is as follows:

Based on an object segmentation methods remarkable in the stereo-picture that depth information and figure cut, its concrete steps are as follows:

Step one, input original image and depth map, carry out pre-segmentation to original image and depth map, and generate Saliency maps;

Step 2, the Saliency maps obtained step one get a fixed threshold value, obtain object/background Seed Points, to complete the initial segmentation of original image;

Step 3, the input that Saliency maps, depth map and original image are cut as figure, utilize Saliency maps, the histogram in conjunction with depth map and significance weighted carrys out design of graphics, and designs cost function;

Step 4, utilize the disposable segmentation completing remarkable object of max-flow min-cut algorithm.

Preferably, the degree of depth of described step one calmodulin binding domain CaM level, color and the spatial information (si) conspicuousness model generation region class Saliency maps that utilizes Fan to provide.

Preferably, image is divided into two parts by the method for Saliency maps being got to threshold value by described step 2, and significance value is greater than the part pixel of threshold value as object Seed Points, is labeled as " obj ", significance value is less than the part pixel Seed Points as a setting of threshold value, is labeled as " bkg ".

Preferably, described step 3 designs cost function in conjunction with the histogram of depth map and significance weighted with design of graphics, and its expression formula is as follows:

E(L)＝R(L)+λ·B(L)+β·E(θ _obj,θ _bkg)

Wherein, L represents the binary vector that pixel marks, and mark comprises object tag and context marker, is designated as " obj " and " bkg "; R (L) is data item, reacts the punishment degree that each pixel is marked as object or background; B (L) is level and smooth item, is mainly used to punish the neighbor obtaining not isolabeling, and this method mainly considers the otherness of color between neighbor; E (θ _obj, θ _bkg) be outward appearance crowded item, reaction subject area and the otherness of background area on color histogram; λ and β is balance factor.

Preferably, max-flow min-cut algorithm described in described step 4 adopts max-flow min-cut algorithm to cut figure and is actually and solves the process of minimum value to the cost function of formula in above-mentioned steps three.

The present invention compared with prior art, there is following apparent substantive distinguishing features and advantage: the invention provides a kind of based on depth information and remarkable object segmentation methods in the stereo-picture of figure cutting, first the method inputs original image and depth map, Iamge Segmentation is carried out to described original image and depth map, obtain several regions, then the degree of depth of calmodulin binding domain CaM level, color and spatial information (si) generate the Saliency maps of original image; Again a fixed threshold value is got to Saliency maps, obtain object/background Seed Points, to complete the initial segmentation of original image; Utilize depth map and the described Saliency maps calculated afterwards, and the histogram of significance weighted carrys out design of graphics and designs cost function; Finally by the method for figure cutting, automatic segmentation is carried out to original image.The method Appropriate application depth information, stereoscopic image can go out saliency object by auto Segmentation more exactly.Compared with prior art, dividing method tool of the present invention has the following advantages: the conspicuousness of Appropriate application depth information and image, for figure cutting provides good Seed Points; In the process of figure cutting, again introduce depth information, to combine design of graphics design cost function with the histogram of Saliency maps, significance weighted, utilize max-flow min-cut algorithm to complete segmentation, obtain better segmentation result.

Accompanying drawing explanation

Fig. 1 is of the present invention based on the process flow diagram of depth information with remarkable object segmentation methods in the stereo-picture of figure cutting;

The schematic diagram that Fig. 2 (a) is original image, the schematic diagram that Fig. 2 (b) is depth map, the schematic diagram that Fig. 2 (c) is Saliency maps, the schematic diagram that Fig. 2 (d) is initial segmentation result, Fig. 2 (e) is the schematic diagram of final segmentation result, the schematic diagram of the remarkable object template that Fig. 2 (f) cuts for people's work point.

Embodiment

Below in conjunction with accompanying drawing, example of the present invention is described in further detail.

The experiment that the present invention carries out be CPU be 2.39GHz, in save as 2G PC test platform on programming realization.

As shown in Figure 1, the present invention is based on the technical scheme that in the stereo-picture of depth information and figure cutting, significantly object segmentation methods adopts is: first input original image and depth map, Iamge Segmentation is carried out to described original image and depth map, then by conspicuousness model generation Saliency maps; Again a fixed threshold value is got to Saliency maps, obtain object/background Seed Points, to complete the initial segmentation of original image; Utilize the Saliency maps of depth map and generation afterwards, and the histogram of significance weighted carry out design of graphics and designs cost function, by the method for figure cutting, automatic segmentation being carried out to original image.Its concrete steps are as follows:

Step one, input original image and depth map, pre-segmentation is carried out to original image and depth map, obtain several regions, and generate Saliency maps, namely the conspicuousness model zoning level Saliency maps that Fan (Fan is the marking area detection technique of stereo-picture) provides is utilized, as shown in Fig. 2 (c).The conspicuousness model generation region class Saliency maps that the degree of depth of step one calmodulin binding domain CaM level, color and spatial information (si) utilize Fan to provide.

Step 2, a fixed threshold value T is got to the Saliency maps that step one generates, obtain object/background Seed Points, to complete the initial segmentation of original image, by the method for Saliency maps being got to threshold value, image is divided into two parts, significance value is greater than the part pixel of threshold value as object Seed Points, be labeled as " obj ", significance value is less than the part pixel Seed Points as a setting of threshold value, is labeled as " bkg ".

Step 3, the input of Saliency maps, depth map and original image being cut as figure, utilize Saliency maps, the histogram in conjunction with depth map, significance weighted carrys out design of graphics and designs cost function, and designs cost function.

Traditional remarkable object segmentation methods only using original image and Saliency maps as input, the present invention introduces depth information to improve segmentation result.The inventive method is using the input as figure cutting of Saliency maps, depth map and original image thus, and utilizes Saliency maps, and the histogram in conjunction with depth map, significance weighted carrys out design of graphics.Different from the figure constructed by generally scheme to cut, add K auxiliary node A in the figure of this method structure ₁, A ₂... A _k... A _k, K is the progression of color histogram, and is connected by auxiliary node corresponding with it for pixel each in image.

Design cost function in conjunction with the histogram of depth map and significance weighted with design of graphics, the form of cost function, as formula (1), comprises data item, level and smooth item and outward appearance crowded item.

E(L)＝R(L)+λ·B(L)+β·E(θ _obj,θ _bkg) (1)

Wherein, L represents the binary vector that pixel marks, and mark comprises object tag and context marker, is designated as " obj " and " bkg "; R (L) is data item, reacts the punishment degree that each pixel is marked as object or background; B (L) is level and smooth item, is mainly used to punish the neighbor obtaining not isolabeling, and this method mainly considers the otherness of color between neighbor; E (θ _obj, θ _bkg) be outward appearance crowded item, reaction subject area and the otherness of background area on color histogram; λ and β is balance factor.

Depth information to be incorporated into the data item of cost function, also need to be analyzed the relation of depth map and Saliency maps.By finding the observation of a large amount of depth map and Saliency maps, Saliency maps is often more reliable than depth map, reason has 2 points: first, and depth information just generates the wherein a part of information used by Saliency maps, and therefore the information that comprises of Saliency maps is more comprehensive; Secondly, depth map is obtained by depth estimation algorithm, the quality of the direct influence depth figure of reliability of algorithm, only is sometimes difficult to distinguish remarkable object and Background by depth information.

In addition as can be seen from depth map also, as Fig. 2 (b), the scope of the depth value of remarkable subject area is narrower than background area, and therefore, the Seed Points that the present invention is only different designs different data item, and only obtains initial markers to after initial segmentation for the pixel of " obj " introduces depth information.The definition of data item R (L) is as shown in table 1.

The definition list of table 1 data item

Wherein, with the mark that after representing initial segmentation and final segmentation respectively, arbitrary pixel obtains; S (p) and D (p) represent the significance value of pixel p in Saliency maps S and depth map D and depth value (Saliency maps and depth map all normalize to [0,1]) respectively; μ _sand μ _drepresent the average of whole Saliency maps and the average of entire depth figure respectively.

Can see from table, when the initial markers that pixel p obtains during for " obj ", if the significance value of this pixel is comparatively large, depth value is less, so illustrate that this pixel belongs to the probability of remarkable object comparatively greatly, the final mark that its obtains be more prone to remain unchanged, i.e. " obj ", now should obtain less penalty value.On the contrary, if the significance value of this pixel p is less, depth value is comparatively large, then illustrate that initial markers is unreliable, mark needs to change, and finally marks be more prone to become " bkg ", the penalty value of now this pixel acquisition is less.And work as initial markers during for " bkg ", data item is only relevant with significance value, and significance value is little, finally marks may be more " bkg " that significance value greatly then final mark may be more " obj ".

Sliding item is mainly used to punish the neighbor obtaining not isolabeling, and mainly consider the otherness of color between neighbor here, its expression formula is as shown in the formula (2):

$B\left(L\right)=\underset{\{p,q\}\∈N}{\mathrm{\Σ}}d{(p,q)}^{-1}\·|L_p-L_q|\·e^{\frac{-{\left|\right|c_p-c_q\left|\right|}^2}{2{\mathrm{\σ}}^2}}---\left(1\right)$

Wherein, d (p, q) represents the Euclidean distance of two pixel p and q position, || c _p-c _q|| be the Euclidean distance of color between pixel p and q, σ ²for the average of the right color distance of all pixels square.Can see from formula (2), two pixels from must more close to, and color value is more close, then they more may obtain identical mark, if and the mark that their obtain is different, then obtain a larger penalty value at level and smooth Xiang Zhonghui, thus force their to obtain same tag.Wherein balance factor λ is set to 9 here, applies the mark smoothing effect of appropriateness.

For design outward appearance crowded item, first define the histogram of significance weighted, as follows:

Will with the number of pixels of a color histogram kth level is belonged to color in background Seed Points in object Seed Points after being defined as initial segmentation respectively.Use c _prepresent the color of pixel p, Q _krepresent a kth level in color histogram, then the histogram of significance weighted can be defined as shown in the formula (3):

$H\left(k\right)=\left\{\begin{array}{c}\underset{p\∈\mathrm{\Ω}}{\mathrm{\Σ}}\frac{S\left(p\right)}{T}\mathrm{\δ}(c_p\∈Q_k),\mathrm{if}{\mathrm{\θ}}_{\mathrm{obj}}^i\left(k\right)>{\mathrm{\θ}}_{\mathrm{bkg}}^i\left(k\right)\\ \underset{p\∈\mathrm{\Ω}}{\mathrm{\Σ}}\frac{1-S\left(p\right)}{T}\mathrm{\δ}(c_p\∈Q_k),\mathrm{otherwise}\end{array}\right.---\left(3\right)$

Wherein δ (.) is indicator function, and when condition in bracket is true time, indicator function value is 1, and when condition is fictitious time, indicator function value is 0.From formula 3, when time, the pixel that great majority belong to kth level obtains object tag after initial segmentation, and they have larger conspicuousness, more may belong to object, therefore, connects auxiliary node A _kwith the capacity on limit between these pixels can utilize significance value to be expanded, makes these limits more difficult cut-off, be set to S (p)/T here, thus the cost on limit that the cut set of optimum cuts off is defined as shown in the formula (4):

$e\left(k\right)=\min [\underset{L_p^f=\mathrm{obj}_{\′\′}^{\′\′}}{\mathrm{\Σ}}\frac{S\left(p\right)}{T}\mathrm{\δ}(c_p\∈Q_k),\underset{L_p^f=\mathrm{bkg}_{\′\′}^{\′\′}}{\mathrm{\Σ}}\frac{S\left(p\right)}{T}\mathrm{\δ}(c_p\∈Q_k)]---\left(4\right)$

From formula 4, because in background, the significance value of pixel is less, thus when scheming cutting, the limit that auxiliary node is connected with background pixel is more easily cut off, thus protects the higher Seed Points of conspicuousness.And work as time, most of pixel more may belong to background, therefore connects auxiliary node A _kwith the capacity setting on limit between pixel is [1-S (p)]/T, thus the cost on limit that the cut set of optimum cuts off is defined as shown in the formula (5):

$e\left(k\right)=\min [\underset{L_p^f=\mathrm{obj}_{\′\′}^{\′\′}}{\mathrm{\Σ}}\frac{1-S\left(p\right)}{T}\mathrm{\δ}(c_p\∈Q_k),\underset{L_p^f=\mathrm{bkg}_{\′\′}^{\′\′}}{\mathrm{\Σ}}\frac{1-S\left(p\right)}{T}\mathrm{\δ}(c_p\∈Q_k)]---\left(5\right)$

Thus, be defined as shown in the formula (6) in conjunction with the histogrammic outward appearance crowded item of significance weighted:

$E({\mathrm{\θ}}_{\mathrm{obj}},{\mathrm{\θ}}_{\mathrm{bkg}})=\underset{k=1}{\overset{K}{\mathrm{\Σ}}}e\left(k\right)---\left(6\right)$

Formula (7) is defined as follows for regulating the balance factor β of above-mentioned outward appearance crowded item weight:

$\mathrm{\β}=0.8\·\frac{\underset{p\∈\mathrm{\Ω}}{\mathrm{\Σ}}\mathrm{\δ}(L_p^i=\mathrm{obj}_{\′\′}^{\′\′})}{\left|\mathrm{\Ω}\right|/2-\underset{k=1}{\overset{K}{\mathrm{\Σ}}}\min [{\mathrm{\θ}}_{\mathrm{obj}}^i\left(k\right),{\mathrm{\θ}}_{\mathrm{bkg}}^i\left(k\right)]}---\left(7\right)$

From formula 7, if with lap more, show after initial segmentation, the color of the color and background of object is not separated well, therefore needs to strengthen the effect in conjunction with the histogrammic outward appearance crowded item of significance weighted, and now β will get larger value; Otherwise β will get less value to weaken the effect of this outward appearance crowded item.

Step 4, max-flow min-cut algorithm is utilized to complete final remarkable Object Segmentation.Max-flow min-cut algorithm described in step 4 adopts max-flow min-cut algorithm to cut figure and is actually and solves the process of minimum value to the cost function of formula in above-mentioned steps three.Finally, with the minimum value of max-flow min-cut algorithm realization cost function, to complete figure cutting.Final segmentation result is as shown in Fig. 2 (e), and compared with the initial segmentation result in Fig. 2 (d), final segmentation result more intactly can be partitioned into whole remarkable object, and maintains good object bounds.

Traditional remarkable object segmentation methods utilizes the information such as color and Saliency maps usually, and the depth information recently risen can be used for improving segmentation performance, therefore first the present invention utilizes depth information color combining, spatial information (si) to generate the higher Saliency maps of quality, again threshold value is got to complete initial segmentation to Saliency maps, finally adopt figure cutting method, utilize the histogram of Saliency maps, depth map and significance weighted to carry out design of graphics, and complete the segmentation of remarkable object.

Claims (5)

1., based on depth information and a remarkable object segmentation methods in the stereo-picture of figure cutting, it is characterized in that, its concrete steps are as follows:

Step one, input original image and depth map, carry out pre-segmentation to original image and depth map, and generate Saliency maps;

Step 2, the Saliency maps obtained step one get a fixed threshold value, obtain object/background Seed Points, to complete the initial segmentation of original image;

Step 3, the input that Saliency maps, depth map and original image are cut as figure, utilize Saliency maps, the histogram in conjunction with depth map and significance weighted carrys out design of graphics, and designs cost function;

Step 4, utilize the disposable segmentation completing remarkable object of max-flow min-cut algorithm.

2. according to claim 1 based on depth information and remarkable object segmentation methods in the stereo-picture of figure cutting, it is characterized in that, the conspicuousness model generation region class Saliency maps that the degree of depth of described step one calmodulin binding domain CaM level, color and spatial information (si) utilize Fan to provide.

3. according to claim 1 based on depth information and remarkable object segmentation methods in the stereo-picture of figure cutting, it is characterized in that, image is divided into two parts by the method for Saliency maps being got to threshold value by described step 2, significance value is greater than the part pixel of threshold value as object Seed Points, is labeled as , significance value is less than the part pixel Seed Points as a setting of threshold value, is labeled as .

4. according to claim 1ly it is characterized in that based on remarkable object segmentation methods in the stereo-picture of depth information and figure cutting, described step 3 designs cost function in conjunction with the histogram of depth map and significance weighted with design of graphics, and its expression formula is as follows:

Wherein, represent the binary vector of pixel mark, mark comprises object tag and context marker, is designated as with ; for data item, react the punishment degree that each pixel is marked as object or background; for level and smooth item, be mainly used to punish the neighbor obtaining not isolabeling, this method mainly considers the otherness of color between neighbor; for outward appearance crowded item, reaction subject area and the otherness of background area on color histogram; with for balance factor.

5. according to claim 4 based on depth information and remarkable object segmentation methods in the stereo-picture of figure cutting, it is characterized in that, max-flow min-cut algorithm described in described step 4 adopts max-flow min-cut algorithm to cut figure and is actually and solves the process of minimum value to the cost function of formula in above-mentioned steps three.