CN110189294B - RGB-D image significance detection method based on depth reliability analysis - Google Patents

RGB-D image significance detection method based on depth reliability analysis Download PDF

Info

Publication number
CN110189294B
CN110189294B CN201910298984.9A CN201910298984A CN110189294B CN 110189294 B CN110189294 B CN 110189294B CN 201910298984 A CN201910298984 A CN 201910298984A CN 110189294 B CN110189294 B CN 110189294B
Authority
CN
China
Prior art keywords
depth
significance
image
map
saliency
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201910298984.9A
Other languages
Chinese (zh)
Other versions
CN110189294A (en
Inventor
周洋
刘晓琪
尉婉丽
梁文青
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Hangzhou Eyecloud Technology Co ltd
Original Assignee
Hangzhou Dianzi University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hangzhou Dianzi University filed Critical Hangzhou Dianzi University
Priority to CN201910298984.9A priority Critical patent/CN110189294B/en
Publication of CN110189294A publication Critical patent/CN110189294A/en
Application granted granted Critical
Publication of CN110189294B publication Critical patent/CN110189294B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/136—Segmentation; Edge detection involving thresholding
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/60—Analysis of geometric attributes
    • G06T7/66—Analysis of geometric attributes of image moments or centre of gravity
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/90—Determination of colour characteristics
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence
    • G06T2207/10021—Stereoscopic video; Stereoscopic image sequence
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20212—Image combination
    • G06T2207/20221—Image fusion; Image merging

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Geometry (AREA)
  • Quality & Reliability (AREA)
  • Image Analysis (AREA)

Abstract

The invention discloses an RGB-D image significance detection method based on depth reliability analysis. The method comprises the steps of firstly, evaluating the reliability of scene far and near degree information reflected by a depth map, leading out a depth reliability factor lambda as a main characteristic index for measuring the significance of a stereo image, and when the depth map can accurately reflect the scene information, namely lambda is less than 0.45, carrying out stereo image significance detection only through depth characteristics; when the depth map reliability is low or the scene is fuzzy, namely lambda is more than or equal to 0.45, and the significance detection is carried out by combining other characteristics such as color and the like. The method fully considers the contribution degree of the clues of the depth map to the stereoscopic vision significance detection, and judges whether the depth map can accurately reflect the scene distance or whether the depth map has distortion or not by calculating the credibility factor of the depth map. The method has low computational complexity, the obtained three-dimensional image saliency map has high quality, and the method can be directly applied to the engineering fields of 3D image perception, 3D quality evaluation, object identification and the like.

Description

RGB-D image significance detection method based on depth reliability analysis
Technical Field
The invention belongs to the technical field of image processing, particularly relates to the technical field of stereo image processing, and relates to a RGB-D image significance detection method based on depth reliability analysis.
Background
With the advent of the "explosion-type" of multimedia tools, huge amounts of information in various ways are emerging in front of people, and images and videos are favored due to the characteristics of abundant expressive force, vivid content and the like, and become the mainstream way of information expression and transmission. How to accurately and efficiently mine significant information with high attention of human eyes in images/videos to reduce the burden of machine computing processing and storage becomes a common research hotspot in the fields of video image processing and computer vision. Most of the existing calculation methods are mainly used for monocular saliency detection, and accurate detection of color images is realized, but in natural scenes, the human visual system can process the distance of the scenes when perceiving information such as colors and shapes, namely perceiving depth information. The traditional significance model ignores the influence of depth features on target detection to a certain extent. In recent years, scholars have developed stereo image saliency detection in combination with depth features in the field of binocular stereo vision. In the human eye vision system, the binocular disparity makes the images of the natural scene projected into the eyes slightly different, the disparity is mainly focused on the horizontal dimension, namely parallax, and the disparity is an important channel for human perception of the stereoscopic depth of the scene. In saliency detection, depth information is generally expressed as a grayscale image, and a pixel value thereof represents a distance from a camera to an object projected in an image plane, reflects a distance of an object from human eyes, and is one of important perceptual features of a stereoscopic image.
In the method for detecting the significance of the three-dimensional scene, students highlight a significant target area by adopting different calculation models and combining plane features such as colors, textures and the like and depth features. In these methods, some depth maps may generate distortion or scene blurring due to different depth map acquisition modes, thereby affecting the significance detection result to generate a certain deviation.
Disclosure of Invention
The invention aims to provide an RGB-D image significance detection method based on depth reliability analysis aiming at the defects of the prior art.
The method comprises the steps of firstly, evaluating the reliability of the information of the degree of distance of a scene reflected by a depth map to obtain a reliability factor lambda of the depth map, taking lambda as a characteristic index for measuring the significance of a three-dimensional image, carrying out three-dimensional visual significance detection based on the depth information when the lambda is less than 0.45, and carrying out three-dimensional visual significance detection by combining color characteristics when the lambda is more than or equal to 0.45; the method comprises three modules: analyzing depth reliability, performing stereoscopic vision significance detection based on depth information, and performing stereoscopic vision significance detection by combining color characteristics; the method comprises the following specific steps:
(1) first, a depth confidence analysis is performed: inputting a depth image: firstly, image segmentation is carried out by adopting a linear iterative clustering SLIC superpixel segmentation algorithm, and a graph G (V, E) is constructed, wherein V represents a node set, and each node in V corresponds to a superpixel block ViThe edge E connects adjacent superpixel blocks; superpixel block v from depth map feature analysisiThe depth confidence of (d) is:
Figure GDA0002880573290000021
wherein m and s represent the mean and standard deviation of pixel values in the whole depth map, respectively, and miRepresenting superpixel blocks viH represents the image entropy,
Figure GDA0002880573290000022
wherein L is the gray level of the depth map, PjIs the probability of the occurrence of the jth gray level in the depth map; c0Is an intensity adjustment factor, C is more than or equal to 10≤10；
Then fuse each superpixel block viDepth confidence value λ ofiObtaining the credibility factor of the whole depth map
Figure GDA0002880573290000023
Wherein N issRepresenting the number of superpixels, Ns∈[100,200,300](ii) a When lambda is less than 0.45, stereoscopic vision significance detection is carried out based on the depth information; and when the lambda is more than or equal to 0.45, performing stereoscopic vision significance detection by combining color characteristics.
(2) The specific steps of performing stereoscopic vision saliency detection based on depth information are as follows:
(2-1) coarse background filtering: comparing the average depth difference degree of the pixel points in the depth map with the average depth difference degree of the lines where the pixel points are located to obtain a preliminary depth map front background distinction:
Figure GDA0002880573290000024
wherein, IkAnd l'kRespectively representing the depth values of the pixel point k in the original depth image and the coarsely filtered image,
Figure GDA0002880573290000025
representing the average depth value of the line where the pixel point k is located;
(2-2) depth compactness significance analysis: in combination with the depth confidence factor, the superpixel block v is first analyzediDepth-based compactness of (2):
Figure GDA0002880573290000026
where Sdc (vi) represents the compact saliency value, n, of each super-pixel blockjIs a super pixel vjNumber of pixels, λ, contained injRepresenting calculated superpixel blocks vjThe depth confidence value of (a) is,
Figure GDA0002880573290000027
is a super pixel block vjThe coordinates of the center of mass of the image,
Figure GDA0002880573290000028
representing the centroid position of the entire depth image; a isijRepresenting the similarity between two superpixel blocks in the preprocessed depth map,
Figure GDA0002880573290000029
m′iand m'jRepresenting a superpixel vi'and v'jAverage value of (d);
Figure GDA00028805732900000210
in order to control the constant amount of the liquid,
Figure GDA00028805732900000211
σ2to control the affinity constant, σ, of the similarity matrix20.1; significance map S for depth compactness calculationcomThe final realization is as follows: scom(vi) 1-norm (Sdc (vi)), norm (. cndot.) as a normalization function, remapping significant values to [0,255 [ (. cndot.) ]]Obtaining a depth compactness saliency map from the range of (1);
(2-3) depth contrast significance analysis: saliency value S of pixel kcon(k) Based on its contrast with all other pixels in the depth image:
Figure GDA00028805732900000212
flrepresenting the frequency of the appearance of different depth values in the depth map after the rough background filtering; wherein D (I'k,I′l)＝||I′k-I′lL, representing image pixel I'kAnd other image pixel points I'lSpatial distance between, pixel point I'kAnd l'lDepth value of [0,255 ]]Within the range;
(2-4) significance of depth compactness result ScomAnd depth contrast saliency result SconAnd (3) fusion, wherein the acquired saliency map based on the depth information is as follows: sdepth＝θScom+(1-θ)SconAnd theta is a positive control parameter between two significance terms, and theta is 0.5.
(3) The method for detecting the stereoscopic vision significance by combining the color characteristics comprises the following specific steps of:
(3-1) parallel structure based on background priors:
firstly, the influence of the background is minimized by eliminating the false boundary, and the realization method comprises the following steps:
Figure GDA0002880573290000031
Dcolor(Ip,Iq) Representing the color distance difference between different sides, R/G/B are respectively red, green and blue channels, p and q are any two sides of the four boundaries,
Figure GDA0002880573290000032
is the mean of the characteristics of the three channels on the boundary p,
Figure GDA0002880573290000033
is the mean value of the characteristics of the three channels on the boundary q; the 4 x 4 matrix A is obtained by calculating the distances D between all the boundaries and normalized, at the sum of the maximum columns
Figure GDA0002880573290000034
Sum of minimum column
Figure GDA0002880573290000035
Satisfies the conditions
Figure GDA0002880573290000036
When the conditions are met, defining the boundary corresponding to the column and the maximum value as a false boundary, and removing the false boundary; tau isCTo set the threshold, 0.1 ≦ τC≤1.0；
Then, respectively carrying out significance sorting based on the background and the foreground; calculating a saliency map based on the edge background for other edges after filtering the false boundaries;
finally, the background prior based significance ranking results are:
Figure GDA0002880573290000037
wherein the content of the first and second substances,
Figure GDA0002880573290000038
in correspondence with the remaining boundary(s),
Figure GDA0002880573290000039
representing a significant result graph based on each background edge, and obtaining a significant sequencing result S based on background prior through final multiplicative fusionb(i) (ii) a The significance implementation of the analysis based on the foreground is expressed as:
Figure GDA00028805732900000310
after respectively solving significance graphs based on background and foreground, multiplicative fusion is carried out to obtain a primary significant target area Sinitial(i)＝Sb(i)·Sf(i)；
(3-2) after obtaining a primary significant result, performing feature optimization by taking the foreground area in the graph as a seed point, and redefining a popular ranking function indication vector as follows:
Figure GDA00028805732900000311
calculating to obtain a final saliency map based on RGB color features after the saliency features are optimized;
(3-3) significance update in combination with depth information:
the significance results are updated by using an iterative method of a cellular automaton, and the method is modified according to the actual situation of the invention:
Figure GDA00028805732900000312
wherein, | | di,djI | represents the depth distance between superpixel blocks i and j, NiIs a neighborhood set of superpixels i, fijRepresenting the similarity, δ, between different superpixel blocks2Indicating a parameter controlling similar intensity, δ2＝0.1；
Based on the depth feature similarity, the significance value of each super pixel is determined by the significance feature value of the super pixel and the neighborhood feature value of the super pixel. And setting the iterative propagation times of the cell machine as K times, wherein K is 5-50, and obtaining a more accurate RGB-D significance map by combining significance updating of depth information.
The method of the invention provides a novel stereo image significance detection technology in the field of stereo image processing research, and can simulate the human eye visual attention mechanism as much as possible. The method improves the existing stereo image salient object detection algorithm, and fully utilizes the important function of depth information in stereo vision salient object detection. The method introduces a novel discrimination mode, namely a depth map credibility evaluation factor, and can accurately measure whether scene information in the depth map has blur or distortion, and further judge whether the three-dimensional image significance detection can be carried out only through the depth map information. When the depth map is fuzzy or distorted, the scene characteristics can not be accurately judged only through the depth map, and the stereoscopic vision significance is analyzed by combining color information. The three-dimensional image saliency model can be directly applied to the engineering fields of 3D video processing, 3D quality evaluation, object recognition and the like.
Drawings
FIG. 1 is a flow chart of the method of the present invention;
FIG. 2 is a stereo image depth map;
fig. 3 is a stereo image depth map SLIC processing result;
FIG. 4 is a depth map pre-processing result diagram;
FIG. 5 is a depth map compact salient results image;
FIG. 6 is a depth map versus saliency results image;
FIG. 7 is a perspective image saliency map result based on depth information;
FIG. 8 is a perspective image saliency map result incorporating color information;
fig. 9 is a diagram of detection results of different stereoscopic image sequences.
Detailed Description
As shown in fig. 1, in the RGB-D image saliency detection method based on depth reliability analysis, firstly, the reliability of the information of the degree of distance of a scene reflected by a depth map is evaluated, a depth reliability factor λ is introduced as a main characteristic index for measuring the saliency of a stereo image, and when the depth map can more accurately reflect the scene information, that is, λ is less than 0.45, the saliency of the stereo image is detected only by depth characteristics; conversely, when the depth map is low in credibility or the scene is fuzzy, the saliency detection is performed by combining other features such as colors and the like, so that the saliency detection complexity of the stereo image is reduced on the whole.
The method comprises three modules: depth reliability analysis, stereoscopic vision significance detection based on depth information, and stereoscopic vision significance detection based on color characteristics. The method comprises the following specific steps:
(1) first, a depth confidence analysis is performed:
as shown in fig. 2, the depth image is input: firstly, the image segmentation is carried out by adopting the existing simple linear iterative clustering SLIC superpixel segmentation algorithm to construct(V, E), where V represents a set of nodes, each node in V corresponding to a superpixel block ViThe edge E connects adjacent superpixel blocks; superpixel block v from depth map feature analysisiThe depth confidence of (d) is:
Figure GDA0002880573290000051
wherein m and s represent the mean and standard deviation of pixel values in the whole depth map, respectively, and miRepresenting superpixel blocks viH represents the image entropy,
Figure GDA0002880573290000052
wherein L is the gray level of the depth map, PjIs the probability of the j-th gray level occurring in the depth map. C0Is an intensity adjustment factor, C is more than or equal to 10≦ 10, example C 02. The image depth map SLIC processing results are shown in fig. 3.
Then fuse each superpixel block viDepth confidence value λ ofiObtaining the credibility factor of the whole depth map
Figure GDA0002880573290000053
Wherein N issRepresenting the number of superpixels, Ns∈[100,200,300]In the present invention, 200 is taken. The smaller lambda is, the higher the reliability of the depth map is, and the scene target information can be extracted more accurately from the depth map. When lambda is less than 0.45, a salient target area is accurately extracted through scene information reflected by the depth map, and stereoscopic vision saliency detection is carried out by adopting depth-based information; when the lambda is more than or equal to 0.45, processing is carried out by combining with other scene characteristics, and the method carries out stereoscopic vision significance detection by combining with color characteristics.
(2) The specific steps of performing stereoscopic vision saliency detection based on depth information are as follows:
(2-1) coarse background filtering: in order to reduce the interference of a non-significant area with a large depth value in a depth map, the invention provides a simple rough background filtering mode, and the purpose of reducing the interference is realized by comparing the average depth difference degree of pixel points and the lines in the depth map, thereby obtaining a preliminaryDepth map front background discrimination:
Figure GDA0002880573290000054
wherein, IkAnd l'kRespectively representing the depth values of the pixel point k in the original depth image and the coarsely filtered image,
Figure GDA0002880573290000055
and the average depth value of the line where the pixel point k is located is represented. The depth map obtained by the coarse background filtering is shown in fig. 4.
(2-2) depth compactness significance analysis: in combination with the depth confidence factor, the superpixel block v is first analyzediDepth-based compactness of (2):
Figure GDA0002880573290000056
where Sdc (vi) represents the compact saliency value, n, of each super-pixel blockjIs a super pixel vjNumber of pixels, λ, contained injRepresenting calculated superpixel blocks vjThe depth confidence value of (a) is,
Figure GDA0002880573290000057
is a super pixel block vjThe coordinates of the center of mass of the image,
Figure GDA0002880573290000058
representing the centroid position of the entire depth image; a isijRepresenting the similarity between two superpixel blocks in the preprocessed depth map,
Figure GDA0002880573290000059
m′iand m'jRepresenting a superpixel vi'and v'jAverage value of (d);
Figure GDA0002880573290000061
in order to control the constant amount of the liquid,
Figure GDA0002880573290000062
σ2to control the affinity constant of the similarity matrix,σ2＝0.1；
significance map S for depth compactness calculationcomThe final realization is as follows: scom(vi) 1-norm (Sdc (vi)), norm (. cndot.) as a normalization function, remapping significant values to [0,255 [ (. cndot.) ]]The range of (a) yields a depth compactness saliency map. The results are shown in FIG. 5.
(2-3) depth contrast significance analysis: saliency value S of pixel kcon(k) Based on its contrast with all other pixels in the depth image:
Figure GDA0002880573290000063
flindicating the frequency of occurrence of different depth values in the depth map after the rough background filtering. Wherein D (I'k,I′l)＝||I′k-I′lL, representing image pixel I'kAnd other image pixel points I'lSpatial distance between, pixel point I'kAnd l'lDepth value of [0,255 ]]Within the range. The results are shown in FIG. 6.
(2-4) significance of depth compactness result ScomAnd depth contrast saliency result SconAnd (3) fusion, wherein the acquired saliency map based on the depth information is as follows: sdepth＝θScom+(1-θ)Scon(ii) a θ is a positive control parameter between two significance terms, and θ is 0.5. The results are shown in FIG. 7.
(3) The method for detecting the stereoscopic vision significance by combining the color characteristics comprises the following specific steps of:
(3-1) parallel structure based on background priors:
the effect of the background is first minimized by eliminating false boundaries to improve the accuracy of the detection. The realization method comprises the following steps:
Figure GDA0002880573290000064
Dcolor(Ip,Iq) Representing the color distance difference between different sides, R/G/B are respectively red, green and blue channels, p and q are any two sides of the four boundaries,
Figure GDA0002880573290000065
is the mean of the characteristics of the three channels on the boundary p,
Figure GDA0002880573290000066
is the mean of the characteristics of the three channels on the boundary q. The 4 x 4 matrix a is obtained by calculating the distance D between all the boundaries and normalized. When a column in the matrix a is maximum, the corresponding boundary has a large characteristic difference from the boundaries of other images, and may contain foreground objects. Sum in maximum column
Figure GDA0002880573290000067
Sum of minimum column
Figure GDA0002880573290000068
Satisfies the conditions
Figure GDA0002880573290000069
When the conditions are met, defining the boundary corresponding to the column and the maximum value as a false boundary, and removing the false boundary; tau isCTo set the threshold, 0.1 ≦ τC1.0. ltoreq. in this example τC＝0.4。
And (3) taking the existing popular ranking algorithm as a support, and respectively performing significance ranking based on the background and the foreground by adopting the general popular ranking algorithm function identification. The method for calculating the saliency map based on the edge background for the other edges after the false boundary filtering is implemented by taking the left boundary as an example, and comprises the following steps:
Figure GDA00028805732900000610
Slfor a significant result calculated with the left boundary of the depth map as the background seed point,
Figure GDA00028805732900000611
is a normalized vector and i denotes a superpixel block index. And calculating the significance result based on other boundaries as the query seed point by adopting the same method.
Finally, the background prior based significance ranking results are:
Figure GDA0002880573290000071
wherein the content of the first and second substances,
Figure GDA0002880573290000072
in correspondence with the remaining boundary(s),
Figure GDA0002880573290000073
representing a significant result graph based on each background edge, and obtaining a significant sequencing result S based on background prior through final multiplicative fusionb(i) In that respect Removing false boundaries may improve the accuracy of salient object detection when one or more boundaries are adjacent to a foreground object.
The same foreground-based significance implementation analyzed using the popularity ranking algorithm is shown as:
Figure GDA0002880573290000074
after the significance maps based on the background and the foreground are respectively obtained, multiplicative fusion is carried out, and the primary significant target area is obtained as follows:
Sinitial(i)＝Sb(i)·Sf(i)。
(3-2) after obtaining a primary significant result, performing feature optimization by taking the foreground area in the graph as a seed point, and redefining a popular ranking function indication vector as follows:
Figure GDA0002880573290000075
and calculating to obtain a final saliency map based on the RGB color features after the saliency features are optimized.
(3-3) significance update in combination with depth information: the contribution of spatial scene position information to saliency is considered in combination with depth information to optimize the initial result. The significance results are updated by using a cell Automata iterative method (SCA), and the method is modified as follows according to the actual situation of the invention:
Figure GDA0002880573290000076
wherein, | | di,djI | represents the depth distance between superpixel blocks i and j, NiIs a neighborhood set of superpixels i, fijRepresenting the similarity, δ, between different superpixel blocks2Indicating a parameter controlling similar intensity, δ2＝0.1。
Based on the depth feature similarity, the significance value of each super pixel is determined by the significance feature value of the super pixel and the neighborhood feature value of the super pixel. The number of iterative propagation times of the cell machine is set to be K times, wherein K is 5-50, and K is 10 in the embodiment, and a more accurate RGB-D significance map is obtained by combining significance updating of depth information. The final test results are shown in fig. 8.
In consideration of the detection results of the invention in different scenes, 5 groups of RGB-D image test sequences in different scenes are also selected for experiments to perform detection by using the stereo video saliency detection method of the invention, and the detection results are shown in fig. 9. The experimental result proves that the method can effectively detect the significant region of the RGB-D stereo image.
The protection content of the present invention is not limited to the above examples. Variations and advantages that may occur to those skilled in the art may be incorporated into the invention without departing from the spirit and scope of the inventive concept, and the scope of the appended claims is intended to be protected.

Claims (9)

1. The RGB-D image significance detection method based on depth reliability analysis is characterized by comprising the following steps of: the method comprises the steps of firstly, evaluating the reliability of the information of the degree of distance of a scene reflected by a depth map to obtain a reliability factor lambda of the depth map, taking lambda as a characteristic index for measuring the significance of a three-dimensional image, carrying out three-dimensional visual significance detection based on the depth information when the lambda is less than 0.45, and carrying out three-dimensional visual significance detection by combining color characteristics when the lambda is more than or equal to 0.45; the method comprises three modules: analyzing depth reliability, performing stereoscopic vision significance detection based on depth information, and performing stereoscopic vision significance detection by combining color characteristics; the method comprises the following specific steps:
(1) first, a depth confidence analysis is performed: inputting a depth image: first using linear iterationThe clustering SLIC superpixel segmentation algorithm carries out image segmentation and constructs a graph G ═ V, E, wherein V represents a node set, and each node in V corresponds to a superpixel block ViThe edge E connects adjacent superpixel blocks; superpixel block v from depth map feature analysisiThe depth confidence of (d) is:
Figure FDA0002880573280000011
wherein m and s represent the mean and standard deviation of pixel values in the whole depth map, respectively, and miRepresenting superpixel blocks viH represents the image entropy,
Figure FDA0002880573280000012
wherein L is the gray level of the depth map, PjIs the probability of the occurrence of the jth gray level in the depth map; c0Is an intensity adjustment factor;
then fuse each superpixel block viDepth confidence value λ ofiObtaining the credibility factor of the whole depth map
Figure FDA0002880573280000013
Wherein N issRepresenting the number of superpixels, Ns∈[100,200,300](ii) a When lambda is less than 0.45, stereoscopic vision significance detection is carried out based on the depth information; when the lambda is more than or equal to 0.45, stereoscopic vision significance detection is carried out by combining color characteristics;
(2) the specific steps of performing stereoscopic vision saliency detection based on depth information are as follows:
(2-1) coarse background filtering: comparing the average depth difference degree of the pixel points in the depth map with the average depth difference degree of the lines where the pixel points are located to obtain a preliminary depth map front background distinction:
Figure FDA0002880573280000014
wherein, IkAnd l'kRespectively representing the depth values of the pixel point k in the original depth image and the coarsely filtered image,
Figure FDA0002880573280000015
representing the average depth value of the line where the pixel point k is located;
(2-2) depth compactness significance analysis: in combination with the depth confidence factor, the superpixel block v is first analyzediDepth-based compactness of (2):
Figure FDA0002880573280000016
where Sdc (vi) represents the compact saliency value, n, of each super-pixel blockjIs a super pixel vjNumber of pixels, λ, contained injRepresenting calculated superpixel blocks vjThe depth confidence value of (a) is,
Figure FDA0002880573280000017
is a super pixel block vjThe coordinates of the center of mass of the image,
Figure FDA0002880573280000018
representing the centroid position of the entire depth image; a isijRepresenting the similarity between two superpixel blocks in the preprocessed depth map,
Figure FDA0002880573280000021
m′iand m'jRepresents superpixel v'iAnd v'jAverage value of (d);
Figure FDA0002880573280000022
to control constant, σ2To control the affinity constant of the similarity matrix;
significance map S for depth compactness calculationcomThe final realization is as follows: scom(vi) 1-norm (Sdc (vi)), norm (. cndot.) as a normalization function, remapping significant values to [0,255 [ (. cndot.) ]]Obtaining a depth compactness saliency map from the range of (1);
(2-3) depth contrast significance analysis: saliency value S of pixel kcon(k) Based on its contrast with all other pixels in the depth image:
Figure FDA0002880573280000023
flrepresenting the frequency of the appearance of different depth values in the depth map after the rough background filtering; wherein D (I'k,I′l)＝||I′k-I′lL, representing image pixel I'kAnd other image pixel points I'lSpatial distance between, pixel point I'kAnd l'lDepth value of [0,255 ]]Within the range;
(2-4) significance of depth compactness result ScomAnd depth contrast saliency result SconAnd (3) fusion, wherein the acquired saliency map based on the depth information is as follows: sdepth＝θScom+(1-θ)Sconθ is a positive control parameter between two significant terms;
(3) the method for detecting the stereoscopic vision significance by combining the color characteristics comprises the following specific steps of:
(3-1) parallel structure based on background priors:
firstly, the influence of the background is minimized by eliminating the false boundary, and the realization method comprises the following steps:
Figure FDA0002880573280000024
Dcolor(Ip,Iq) Representing the color distance difference between different sides, R/G/B are respectively red, green and blue channels, p and q are any two sides of the four boundaries,
Figure FDA0002880573280000025
is the mean of the characteristics of the three channels on the boundary p,
Figure FDA0002880573280000026
is the mean value of the characteristics of the three channels on the boundary q; the 4 x 4 matrix A is obtained by calculating the distances D between all the boundaries and normalized, at the sum of the maximum columns
Figure FDA0002880573280000027
Sum of minimum column
Figure FDA0002880573280000028
Satisfies the conditions
Figure FDA0002880573280000029
When the conditions are met, defining the boundary corresponding to the column and the maximum value as a false boundary, and removing the false boundary; tau isCSetting a threshold value;
then, respectively carrying out significance sorting based on the background and the foreground; calculating a saliency map based on the edge background for other edges after filtering the false boundaries;
finally, the background prior based significance ranking results are:
Figure FDA00028805732800000210
wherein the content of the first and second substances,
Figure FDA00028805732800000211
in correspondence with the remaining boundary(s),
Figure FDA00028805732800000212
representing a significant result graph based on each background edge, and obtaining a significant sequencing result S based on background prior through final multiplicative fusionb(i) (ii) a The significance implementation of the analysis based on the foreground is expressed as:
Figure FDA00028805732800000213
after the significance maps based on the background and the foreground are respectively obtained, multiplicative fusion is carried out, and the primary significant target area is obtained as follows:
Sinitial(i)＝Sb(i)·Sf(i)；
(3-2) after obtaining a primary significant result, performing feature optimization by taking the foreground area in the graph as a seed point, and redefining a popular ranking function indication vector as follows:
Figure FDA0002880573280000031
calculating to obtain final saliency based on RGB color features after the saliency feature optimizationA sex map;
(3-3) significance update in combination with depth information:
updating the significance result by using a cellular automaton iteration method, and modifying the method according to the actual situation:
Figure FDA0002880573280000032
wherein, | | di,djI | represents the depth distance between superpixel blocks i and j, NiIs a neighborhood set of superpixels i, fijRepresenting the similarity, δ, between different superpixel blocks2A parameter indicating control of similar intensity;
based on the depth feature similarity, the significance value of each super pixel is determined by the significance feature value of each super pixel and the neighborhood feature value of each super pixel; and setting the iterative propagation times of the cell machine as K times, and obtaining a more accurate RGB-D significance map by combining significance updating of depth information.
2. The RGB-D image saliency detection method based on depth credibility analysis as claimed in claim 1, characterized by: 1 is less than or equal to C0≤10。
3. The RGB-D image saliency detection method based on depth credibility analysis as claimed in claim 1, characterized by:
Figure FDA0002880573280000033
4. the RGB-D image saliency detection method based on depth credibility analysis as claimed in claim 1, characterized by: sigma2＝0.1。
5. The RGB-D image saliency detection method based on depth credibility analysis as claimed in claim 1, characterized by: n is a radical ofs＝200。
6. The RGB-D image saliency detection method based on depth credibility analysis as claimed in claim 1, characterized by: θ is 0.5.
7. The RGB-D image saliency detection method based on depth credibility analysis as claimed in claim 1, characterized by: tau is not less than 0.1C≤1.0。
8. The RGB-D image saliency detection method based on depth credibility analysis as claimed in claim 1, characterized by: delta2＝0.1。
9. The RGB-D image saliency detection method based on depth credibility analysis as claimed in claim 1, characterized by: k is 5-50.
CN201910298984.9A 2019-04-15 2019-04-15 RGB-D image significance detection method based on depth reliability analysis Active CN110189294B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201910298984.9A CN110189294B (en) 2019-04-15 2019-04-15 RGB-D image significance detection method based on depth reliability analysis

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201910298984.9A CN110189294B (en) 2019-04-15 2019-04-15 RGB-D image significance detection method based on depth reliability analysis

Publications (2)

Publication Number Publication Date
CN110189294A CN110189294A (en) 2019-08-30
CN110189294B true CN110189294B (en) 2021-05-07

Family

ID=67714177

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201910298984.9A Active CN110189294B (en) 2019-04-15 2019-04-15 RGB-D image significance detection method based on depth reliability analysis

Country Status (1)

Country Link
CN (1) CN110189294B (en)

Families Citing this family (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110992304B (en) * 2019-10-30 2023-07-07 浙江力邦合信智能制动***股份有限公司 Two-dimensional image depth measurement method and application thereof in vehicle safety monitoring
CN111028259B (en) * 2019-11-15 2023-04-28 广州市五宫格信息科技有限责任公司 Foreground extraction method adapted through image saliency improvement
CN111353508A (en) * 2019-12-19 2020-06-30 华南理工大学 Saliency detection method and device based on RGB image pseudo-depth information
CN111476767B (en) * 2020-04-02 2022-04-12 南昌工程学院 High-speed rail fastener defect identification method based on heterogeneous image fusion
CN111709938B (en) * 2020-06-18 2023-07-07 武汉唯理科技有限公司 Pavement defect and casting detection method based on depth map
CN111881925B (en) * 2020-08-07 2023-04-18 吉林大学 Significance detection method based on camera array selective light field refocusing
CN114998320B (en) * 2022-07-18 2022-12-16 银江技术股份有限公司 Method, system, electronic device and storage medium for visual saliency detection

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103914834A (en) * 2014-03-17 2014-07-09 上海交通大学 Significant object detection method based on foreground priori and background priori
CN105761238A (en) * 2015-12-30 2016-07-13 河南科技大学 Method of extracting saliency target through gray statistical data depth information
CN105869173A (en) * 2016-04-19 2016-08-17 天津大学 Stereoscopic vision saliency detection method
CN106952301A (en) * 2017-03-10 2017-07-14 安徽大学 RGB-D image significance calculation method
CN107085848A (en) * 2017-04-20 2017-08-22 安徽大学 Method for detecting significance of RGB-D (Red, Green and blue-D) image
CN108470178A (en) * 2018-02-07 2018-08-31 杭州电子科技大学 A kind of depth map conspicuousness detection method of the combination depth trust evaluation factor

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160189419A1 (en) * 2013-08-09 2016-06-30 Sweep3D Corporation Systems and methods for generating data indicative of a three-dimensional representation of a scene
US9984473B2 (en) * 2014-07-09 2018-05-29 Nant Holdings Ip, Llc Feature trackability ranking, systems and methods
CN108154150B (en) * 2017-12-18 2021-07-23 北京工业大学 Significance detection method based on background prior
CN109255357B (en) * 2018-08-03 2021-09-17 天津大学 RGBD image collaborative saliency detection method

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103914834A (en) * 2014-03-17 2014-07-09 上海交通大学 Significant object detection method based on foreground priori and background priori
CN105761238A (en) * 2015-12-30 2016-07-13 河南科技大学 Method of extracting saliency target through gray statistical data depth information
CN105869173A (en) * 2016-04-19 2016-08-17 天津大学 Stereoscopic vision saliency detection method
CN106952301A (en) * 2017-03-10 2017-07-14 安徽大学 RGB-D image significance calculation method
CN107085848A (en) * 2017-04-20 2017-08-22 安徽大学 Method for detecting significance of RGB-D (Red, Green and blue-D) image
CN108470178A (en) * 2018-02-07 2018-08-31 杭州电子科技大学 A kind of depth map conspicuousness detection method of the combination depth trust evaluation factor

Non-Patent Citations (4)

* Cited by examiner, † Cited by third party
Title
RGB-D Salient Object Detection via Minimum Barrier Distance Transform and Saliency Fusion;Anzhi Wang et al.;《IEEE SIGNAL PROCESSING LETTERS》;20170531;第24卷(第5期);第663-667页 *
Robust Saliency Detection via Regularized Random Walks Ranking;Changyang Li et al.;《CVPR2015》;20151231;第2710-2717页 *
Saliency Detection for Stereoscopic Images Based on Depth Confidence Analysis and Multiple Cues Fusion;Runmin Cong et al.;《arXiv:1710.05174v1》;20171014;第1-5页 *
Saliency Detection via Graph-Based Manifold Ranking;Chuan Yang et al.;《2013 IEEE Conference Computer Vision and Pattern Recognition》;20130628;第1-8页 *

Also Published As

Publication number Publication date
CN110189294A (en) 2019-08-30

Similar Documents

Publication Publication Date Title
CN110189294B (en) RGB-D image significance detection method based on depth reliability analysis
CN107767413B (en) Image depth estimation method based on convolutional neural network
CN108648161B (en) Binocular vision obstacle detection system and method of asymmetric kernel convolution neural network
US8953874B2 (en) Conversion of monoscopic visual content using image-depth database
CN104850850B (en) A kind of binocular stereo vision image characteristic extracting method of combination shape and color
CN108470178B (en) Depth map significance detection method combined with depth credibility evaluation factor
CN111480183B (en) Light field image rendering method and system for generating perspective effect
CN105740775A (en) Three-dimensional face living body recognition method and device
CN108257165B (en) Image stereo matching method and binocular vision equipment
US9769460B1 (en) Conversion of monoscopic visual content to stereoscopic 3D
CN110827312B (en) Learning method based on cooperative visual attention neural network
WO2018053952A1 (en) Video image depth extraction method based on scene sample library
CN111507183B (en) Crowd counting method based on multi-scale density map fusion cavity convolution
CN106997478B (en) RGB-D image salient target detection method based on salient center prior
CN113963032A (en) Twin network structure target tracking method fusing target re-identification
CN112884682A (en) Stereo image color correction method and system based on matching and fusion
CN112365586B (en) 3D face modeling and stereo judging method and binocular 3D face modeling and stereo judging method of embedded platform
CN112288758B (en) Infrared and visible light image registration method for power equipment
WO2022116104A1 (en) Image processing method and apparatus, and device and storage medium
CN110070574A (en) A kind of binocular vision Stereo Matching Algorithm based on improvement PSMNet
CN113592018B (en) Infrared light and visible light image fusion method based on residual dense network and gradient loss
CN113159043A (en) Feature point matching method and system based on semantic information
CN108388901B (en) Collaborative significant target detection method based on space-semantic channel
CN114648482A (en) Quality evaluation method and system for three-dimensional panoramic image
CN111882516B (en) Image quality evaluation method based on visual saliency and deep neural network

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant
TR01 Transfer of patent right
TR01 Transfer of patent right

Effective date of registration: 20231008

Address after: Room 701, Building 5, No. 643 Shuangliu, Xihu District, Hangzhou City, Zhejiang Province, 310000

Patentee after: HANGZHOU EYECLOUD TECHNOLOGY Co.,Ltd.

Address before: 310018 No. 2 street, Xiasha Higher Education Zone, Hangzhou, Zhejiang

Patentee before: HANGZHOU DIANZI University