CN110223252A - A kind of depth image reparation algorithm based on compound adaptive region growth criterion - Google Patents

A kind of depth image reparation algorithm based on compound adaptive region growth criterion Download PDF

Info

Publication number
CN110223252A
CN110223252A CN201910495178.0A CN201910495178A CN110223252A CN 110223252 A CN110223252 A CN 110223252A CN 201910495178 A CN201910495178 A CN 201910495178A CN 110223252 A CN110223252 A CN 110223252A
Authority
CN
China
Prior art keywords
point
pixel
repaired
depth
image
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201910495178.0A
Other languages
Chinese (zh)
Other versions
CN110223252B (en
Inventor
陈光柱
陈正阳
侯睿
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Chengdu Univeristy of Technology
Original Assignee
Chengdu Univeristy of Technology
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Chengdu Univeristy of Technology filed Critical Chengdu Univeristy of Technology
Priority to CN201910495178.0A priority Critical patent/CN110223252B/en
Publication of CN110223252A publication Critical patent/CN110223252A/en
Application granted granted Critical
Publication of CN110223252B publication Critical patent/CN110223252B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00Image enhancement or restoration
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00Image analysis
    • G06T7/10Segmentation; Edge detection
    • G06T7/187Segmentation; Edge detection involving region growing; involving region merging; involving connected component labelling
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00Image analysis
    • G06T7/30Determination of transform parameters for the alignment of images, i.e. image registration
    • G06T7/33Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00Image analysis
    • G06T7/80Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
    • G06T7/85Stereo camera calibration
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/10Image acquisition modality
    • G06T2207/10024Color image
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/10Image acquisition modality
    • G06T2207/10028Range image; Depth image; 3D point clouds
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/20Special algorithmic details
    • G06T2207/20172Image enhancement details
    • G06T2207/20192Edge enhancement; Edge preservation

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Image Processing (AREA)
  • Image Analysis (AREA)

Abstract

The invention proposes a kind of depth images based on compound adaptive region growth criterion to repair algorithm, which is proposed the calculation method of confidence item and Weighting type priority, the reparation order of depth image is determined with Weighting type priority;Under the guidance of Weighting type priority, criterion is grown by compound adaptive region, the pixel more like with point feature to be repaired is found and constructs vertex neighborhood to be repaired;Under the guidance of RGB image, the estimation of Depth of complex point to be repaired is determined with the linear weighted combination of pixel in vertex neighborhood to be repaired.The algorithm can obtain higher reparation precision in the case where keeping the complete situation of boundary information.

Description

A kind of depth image reparation algorithm based on compound adaptive region growth criterion
Technical field
The present invention relates to depth image reparations, using the reparation order of Weighting type priority guidance depth image, and utilize RGB image information finds neighborhood of a point to be repaired, and the bilateral weighted by repairing pixel in vertex neighborhood obtains complex point to be repaired Estimation of Depth repairs the pixel missing of depth image with this.
Background technique
Being constantly progressive and develop with the epoch, Digital image technology is all widely used in various fields.Its As the important component in computer graphics and machine vision, accurate image information often determines final acquired As a result accuracy, and usually there is the problems such as fuzzy or even damaged in most of image, image restoration technology becomes therewith to be worked as Preceding research hotspot.The appearance of depth information further enriches the neighborhood that machine vision is related to, and depth image refers to quilt Surveying object indicates image generated to the distance between depth camera with pixel value, which describes object in three-dimensional space Between in position and characteristic information.Depth information is in 3 D scene rebuilding, human-computer interaction, target following, free viewpoint video wash with watercolours The various aspects such as dye, augmented reality have vital effect.
The progress and development of optical technology make it is more convenient it is more universal using depth sensing equipment sampling depth information at It is possible.Up to the present, depth sensing equipment substantially can be divided into passive depth according to the acquisition principle of depth information to pass Sense and two class of active depth sensing.Active depth sensing mainly obtains required information using the transmitting-receiving of detectable signal, Laser ranging is based on the flight time, based on technologies such as structure lights.Based on structure light and based on the sensing technology of flight time Then the new lover in development as depth sensing technology all has portability although the measuring principle of the two is less identical Various advantages such as height, robustness are high, image taking speed is fast, and the dramatic of scene can be obtained in real time, it is adapted to It is such in more applications, but because of various influences such as hardware condition restriction, environment influence and measuring principle restrictions Depth camera depth image obtained has a certain defect, includes apparent noise more, cannot complete reactant feature or React wrong.Therefore, it probes into the degeneration of depth image and more reasonable reparation depth image becomes inevitable.
In terms of depth image repairing research, although domestic and foreign scholars propose numerous correcting strategies, such as more images Fusion, machine learning, depth image and combination of intensity image etc..But these methods or unsuitable dynamic scene or preceding Set that work is sufficiently complex, or because excessively high time complexity is dfficult to apply to actual scene.
Summary of the invention
Weighting type priority is introduced into neighborhood and repaired in algorithm by the present invention, real as the criterion for determining image repair order The correct extension of image texture is showed.In the case where correctly repairing order, propose that comprehensively consider pixel similarity similar with texture Property compound adaptive region grow criterion, with obtain with the higher vertex neighborhood to be repaired of points correspondence degree to be repaired.Pass through The bilateral weighted of pixel obtains the depth value estimated value of complex point to be repaired in vertex neighborhood to be repaired.
Innovative point of the invention first is that the proposition of Weighting type priority calculation method.Consider Criminisi priority Deficiency, to improve the robustness that priority calculates, new confidence item and priority calculation method are as follows:
(1) the confidence item of p point indicates that calculation method is as follows with C (p):
In formula, ψpFor p neighborhood of a point, q is neighborhood ψpIn non-empty pixel, Φ is non-empty pixel in depth image The set of point, (xp,yp), (xq,yq) it is respectively p, the pixel coordinate of q two o'clock.
(2) mentioned confidence item in (1) is utilized, the calculation expression of priority P (p) is as follows:
P (p)=β C (p)+γ D (p)
In formula, D (p) is the data item of p point, and β and γ are the coefficient of balance of confidence item and data item.
Another innovation of the invention is a kind of building of new vertex neighborhood to be repaired.Utilize pixel value tag and office The constraint of portion's textural characteristics, compound adaptive region growth criterion are as follows:
In formula, m is corresponding pixel after p point is mapped in RGB image, and as initial seed point, Ω is the neighbour of m point The set put in domain, τ are a threshold value, dist, Х(m,n)It is shown below:
In formula, R, G, B are the triple channel of RGB image,WithRespectively represent the pixel value of m point and the channel n point c, (xn, yn) be n point pixel coordinate, (xm,ym) be m point pixel coordinate, κ and λ are weight.
In formula, ULBPmAnd ULBPnFor the uniform LBP code of m, n two o'clock.
The utility model has the advantages that priority calculation method proposed by the present invention, improves the shortcoming of Criminisi priority, So that the reparation order of image is more reasonable, the correct propagation of image texture characteristic in repair process ensure that.And the present invention mentions The construction method of the adaptive neighborhood based on region growing out, can be improved the reparation precision of image, and inhibit object edge The blurring for repairing result, guarantees the accuracy of image feature information.
Detailed description of the invention
Fig. 1: the depth image based on compound adaptive region growth criterion repairs algorithm structure block diagram
Fig. 2: the depth image based on compound adaptive region growth criterion repairs algorithm flow chart
Specific embodiment
1 to attached drawing 2 with reference to the accompanying drawing, and the invention will be further described.
With reference to attached drawing 1, the present invention is firstly the need of obtaining depth image and corresponding the RGB image after being registrated, to depth All empty pixels, which do priority and calculate to obtain, in degree image has top-priority pixel as complex point to be repaired, passes through Compound adaptive region growth criterion obtains neighborhood of a point to be repaired, is estimated with repairing the bilateral weighted of pixel in vertex neighborhood Complex point depth value to be repaired, final updating hole region repeat step to repair complete.
The algorithm includes the following steps:
S1: calibration RGB-D camera obtains internal reference and the outer ginseng of RGB camera and depth camera, complete using pinhole imaging system principle At the registration of depth image and RGB image;
S2: traversal depth image finds all empty pixels, obtains priority by Weighting type priority calculation method Highest cavity pixel obtains initial seed point in RGB image after complex point to be repaired to be mapped to registration as complex point to be repaired;
S3: criterion is grown using compound adaptive region, indexes the pixel in RGB image in initial seed neighborhood of a point The pixel for meeting compound adaptive region growth criterion is included into a set by point, by the compound mapping into depth image, Obtain neighborhood of a point to be repaired.
S4: complex point to be repaired in depth image and neighborhood of a point to be repaired are mapped in RGB image together, obtain corresponding picture The codomain weight of vegetarian refreshments obtains the estimating depth value of complex point to be repaired with the bilateral weighted of codomain and spatial domain;
S5: repeating step S2-S4, until all empty pixels be repaired it is complete.
With reference to attached drawing 2, the depth image based on compound adaptive region growth criterion repairs algorithm, and specific step is as follows:
Step S1 specifically comprises the following steps:
S11: it demarcates to obtain the internal reference K of depth camera and RGB camera by Zhang Shi1、K2With outer ginseng R1、R2(camera coordinates arrive The spin matrix of world coordinates experience), T1、T2(translation vector that camera coordinates to world coordinates are undergone).
S12: the mapping relations between depth camera coordinate are obtained according to pinhole imaging system principle:
In formula, P1、P2For same point under world coordinates in two camera coordinates corresponding point.
S13: according to camera imaging model, pass through the mapping between the available pixel coordinate of camera internal reference and camera coordinates Relationship:
In formula, p1、p2For this o'clock in two pixel coordinates corresponding point.
S14: the mapping relations between depth image and RGB image can be obtained by formula (1) (2):
Step S2 specifically comprises the following steps:
S21: traversal depth image finds all pixels point that pixel value is 0.
S22: by the RGB image gray processing after registration and seeking gradient information, calculates empty pixel using gradient information Data item calculates the priority of empty pixel using designed confidence item and Weighting type priority calculation method.It finds excellent Maximum cavity pixel p is first weighed as complex point to be repaired.Designed confidence item C (p) and Weighting type priority P (p) calculation method It is as follows:
In formula, ψpFor p neighborhood of a point, q is neighborhood ψpIn non-empty pixel, Φ is non-empty pixel in depth image The set of point, (xp,yp), (xq,yq) it is respectively p, the pixel coordinate of q two o'clock.
P (p)=β C (p)+γ D (p) (5)
In formula, D (p) is the data item of p point, and β and γ are the coefficient of balance of confidence item and data item.
In formula,Indicate p point isophote direction, npIndicate p point normal vector, α is normalization factor, for 8-bit amount α=255 for the digital picture of change.
S23: in the RGB image after p point to be mapped to registration, using corresponding pixel as initial seed point.
Step S3 specifically comprises the following steps:
S31: doing region growing to initial seed point, in the case where compound adaptive region grows criterion to initial seed vertex neighborhood Pixel makes a decision, and the pixel for meeting criterion incorporates into a set.Compound adaptive region growth criterion is as follows:
In formula, m is corresponding pixel after p point is mapped in RGB image, and as initial seed point, Ω is the neighbour of m point The set put in domain, τ are a threshold value, dist, Х(m,n)It is shown below:
In formula, R, G, B are the triple channel of RGB image,WithRespectively represent the pixel value of m point and the channel n point c, (xn, yn) be n point pixel coordinate, (xm,ym) be m point pixel coordinate, κ and λ are weight.
In formula, ULBPmAnd ULBPnFor the uniform LBP code of m, n two o'clock.
S32: using the pixel for meeting criterion in step S31 as new seed point, continue in new seed neighborhood of a point It finds the pixel for meeting compound adaptive region growth criterion and is included into identity set, until meeting termination condition.It terminates Condition is described as follows:
1) when being grown again using the pixel obtained for meeting growth criterion as seed point, in seed vertex neighborhood It inside can not find the pixel that can satisfy criterion.
2) the pixel number for meeting growth criterion has reached a upper limit ρ.
S33: the region growing pixel collection obtained for meeting adaptive region growth criterion is mapped to depth map As in, neighborhood of a point to be repaired is obtained.
Step S4 specifically comprises the following steps:
S41: by RGB image information, the codomain weight W of pixel in vertex neighborhood to be repaired is obtainedrAnd spatial domain weight Wd, as shown in formula 10,11.
In formula, I is the pixel value of corresponding pixel points in RGB image, and (x, y) is the pixel coordinate of p point, (i, j) is that p point is adjacent The pixel coordinate of pixel, σ in domainrSize control pixel because of the size with p point value differences cause codomain weight Change number.
In formula, σdSize control be used to estimate the size of neighborhood of a point to be repaired.
S42: according to the weight of pixel in the available vertex neighborhood to be repaired of formula 10,11:
W=Wr·Wd (12)
Linear weighted function is done to pixel in vertex neighborhood to be repaired and obtains the estimation of Depth value D of complex point to be repairedp':
In formula, Dp、DqFor the depth pixel value of pixel in complex point p to be repaired and p vertex neighborhood.
Experimental result and analysis:
To verify the performance that the depth image of the present invention for growing criterion based on compound adaptive region repairs algorithm, adopt Use the depth image data collection in Middlebury as identifying object, with root-mean-square error (RMSE) and Y-PSNR (PSRN) it is bilateral to compare reparation algorithm based on Navier_Stokes equation, joint for the standard that result superiority and inferiority is repaired as evaluation Filtering algorithm and algorithm of the present invention, Art etc. are the different scenes image in data set, and the results are shown in Table 1 for reparation. For multiple depth scenes, the present invention all has preferably RMSE and PSRN, and algorithm has robustness.
The different algorithm reparation results of repairing of table 1 compare
It is verified through emulation experiment, the present invention achieves ideal repairing effect in depth image reparation.
The above content is a further detailed description of the present invention in conjunction with specific preferred embodiments, and it cannot be said that Specific implementation of the invention is only limited to these instructions.For those of ordinary skill in the art to which the present invention belongs, exist Under the premise of not departing from present inventive concept, several simple deduction or replace can also be made, all shall be regarded as belonging to of the invention Protection scope.

Claims (3)

1. a kind of depth image based on compound adaptive region growth criterion repairs algorithm, it is characterised in that: the algorithm includes Following steps:
S1: calibration RGB-D camera obtains internal reference and the outer ginseng of RGB camera and depth camera, is completed using pinhole imaging system principle deep Spend the registration of image and RGB image;
S2: traversal depth image finds all empty pixels, obtains priority highest by Weighting type priority calculation method Empty pixel as complex point to be repaired, obtain initial seed point in the RGB image after complex point to be repaired to be mapped to registration;
S3: growing criterion using compound adaptive region, index the pixel in RGB image in initial seed neighborhood of a point, will The pixel for meeting compound adaptive region growth criterion is included into a set, by the compound mapping into depth image, is obtained Neighborhood of a point to be repaired.
S4: complex point to be repaired in depth image and neighborhood of a point to be repaired are mapped in RGB image together, obtain corresponding pixel points Codomain weight, the estimating depth value of complex point to be repaired is obtained with the bilateral weighted of codomain and spatial domain;
S5: repeating step S2-S4, until all empty pixels be repaired it is complete.
2. a kind of depth image based on compound adaptive region growth criterion according to claim 1 repairs algorithm, It is characterized in that: Weighting type priority calculation method in the step S2:
(1) complex point to be repaired in depth image, is indicated with p, and the confidence item of p point indicates that calculation method is as follows with C (p):
In formula, ψpFor p neighborhood of a point, q is neighborhood ψpIn non-empty pixel, Φ is non-empty pixel in depth image Set, (xp,yp), (xq,yq) it is respectively p, the pixel coordinate of q two o'clock.
(2) mentioned confidence item C (p) in (1) is utilized, the calculation expression of priority P (p) is as follows:
P (p)=β C (p)+γ D (p)
In formula, D (p) is the data item of p point, and β and γ are the coefficient of balance of confidence item and data item.
3. a kind of depth image based on compound adaptive region growth criterion according to claim 1 repairs algorithm, It is characterized in that: compound adaptive region growth criterion, expression formula in the step S3 are as follows:
In formula, m is corresponding pixel after p point is mapped in RGB image, and as initial seed point, Ω is in m neighborhood of a point The set of point, τ are a threshold value, dist, Х(m,n)It is shown below:
In formula, R, G, B are the triple channel of RGB image,WithRespectively represent the pixel value of m point and the channel n point c, (xn,yn) it is n The pixel coordinate of point, (xm,ym) be m point pixel coordinate, κ and λ are weight.
In formula, ULBPmAnd ULBPnFor the uniform LBP code of m, n two o'clock.
CN201910495178.0A 2019-06-10 2019-06-10 Depth image restoration algorithm based on composite self-adaptive region growing criterion Active CN110223252B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201910495178.0A CN110223252B (en) 2019-06-10 2019-06-10 Depth image restoration algorithm based on composite self-adaptive region growing criterion

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201910495178.0A CN110223252B (en) 2019-06-10 2019-06-10 Depth image restoration algorithm based on composite self-adaptive region growing criterion

Publications (2)

Publication Number Publication Date
CN110223252A true CN110223252A (en) 2019-09-10
CN110223252B CN110223252B (en) 2022-12-06

Family

ID=67815923

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201910495178.0A Active CN110223252B (en) 2019-06-10 2019-06-10 Depth image restoration algorithm based on composite self-adaptive region growing criterion

Country Status (1)

Country Link
CN (1) CN110223252B (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114066779A (en) * 2022-01-13 2022-02-18 杭州蓝芯科技有限公司 Depth map filtering method and device, electronic equipment and storage medium

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20100278426A1 (en) * 2007-12-14 2010-11-04 Robinson Piramuthu Systems and methods for rule-based segmentation for objects with full or partial frontal view in color images
US20120313937A1 (en) * 2010-01-18 2012-12-13 Disney Enterprises, Inc. Coupled reconstruction of hair and skin
CN103996174A (en) * 2014-05-12 2014-08-20 上海大学 Method for performing hole repair on Kinect depth images
CN107316283A (en) * 2017-07-14 2017-11-03 天津大学 Algorithm is repaired based on the digital picture that multiwindow is merged

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20100278426A1 (en) * 2007-12-14 2010-11-04 Robinson Piramuthu Systems and methods for rule-based segmentation for objects with full or partial frontal view in color images
US20120313937A1 (en) * 2010-01-18 2012-12-13 Disney Enterprises, Inc. Coupled reconstruction of hair and skin
CN103996174A (en) * 2014-05-12 2014-08-20 上海大学 Method for performing hole repair on Kinect depth images
CN107316283A (en) * 2017-07-14 2017-11-03 天津大学 Algorithm is repaired based on the digital picture that multiwindow is merged

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
YING LIU等: "A Novel Exemplar-based Image Inpainting Algorithm", 《2015 INTERNATIONAL CONFERENCE ON INTELLIGENT NETWORKING AND COLLABORATIVE SYSTEMS》 *
兰小丽等: "基于纹理块与梯度特征的图像修复改进算法", 《计算机工程与应用》 *
吴亚娟等: "图像修复算法中匹配点搜索和修复顺序研究", 《西华师范大学学报(自然科学版)》 *

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114066779A (en) * 2022-01-13 2022-02-18 杭州蓝芯科技有限公司 Depth map filtering method and device, electronic equipment and storage medium

Also Published As

Publication number Publication date
CN110223252B (en) 2022-12-06

Similar Documents

Publication Publication Date Title
CN1871622B (en) Image collation system and image collation method
CN114782691B (en) Robot target identification and motion detection method based on deep learning, storage medium and equipment
US11830216B2 (en) Information processing apparatus, information processing method, and storage medium
KR100831187B1 (en) Object posture estimation/correlation system using weight information
CN109974743B (en) Visual odometer based on GMS feature matching and sliding window pose graph optimization
CN108010081B (en) RGB-D visual odometer method based on Census transformation and local graph optimization
CN111723691B (en) Three-dimensional face recognition method and device, electronic equipment and storage medium
CN111462207A (en) RGB-D simultaneous positioning and map creation method integrating direct method and feature method
CN110766024B (en) Deep learning-based visual odometer feature point extraction method and visual odometer
CN107563323A (en) A kind of video human face characteristic point positioning method
JP7479324B2 (en) Information processing device, information processing method, and program
CN114677323A (en) Semantic vision SLAM positioning method based on target detection in indoor dynamic scene
CN111998862A (en) Dense binocular SLAM method based on BNN
CN110428461B (en) Monocular SLAM method and device combined with deep learning
CN114612393A (en) Monocular vision-based reflective part pose estimation method
CN110223252A (en) A kind of depth image reparation algorithm based on compound adaptive region growth criterion
CN105339981B (en) Method for using one group of primitive registration data
KR101766823B1 (en) Robust visual odometry system and method to irregular illumination changes
CN113642397A (en) Object length measuring method based on mobile phone video
CN113720323B (en) Monocular vision inertial navigation SLAM method and device based on point-line feature fusion
CN109377524A (en) A kind of single image depth recovery method and system
CN114972937A (en) Feature point detection and descriptor generation method based on deep learning
CN110211239B (en) Augmented reality method, apparatus, device and medium based on label-free recognition
Liu et al. Visual odometry algorithm based on deep learning
CN113256711A (en) Pose estimation method and system of monocular camera

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant