CN108961176B - Self-adaptive bilateral reference restoration method for range-gated three-dimensional imaging - Google Patents
Self-adaptive bilateral reference restoration method for range-gated three-dimensional imaging Download PDFInfo
- Publication number
- CN108961176B CN108961176B CN201810611102.5A CN201810611102A CN108961176B CN 108961176 B CN108961176 B CN 108961176B CN 201810611102 A CN201810611102 A CN 201810611102A CN 108961176 B CN108961176 B CN 108961176B
- Authority
- CN
- China
- Prior art keywords
- value
- image
- depth image
- depth
- neighborhood
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000002146 bilateral effect Effects 0.000 title claims abstract description 47
- 238000003384 imaging method Methods 0.000 title claims abstract description 39
- 238000000034 method Methods 0.000 title claims abstract description 32
- 238000009499 grossing Methods 0.000 claims abstract description 15
- 238000001914 filtration Methods 0.000 claims abstract description 7
- 230000003044 adaptive effect Effects 0.000 claims description 7
- 230000000694 effects Effects 0.000 claims description 7
- 230000009471 action Effects 0.000 claims description 5
- 239000004576 sand Substances 0.000 claims description 5
- 238000010606 normalization Methods 0.000 claims description 2
- 238000005516 engineering process Methods 0.000 abstract description 13
- 238000010586 diagram Methods 0.000 description 3
- 230000001934 delay Effects 0.000 description 2
- 238000005286 illumination Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000003111 delayed effect Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 230000005764 inhibitory process Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000008439 repair process Effects 0.000 description 1
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/70—Denoising; Smoothing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30004—Biomedical image processing
- G06T2207/30008—Bone
Landscapes
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Image Processing (AREA)
Abstract
A method for distance gating three-dimensional imaging self-adaptive bilateral reference restoration comprises the following steps: step 1: firstly, gating adjacent two-dimensional space slice images according to the distance, respectively representing the two-dimensional space slice images by A, B frames, setting a 3D threshold value, and obtaining an original depth image to be restored by using a three-dimensional inversion algorithm; step 2: secondly, partially filling the original depth image and the A frame image with missing data by adopting a self-adaptive bilateral reference algorithm to obtain a partially filled depth image; and step 3: filling missing data completely by adopting a self-adaptive bilateral reference algorithm for the partially filled depth image and the B frame image to obtain a completely filled depth image; and 4, step 4: and finally, smoothing and filtering the completely filled depth image and the (A + B)/2 image by adopting a self-adaptive bilateral reference algorithm to obtain a complete and smooth repaired depth image. The invention can achieve the purposes of effectively complementing the cavity of the depth image acquired by the range gating three-dimensional imaging technology and removing noise.
Description
Technical Field
The invention belongs to the technical field of image processing, and particularly relates to a technology for repairing a range-gated three-dimensional imaging depth image.
Background
The range-gated imaging is an active imaging technology, a pulse laser is used as an illumination light source, a gated imaging device is used as a detector, space slice imaging can be realized by controlling the time delay between laser pulses and gated pulses, the background is filtered, meanwhile, backscattering noises such as atmosphere and the like are inhibited, and target detection is realized.
In fact, each gated image contains not only luminance information, but also distance information. And distance information correlation exists between a plurality of images with the same door width and different delays. The three-dimensional information of the target can be inverted by an algorithm by utilizing the information correlation. At present, three-dimensional imaging algorithms of laser gating mainly comprise three types of stepping delay three-dimensional imaging, gain modulation three-dimensional imaging and distance energy correlation three-dimensional imaging (also called super-resolution three-dimensional imaging).
The stepping time-delay three-dimensional imaging has poor real-time performance due to large amount of original data, and is difficult to be used for online real-time three-dimensional imaging. Gain-modulated three-dimensional imaging is difficult to effectively filter out background noise and device noise because two slices are obtained under different system gains. The distance energy correlation three-dimensional imaging algorithm can calculate the distance of a target by utilizing the relationship between the distance and the energy by only modifying the delay to obtain gating images under different delays, has low complexity and less data volume (at least two frames), has smaller three-dimensional reconstruction subinterval compared with a gain modulation three-dimensional imaging algorithm and higher ranging precision under the same parameters, and therefore becomes a main technical means of online real-time laser gating three-dimensional imaging in practical application in recent years.
Although the distance gating three-dimensional imaging technology has the characteristics of high spatial resolution, long action distance and effective inhibition of backscattering of atmosphere or water, the problems of short action distance and low spatial resolution existing in the traditional three-dimensional imaging can be solved, and target three-dimensional information with large depth of field and high distance resolution can be quickly acquired in real time, the method acquires a depth image or a three-dimensional point cloud image which still has a plurality of places to be repaired, and is mainly embodied in that hollow areas (lack depth data) exist in the three-dimensional image and certain noise exists due to the fact that the optical reflection characteristics of the target and the distribution of illumination light intensity are not uniform. The depth image holes and noise can reduce the distance resolution of the three-dimensional information of the acquired target, and simultaneously can reduce the action distance of the range gating three-dimensional imaging technology, so that the three-dimensional information of the technology can be blurred when the technology acquires a fine structure target and a long-distance target, and the acquired data can be completely wrong seriously. Therefore, filling the cavity area and removing the noise of the depth image obtained by the range-gated three-dimensional imaging are problems which need to be solved urgently.
At present, an algorithm for repairing a depth image mainly aims at the depth image obtained by binocular stereo vision, a structured light method, a time flight algorithm and the like, and no algorithm for repairing the depth image completely aims at a distance gating three-dimensional imaging technology.
Disclosure of Invention
Aiming at the defects in the prior art, the invention mainly aims to provide a self-adaptive bilateral reference restoration method so as to achieve the purposes of effectively complementing the cavity of a depth image acquired by a range-gated three-dimensional imaging technology and removing noise.
In order to achieve the purpose, the technical scheme provided by the invention is as follows:
a method for distance gating three-dimensional imaging self-adaptive bilateral reference restoration comprises the following steps:
step 1: firstly, gating adjacent two-dimensional space slice images according to the distance, respectively representing the two-dimensional space slice images by A, B frames, setting a 3D threshold value, and obtaining an original depth image to be restored by using a three-dimensional inversion algorithm;
step 2: secondly, partially filling the original depth image and the A frame image with missing data by adopting a self-adaptive bilateral reference algorithm to obtain a partially filled depth image;
and step 3: filling missing data completely by adopting a self-adaptive bilateral reference algorithm for the partially filled depth image and the B frame image to obtain a completely filled depth image;
and 4, step 4: and finally, smoothing and filtering the completely filled depth image and the (A + B)/2 image by adopting a self-adaptive bilateral reference algorithm to obtain a complete and smooth repaired depth image.
According to the technical scheme, the invention has the following beneficial effects:
by utilizing the method, the value of weighted average of all depth data in the neighborhood of the pixel point to be repaired is assigned to the pixel point to be repaired in the proposed self-adaptive bilateral reference algorithm, and if the pixel point to be repaired has the depth data, the assigned value plays a role in smooth filtering; if the depth data of the pixel point to be repaired is missing, the given value plays a role in complementing the depth data. Therefore, the self-adaptive bilateral reference algorithm provided by the invention can be used for filling the cavity and removing noise.
By using the method, when the weighted average of each depth data in the neighborhood is calculated, the weight coefficient is composed of the depth image space domain weight and the gray domain weight of the two-dimensional slice image, and the two-dimensional slice image is a depth image and a base and contains rich target texture information. Therefore, compared with the method only operating the depth image, the data filled or smoothed by the self-adaptive bilateral reference algorithm provided by the invention is closer to the real data of the target.
By using the method and the device, the 3D inversion threshold of the two-dimensional slice image is referred when the depth data is supplemented, and the depth data is supplemented when the gray value of the same position of the two-dimensional slice image corresponding to the pixel point of the depth data to be supplemented is greater than the 3D inversion threshold. Therefore, when the self-adaptive bilateral reference algorithm is used for filling the cavity, the position which is really required to be filled can be filled more accurately, and the phenomenon of over filling or under filling is reduced as much as possible.
By using the method and the device, the self-adaptive bilateral reference algorithm is used for filling the hole only at the position where the depth data is missing and needs to be supplemented, and the self-adaptive bilateral reference algorithm is used for smoothing the depth data only at the position where the depth data exists. Therefore, the self-adaptive bilateral reference algorithm provided by the invention can reduce the pixel points of the operation and has higher repair speed.
By using the method, as for each depth data to be restored, the neighborhood window value, the airspace weight and the value domain weight can be self-adaptive, compared with the algorithm of the fixed window value, the fixed airspace weight and the fixed value domain weight, the self-adaptive bilateral reference algorithm provided by the invention, the filled or smooth depth data is closer to the real three-dimensional information of the target.
Drawings
For a better understanding of the objects, aspects and advantages of the present invention, reference is made to the following detailed description of the invention, which is to be read in connection with the accompanying drawings, wherein:
FIG. 1 is a schematic flow chart of a method for range-gated three-dimensional imaging adaptive bilateral reference restoration according to the present invention;
FIG. 2 is a schematic flow chart of step 2-3 in FIG. 1;
FIG. 3 is a schematic flow chart of step 4 in FIG. 1;
FIG. 4 is a two-dimensional slice image according to an embodiment of the invention, in which: (a) a two-dimensional slice a frame image, (B) a two-dimensional slice B frame image, (c) a two-dimensional slice (a + B)/2 frame image;
fig. 5 shows a depth image generated by a two-dimensional slice image using a triangle distance energy correlation algorithm and a repaired result thereof according to an embodiment of the present invention, (a) an original depth image, (B) a depth image partially filled according to an a frame, (c) a depth image completely filled according to a B frame, and (d) a repaired depth image after smoothing and denoising.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to specific embodiments and the accompanying drawings. It should be noted that in the drawings or description, the same drawing reference numerals are used for similar or identical parts. Implementations not depicted or described in the drawings are of a form known to those of ordinary skill in the art. Additionally, while exemplifications of parameters including particular values may be provided herein, it is to be understood that the parameters need not be exactly equal to the respective values, but may be approximated to the respective values within acceptable error margins or design constraints.
The invention discloses a method for restoring a depth image by self-adaptive bilateral reference, which aims to effectively compensate holes of the depth image acquired by a range-gated three-dimensional imaging technology and remove noise.
Fig. 1 in the accompanying drawings is a schematic flow chart of a method for distance-gated three-dimensional imaging adaptive bilateral reference restoration according to the present invention, and it can be seen from the diagram that the present invention is mainly divided into 4 parts: and performing three-dimensional inversion to generate an original depth image to be restored, referring to the A frame to obtain a partially filled depth image, referring to the B frame again to obtain a completely filled depth image, and finally referring to the (A + B)/2 frame for smooth denoising to obtain a completely restored depth image. Since the three-dimensional inversion of the first part is only to obtain a three-dimensional depth image, and is not the content of the main teaching of the present invention, but the main function of the present invention, the three-dimensional inversion is not deeply taught. The latter three parts are the main implementation steps of the present invention, and in order to explain the method of the present invention in more detail, the detailed flow diagrams of the latter three parts in fig. 1 are shown in fig. 2 and fig. 3. Next, the present embodiment will be explained in detail with reference to the schematic flow diagrams shown in fig. 1, fig. 2, and fig. 3.
The specific steps of this example are as follows:
step 1: firstly, adjacent two-dimensional space slice images are gated according to the distance, represented by A, B frames respectively, a 3D threshold value is set, and an original depth image to be restored is obtained by using a three-dimensional inversion algorithm. The two-dimensional slice image data of the plant at night acquired by the laser range-gated three-dimensional imaging system independently developed by the author laboratory is selected, and the enhanced images of the adjacent two-dimensional space slice images are shown in the attached drawings, namely fig. 4(a) and fig. 4 (b). The a frame represents a slice image at a distance of 201m from the data acquisition system, and the B frame represents a slice image delayed by 50ns with respect to the a frame. According to the gray value of the background in A, B frames, setting the 3D threshold value to be 5 (namely, pixel points with the gray value less than 5 in A frames or B frames do not participate in three-dimensional inversion), and obtaining the original depth image to be restored from the adjacent A, B frames by using a distance-gated super-resolution three-dimensional inversion algorithm (actually, a triangular distance energy correlation three-dimensional inversion algorithm). The original depth image to be restored is shown in fig. 5(a) of the accompanying drawings, and it can be seen from the figure that there are many holes with missing depth data, the value of which is NaN, and the holes are represented as white holes in the image; in addition, there are many spikes in the image due to noise in the depth image. Therefore, the quality of the original depth image to be restored is poor, and the difference between the original depth image to be restored and the real target three-dimensional information is far. The following explains in detail how the original depth image is restored step by step.
The three-dimensional inversion algorithm used for obtaining the original depth image in the step 1 is a triangular distance energy correlation algorithm, in the algorithm, laser pulses and gating pulses are also rectangular pulses, the gating gate width is equal to the laser pulse width, therefore, under the action of an echo broadening effect, the energy envelope of the target distance direction is triangular, the distance information of the target can be obtained by establishing the energy gray scale ratio relation between adjacent slices, and the depth image can be obtained according to the adjacent slice images.
Step 2: secondly, partially filling the original depth image and the A frame image with missing data by adopting a self-adaptive bilateral reference algorithm to obtain a partially filled depth image;
and step 3: filling missing data completely by adopting a self-adaptive bilateral reference algorithm for the partially filled depth image and the B frame image to obtain a completely filled depth image;
wherein in the step 2-step 3:
the specific steps of self-adaptive bilateral reference completion missing data are as follows:
step 1 a: and finding the position of the missing depth data in the original depth image. The method comprises the steps that from the upper left corner of an original depth image, the position of a hole with missing depth data in the original depth image is found, and pixel points which are white and have a NaN value in the image are the position with missing depth data;
step 2 a: and for each depth data missing position, adjusting the size of a neighborhood Window according to the variance of all depth data in a neighborhood taking the depth data missing position as the center, so that the Window value is continuously changed between the allowed minimum value Window _ min and the maximum value Window _ max. Firstly, setting the initial value of the neighborhood Window value as the minimum value Window _ min, and setting the initial value of the variance in the neighborhood as the minimum value Var _ min. And calculating the variance in the neighborhood taking the first missing data position as the center, comparing the variance with a set initial value, if the variance value is greater than the initial value, reducing the neighborhood window value of the second missing data position by 1, if the variance value is less than the initial value, increasing by 1, and if the variance value is equal to the initial value, keeping the variance unchanged. And by analogy, the variance in the neighborhood of the current position is compared with the variance in the neighborhood of the previous position to determine the size of the next neighborhood window value. In this embodiment, Window _ min is 3, Window _ max is 15, and Var _ min is 0.
Step 3 a: assigning a weighted average value of each depth data in a neighborhood at the depth data missing position to a pixel point with depth data missing for complementing the missing depth data, wherein the complemented point is a neighborhood center; the assigned value is expressed by the following expression (1):
i' (x, y) is a value assigned to a missing data pixel in the center of the neighborhood, Ω is a domain range of the pixel (x, y), generally, a rectangular region centered on (x, y), w (I, j) is a weight coefficient at the pixel (I, j), and w (I, j) is a weight coefficient at the pixel (I, j)pIs the normalization parameter, and I (I, j) is the depth data value in the neighborhood. Wherein:
for the bilateral reference algorithm proposed by the method, the weight coefficient w (i, j) is determined by the depth image space domain weight wsAnd the gray domain weight w of the two-dimensional slice imagerThe composition is as follows:
w(i，j)＝ws(i，j)·wr(i，j) (8)
wherein wsAnd wrExpressed as a gaussian function:
σs、σrg (i, j) is the gray value of the two-dimensional slice image at pixel point (i, j) for the standard deviation based on the gaussian function.
σs、σrThe performance of the bilateral reference algorithm is determined. At σsIn the fixed case, σrIf the gray scale difference is too large, the weights w corresponding to different gray scale differences are all large, the effect of retaining edge information by utilizing gray scale change is lost, and the bilateral reference algorithm degenerates into a Gaussian filter; sigmarIf the weight w is too small, the filtering effect is lost if the weight w is too sensitive to different gray level differences. Bilateral reference algorithm between considered pixelsThe spatial correlation also considers the similarity degree of the pixel values, so that the detail information of the image can be kept not to be lost during denoising.
Step 4 a: standard deviation sigma of depth image space domain weight gaussian functionsMaking corresponding adjustment according to the neighborhood window value; taking the standard deviation of all pixel positions in the rectangular neighborhood window as the standard deviation sigma of the space domain weight Gaussian function of the neighborhood windowsThe window value is self-adaptive and continuously adjusted, so that the airspace weight is also self-adaptive and continuously adjusted; standard deviation sigma of two-dimensional slice image value domain weight Gaussian functionrPerforming corresponding adjustment according to all gray values in a neighborhood window of the two-dimensional slice image; taking the standard deviation of all gray values in the rectangular neighborhood window as the standard deviation sigma of the value domain weight Gaussian function of the neighborhood windowrAnd because the gray value in the window is continuously changed, the value domain weight is also self-adaptively and continuously adjusted.
In said step 4a, so σs、σrThe performance of the bilateral reference algorithm is determined, and therefore, it cannot be arbitrarily set to a fixed value. Standard deviation sigma of depth image space domain weight Gaussian functionsMaking corresponding adjustment according to the value of the neighborhood window, and taking the standard deviation of the positions of all pixel points in the rectangular neighborhood window as the standard deviation sigma of the space domain weight Gaussian function of the neighborhood windowsAnd the window value is self-adaptively and continuously adjusted, so that the spatial domain weight value is also self-adaptively and continuously adjusted. Standard deviation sigma of two-dimensional slice image value domain weight Gaussian functionrPerforming corresponding adjustment according to all gray values in a neighborhood window of the two-dimensional slice image, and taking the standard deviation of all gray values in a rectangular neighborhood window as the standard deviation sigma of a value domain weight Gaussian function of the neighborhood windowrAnd because the gray value in the window is continuously changed, the value domain weight is also self-adaptively and continuously adjusted.
According to the steps 3a to 4a, the neighborhood window value of each depth data to be restored and the standard deviation of the gaussian function in the domain are continuously changed according to the depth data in the previous domain window and the depth data of the current domain window, so that the value given to the domain center can reflect the actual situation in the current domain more truly, and the filling or smoothing depth data is more accurate in self-adaptation.
Step 5 a: when the depth data is supplemented, referring to the 3D inversion threshold set in the step 1; completing the depth data when the gray value of the same position of the two-dimensional slice image corresponding to the pixel point of the depth data to be completed is larger than the 3D inversion threshold value, otherwise skipping the pixel point;
step 6 a: and traversing the whole original depth image, and repeating the steps 1 a-5 a until the depth data of all the positions meeting the condition are completed.
Wherein in the step 5a to the step 6 a:
and when determining whether to fill in the missing depth data, referring to the 3D inversion threshold 5 set in the step 1, filling in the depth data when the gray value of the same position of the A frame corresponding to the pixel point of the depth data to be filled is greater than 5, otherwise, skipping the pixel point. Therefore, the positions of all missing depth data can be prevented from being filled, and the excessive filling is avoided. In the above operation performed for a position where a certain depth data is missing, the entire original depth image is traversed, and steps 1a to 5a are repeated until the depth data of all positions satisfying the condition (corresponding to the gray value of the a frame being greater than 5) are supplemented, so that the partially filled depth image of the reference a frame cannot be obtained. The depth image partially filled from the a frame is shown in fig. 5(b), and it can be known from the figure that partial depth data has been filled, such as branches in the upper left part of the image are gradually complete, compared to the original depth image in fig. 5 (a). The above filling is only for the a frame, and the filling is not complete, so the above filled depth image should be performed similarly to the B frame again to completely fill the depth image. Therefore, the steps 1a to 5a are repeated again, the gray value of the B frame is changed when the value domain weight is calculated, the pixel points with the gray value larger than 5 in the B frame are referred, and the depth data of all positions which meet the condition (the corresponding gray value of the B frame is larger than 5) are completed by the partially filled depth image and the B frame image by adopting the self-adaptive bilateral reference algorithm, so that the completely filled depth image is obtained. The self-adaptive bilateral reference algorithm provided by the invention can effectively fill the depth image obtained by the range gating three-dimensional imaging technology.
And 4, step 4: and finally, smoothing and filtering the completely filled depth image and the (A + B)/2 image by adopting a self-adaptive bilateral reference algorithm to obtain a complete and smooth repaired depth image.
Wherein in step 4:
the self-adaptive bilateral reference algorithm smooth denoising method comprises the following specific steps:
step 1 b: finding the position with the depth data in the completely filled depth image;
and step 2 b: the method for smoothing the depth data is consistent with the steps 2a to 4a, and only needs to replace the frame A or the frame B of the two-dimensional slice image with (A + B)/2 frames, and replace the position of the missing depth data with the position with the depth data;
and step 3 b: and traversing the whole filled complete depth image, and repeating the steps 1 b-2 b until the depth data of all positions are smoothed.
After the complete depth image is filled, because the original pixel points with the depth data have not undergone any operation, the depth data have many peaks, and the newly filled depth data may also introduce the peaks, all the depth data are subjected to smooth filtering to obtain a smooth depth image. To reduce the computation, the smoothing filter is only performed at locations where there is depth data, so locations where there is depth data in the full depth image are found first. For each depth data, the weighted average value of all depth data in the neighborhood needs to be substituted for the value of the neighborhood center to achieve the effect of smoothing the depth data, the method is consistent with the step 4a in the step 2a, and only the two-dimensional slice image a frame or B frame needs to be substituted by (a + B)/2 frames (as shown in fig. 4 (c)), and the position of missing depth data needs to be substituted by the position with depth data. Unlike complementing missing depth data, smoothing depth data does not require reference to a 3D threshold. And (3) the smooth depth data are specific to a certain pixel point, the whole completely filled depth image is traversed, and the steps 1b to 2b are repeated until the depth data of all positions are smoothed, so that the repaired depth image can be obtained. As shown in fig. 5(d), the restored depth image has a much smaller number of peaks than before smoothing, and the image is much smoother. To measure the effect of smoothing, the signal-to-noise ratio is objectively quantified. The signal-to-noise ratio of the original depth image is 25.7889dB, and the signal-to-noise ratio of the repaired depth image is 31.9161dB, which is improved by about 6 dB. The depth image obtained by the three-dimensional imaging technology can be effectively smoothed by the self-adaptive bilateral reference algorithm provided by the invention.
Up to this point, the present embodiment has been described in detail with reference to the accompanying drawings. From the above description, those skilled in the art should clearly recognize that the adaptive bilateral reference restoration method for range-gated three-dimensional imaging depth map of the present invention.
In addition, the definition of the above method is not limited to the specific structures, shapes or manners mentioned in the embodiments, and those skilled in the art can easily modify or replace them.
In conclusion, the adaptive bilateral reference restoration method provided by the invention achieves the purposes of effectively complementing the cavities of the depth image obtained by the range-gated three-dimensional imaging technology and removing noise, has the advantages of truly and effectively complemented depth data, obvious effect of smoothing the depth data, capability of keeping image detail information, high restoration speed, strong adaptability and the like.
The above-mentioned embodiments are intended to illustrate the objects, technical solutions and advantages of the present invention in further detail, and it should be understood that the above-mentioned embodiments are only exemplary embodiments of the present invention, and are not intended to limit the present invention, and any modifications, equivalents, improvements and the like made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (4)
1. A method for distance gating three-dimensional imaging self-adaptive bilateral reference restoration comprises the following steps:
step 1: firstly, gating adjacent two-dimensional space slice images according to the distance, respectively representing the two-dimensional space slice images by A, B frames, setting a 3D threshold value, and obtaining an original depth image to be restored by using a three-dimensional inversion algorithm;
step 2: secondly, partially filling the original depth image and the A frame image with missing data by adopting a self-adaptive bilateral reference algorithm to obtain a partially filled depth image;
and step 3: filling missing data completely by adopting a self-adaptive bilateral reference algorithm for the partially filled depth image and the B frame image to obtain a completely filled depth image;
and 4, step 4: and finally, smoothing and filtering the completely filled depth image and the (A + B)/2 image by adopting a self-adaptive bilateral reference algorithm to obtain a complete and smooth repaired depth image.
2. The method for range-gated three-dimensional imaging adaptive bilateral reference restoration according to claim 1, wherein the three-dimensional inversion algorithm used for obtaining the original depth image in step 1 is a triangular range energy correlation algorithm; in the algorithm, the laser pulse and the gating pulse are both rectangular pulses, and the gating gate width is equal to the laser pulse width, so that under the action of an echo broadening effect, the energy envelope of the target distance direction is triangular, and the distance information of the target can be obtained by establishing the energy gray scale ratio relation between adjacent slices, so that a depth image can be obtained according to the adjacent slice images.
3. The method for range-gated three-dimensional imaging adaptive bilateral reference restoration according to claim 1, wherein in steps 2-3:
the specific steps of self-adaptive bilateral reference completion missing data are as follows:
step 1 a: finding out the position of missing depth data in the original depth image, wherein pixel points which are represented as white and have a NaN value in the image are the position of missing depth data;
step 2 a: adjusting the size of a neighborhood Window according to the variance of all depth data in a neighborhood with the depth data missing position as the center, so that the Window value is continuously changed between the allowed minimum value Window _ min and the maximum value Window _ max; firstly, setting an initial value of a neighborhood window value as a minimum value, and setting an initial value of a variance in a neighborhood as a minimum value; calculating the variance in the neighborhood with the first missing data position as the center, comparing the variance with a set initial value, if the variance value is greater than the initial value, reducing the neighborhood window value of the second missing data position by 1, if the variance value is less than the initial value, increasing by 1, and if the variance value is equal to the initial value, keeping the variance unchanged; by analogy, comparing the variance in the neighborhood of the current position with the variance in the neighborhood of the previous position to determine the size of the next neighborhood window value;
step 3 a: assigning a weighted average value of each depth data in a neighborhood at the depth data missing position to a pixel point with depth data missing for complementing the missing depth data, wherein the complemented point is a neighborhood center; the assigned value is expressed by the following expression (1):
i' (x, y) is a value assigned to a missing data pixel in the center of the neighborhood, Ω is the domain range of the pixel (x, y), a rectangular region centered at (x, y), w (I, j) is a weight coefficient at the pixel (I, j)pIs a normalization parameter, I (I, j) is the value of each depth data in the neighborhood range; wherein:
for the bilateral reference algorithm proposed by the method, the weight coefficient w (i, j) is determined by the depth image space domain weight wsAnd the gray domain weight w of the two-dimensional slice imagerThe composition is as follows:
w(i，j)＝ws(i，j)·wr(i，j) (3)
wherein wsAnd wrExpressed as a gaussian function:
wherein sigmas、σrG (i, j) is the gray value of the two-dimensional slice image at the pixel point (i, j) based on the standard deviation of the Gaussian function;
step 4 a: standard deviation sigma of depth image space domain weight gaussian functionsMaking corresponding adjustment according to the neighborhood window value; taking the standard deviation of all pixel positions in the rectangular neighborhood window as the standard deviation sigma of the space domain weight Gaussian function of the neighborhood windowsThe window value is self-adaptive and continuously adjusted, so that the airspace weight is also self-adaptive and continuously adjusted; standard deviation sigma of two-dimensional slice image value domain weight Gaussian functionrPerforming corresponding adjustment according to all gray values in a neighborhood window of the two-dimensional slice image; taking the standard deviation of all gray values in the rectangular neighborhood window as the standard deviation sigma of the value domain weight Gaussian function of the neighborhood windowrBecause the gray value in the window is continuously changed, the value domain weight is also self-adaptively and continuously adjusted;
step 5 a: using the 3D inversion threshold set in step 1 when completing the depth data; completing the depth data when the gray value of the same position of the two-dimensional slice image corresponding to the pixel point of the depth data to be completed is larger than the 3D inversion threshold value, otherwise skipping the pixel point;
step 6 a: and traversing the whole original depth image, and repeating the steps 1 a-5 a until the depth data of all the positions meeting the condition are completed.
4. The method for range-gated three-dimensional imaging adaptive bilateral reference restoration according to claim 3, wherein in step 4:
the self-adaptive bilateral reference algorithm smooth denoising method comprises the following specific steps:
step 1 b: finding the position with the depth data in the completely filled depth image;
and step 2 b: the method for smoothing the depth data is consistent with the steps 2a to 4a, and only needs to replace the frame A or the frame B of the two-dimensional slice image with (A + B)/2 frames, and replace the position of the missing depth data with the position with the depth data;
and step 3 b: and traversing the whole filled complete depth image, and repeating the steps 1 b-2 b until the depth data of all positions are smoothed.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810611102.5A CN108961176B (en) | 2018-06-14 | 2018-06-14 | Self-adaptive bilateral reference restoration method for range-gated three-dimensional imaging |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810611102.5A CN108961176B (en) | 2018-06-14 | 2018-06-14 | Self-adaptive bilateral reference restoration method for range-gated three-dimensional imaging |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108961176A CN108961176A (en) | 2018-12-07 |
| CN108961176B true CN108961176B (en) | 2021-08-03 |
Family
ID=64488685
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810611102.5A Active CN108961176B (en) | 2018-06-14 | 2018-06-14 | Self-adaptive bilateral reference restoration method for range-gated three-dimensional imaging |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108961176B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110189265B (en) * | 2019-05-06 | 2021-06-08 | 中国科学院半导体研究所 | Distance energy correlation three-dimensional imaging super pixel repairing method |
| CN116228601B (en) * | 2023-05-08 | 2023-07-21 | 山东中都机器有限公司 | Visual monitoring method for coal leveling effect of bidirectional coal leveling of train |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101860667A (en) * | 2010-05-06 | 2010-10-13 | 中国科学院西安光学精密机械研究所 | Method for quickly removing mixed noise in image |
| CN103438863A (en) * | 2013-06-13 | 2013-12-11 | 中国科学院半导体研究所 | Range gating code super-resolution three-dimensional imaging method |
| CN105701778A (en) * | 2016-01-11 | 2016-06-22 | 赛诺威盛科技（北京）有限公司 | Method of removing metal artifact from CT image |
| CN106910242A (en) * | 2017-01-23 | 2017-06-30 | 中国科学院自动化研究所 | The method and system of indoor full scene three-dimensional reconstruction are carried out based on depth camera |
-
2018
- 2018-06-14 CN CN201810611102.5A patent/CN108961176B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101860667A (en) * | 2010-05-06 | 2010-10-13 | 中国科学院西安光学精密机械研究所 | Method for quickly removing mixed noise in image |
| CN103438863A (en) * | 2013-06-13 | 2013-12-11 | 中国科学院半导体研究所 | Range gating code super-resolution three-dimensional imaging method |
| CN105701778A (en) * | 2016-01-11 | 2016-06-22 | 赛诺威盛科技（北京）有限公司 | Method of removing metal artifact from CT image |
| CN106910242A (en) * | 2017-01-23 | 2017-06-30 | 中国科学院自动化研究所 | The method and system of indoor full scene three-dimensional reconstruction are carried out based on depth camera |
Non-Patent Citations (1)
| Title |
|---|
| Lower-upper-threshold correlation for underwater range-gated imaging self-adaptive enhancement;Sun Liang 等;《Applied Optics》;20161231;第55卷(第29期);第8248-8255页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108961176A (en) | 2018-12-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Hasselgren et al. | Shape, light, and material decomposition from images using monte carlo rendering and denoising | |
| CN106157267B (en) | Image defogging transmissivity optimization method based on dark channel prior | |
| US9710715B2 (en) | Image processing system, image processing device, and image processing method | |
| WO2022016310A1 (en) | Point cloud data-based three-dimensional reconstruction method and apparatus, and computer device | |
| US20150154467A1 (en) | Method for Extracting Planes from 3D Point Cloud Sensor Data | |
| KR102049080B1 (en) | Image processing apparatus and method thereof | |
| CN110189265B (en) | Distance energy correlation three-dimensional imaging super pixel repairing method | |
| CN108961176B (en) | Self-adaptive bilateral reference restoration method for range-gated three-dimensional imaging | |
| CN112991193B (en) | Depth image restoration method, device and computer-readable storage medium | |
| CN110866882B (en) | Layered joint bilateral filtering depth map repairing method based on depth confidence | |
| KR20110014067A (en) | Method and system for transformation of stereo content | |
| WO2023131189A1 (en) | Image inpainting method and device | |
| CN111047698B (en) | Real projection image acquisition method | |
| EP1465116A1 (en) | Computer graphics | |
| CN109903379A (en) | A kind of three-dimensional rebuilding method based on spots cloud optimization sampling | |
| RU2419880C2 (en) | Method and apparatus for calculating and filtering disparity map based on stereo images | |
| JP2019091122A (en) | Depth map filter processing device, depth map filter processing method and program | |
| CN108805841B (en) | Depth map recovery and viewpoint synthesis optimization method based on color map guide | |
| CN105913391B (en) | A kind of defogging method can be changed Morphological Reconstruction based on shape | |
| CN111179333A (en) | Defocus fuzzy kernel estimation method based on binocular stereo vision | |
| KR20200057612A (en) | Method and apparatus for generating virtual viewpoint image | |
| KR102327304B1 (en) | A method of improving the quality of 3D images acquired from RGB-depth camera | |
| Othman et al. | Enhanced single image dehazing technique based on hsv color space | |
| CN113077504B (en) | Large scene depth map generation method based on multi-granularity feature matching | |
| Abbaspour et al. | A new fast method for foggy image enhancement |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |