CN114679579A - Hole filling method for converting 2D video into 3D video - Google Patents
Hole filling method for converting 2D video into 3D video Download PDFInfo
- Publication number
- CN114679579A CN114679579A CN202011557582.5A CN202011557582A CN114679579A CN 114679579 A CN114679579 A CN 114679579A CN 202011557582 A CN202011557582 A CN 202011557582A CN 114679579 A CN114679579 A CN 114679579A
- Authority
- CN
- China
- Prior art keywords
- hole
- image
- video
- block
- filling
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 29
- 239000011796 hollow space material Substances 0.000 claims description 2
- 230000015572 biosynthetic process Effects 0.000 abstract description 2
- 238000003786 synthesis reaction Methods 0.000 abstract description 2
- 230000000007 visual effect Effects 0.000 abstract description 2
- 238000004364 calculation method Methods 0.000 description 5
- 238000012545 processing Methods 0.000 description 4
- 238000007781 pre-processing Methods 0.000 description 3
- 238000005070 sampling Methods 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000006243 chemical reaction Methods 0.000 description 2
- 238000012805 post-processing Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/20—Image signal generators
- H04N13/261—Image signal generators with monoscopic-to-stereoscopic image conversion
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
- H04N13/106—Processing image signals
- H04N13/122—Improving the 3D impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)
Abstract
The invention discloses a hole filling method for converting a 2D video into a 3D video, which comprises the following steps: s1: and inputting information such as a virtual viewpoint containing a hole and a corresponding depth map, outputting a virtual viewpoint image after filling the hole, and judging whether the hole is processed. S2: and when the hole is not processed, extracting the boundary point of the hole area. And traversing all the hole points to obtain a hole area boundary point set. S3: and traversing the information source image and extracting a matching block. S4: and calculating the matching cost of the hole color image and the depth image target block of the central point and the source image block to be matched. S5: and obtaining the central point of the optimal matching block, and updating the cavity image by the source image and the depth image point of the neighborhood block. Through the steps, the filling quality of the hole can be effectively guaranteed, the visual consistency of the virtual viewpoint synthesis is improved, and the result is as real as possible and close to the original image.
Description
Technical Field
The invention relates to a hole filling method, in particular to a hole filling method for converting a 2D video into a 3D video.
Background
The most important link problem of 2D-3D video conversion is hole filling, and how to fill the generated holes is a crucial problem and a difficult problem in the three-dimensional video conversion technology based on the disparity map. According to different processing stages, the method for eliminating the holes can be divided into two types, namely a pre-processing algorithm and a post-processing algorithm. The preprocessing algorithm is to perform depth image preprocessing before creating left and right views to reduce the number of generated holes, which has been described above, and although this processing process can effectively reduce the area of holes, it will generate distortion phenomenon, so that the shape of the object is geometrically distorted, and finally the image quality is reduced; the post-processing algorithm is processing after the left view and the right view are created, and the main methods include 1, filling the hollow points by utilizing non-hollow points around the hollow points to perform linear interpolation, although the method is simple, the image quality after the hollow filling is performed by the method is not ideal, and some hollow points can be filled wrongly, 2, filling the hollow points in the target image by using points in the reference viewpoint, and finding out matched points capable of filling the hollow points by fully considering the parallax information of the pixel points in the reference viewpoint during filling. Image texture and structural information cannot be well expressed in the aspect of filling holes, and filling areas are blurred in complex texture areas.
For the reasons, the hole filling method in the prior art is time-consuming and labor-consuming, and has low accuracy and unsatisfactory image quality.
Therefore, in view of the above disadvantages, it is desirable to provide a method for filling holes from 2D video to 3D video.
Disclosure of Invention
The invention aims to provide a hole filling method for converting a 2D video into a 3D video, which aims to solve the problems that the hole filling method in the prior art is time-consuming and labor-consuming and has unsatisfactory image quality.
The method for filling the hole for converting the 2D video into the 3D video comprises the following steps: s1: and inputting information such as a virtual viewpoint containing a hole and a corresponding depth map, outputting a virtual viewpoint image after filling the hole, and judging whether the hole is processed. S2: and when the hole is not processed, extracting the boundary point of the hole area. And traversing all the hole points to obtain a hole area boundary point set. S3: and traversing the information source image and extracting a matching block. S4: and calculating the matching cost of the hole color image and the depth image target block of the central point and the source image block to be matched. S5: and obtaining the central point of the optimal matching block, and updating the cavity image by the source image and the depth image point of the neighborhood block.
In the method for filling a hole in a 2D video to 3D video as described above, S2 preferably includes:
S21: distributing priority to the edge pixels of the hollow space, and filling a target block where an edge point with higher priority is located;
s22: generating discontinuous regions of depth values of the image for holes in the virtual viewpoint, taking information of the depth map into consideration in determining the priority.
In the method for filling a hole in a 2D video to 3D video, S22 further preferably further includes a priority level mainly including three items, i.e., a confidence level item, a data item, and a depth value constraint item.
In the method for filling a hole in a 2D video to 3D video as described above, S3 preferably includes: s31: defining an image window for searching source information as a search area and searching an optimal matching block in the search area;
s32: and calculating the matching cost of each block in the region and the target block according to a specific formula, wherein the block with the minimum cost is the most similar matching block with the target block.
Compared with the prior art, the method for filling the cavity for converting the 2D video into the 3D video, disclosed by the invention, has the following beneficial effects:
according to the hole filling method for converting the 2D video into the 3D video, provided by the invention, the information required by the current hole area is found by utilizing inter-frame sampling of the video, and the hole area is filled. On the basis of analyzing the existing algorithm, the advantages and the disadvantages of the original priority item calculation method are found, and a new priority item depth constraint calculation mode is provided; and a matching cost calculation formula is improved, and when the target block has more interference information, the cavity filling quality is ensured. By utilizing the redundancy characteristic of video information, the holes are filled in the inter-frame sampling information, the visual consistency of virtual viewpoint synthesis is improved, and meanwhile, the phenomenon of image blurring cannot occur.
Detailed Description
Example 1:
as shown in fig. 1, the present embodiment provides a method for filling a hole from a 2D video to a 3D video, including:
s1: and inputting information such as a virtual viewpoint containing the hole, a corresponding depth map and the like, outputting a virtual viewpoint image after the hole is filled, and judging whether the hole is processed or not.
S2: and when the hole is not processed, extracting the boundary point of the hole area. And traversing all the hole points to obtain a hole area boundary point set.
S3: and traversing the information source image and extracting a matching block.
S4: and calculating the matching cost of the hole color image and the depth image target block of the central point and the source image block to be matched.
S5: and obtaining the central point of the optimal matching block, and updating the cavity image by the source image and the depth image point of the neighborhood block.
Further, S1 includes:
s11: in the generation process of the virtual viewpoint depth map, the position of the hole mainly appears at the boundary of the foreground and the background. Taking the edge position i on one side of the cavity background as the center and the depth as D, selecting a rectangular region with the size of M x N in the neighborhood, carrying out k-means classification on the depth values of the pixel points in the region, and dividing the depth values into two classes, wherein the classification centers are Cin and Cma respectively. After the classification center is determined, the padding value in the row area can be determined by Cm therein.
S12: after the depth values on the boundary are determined, the depth values of the same row in the area are all filled by the depth values. Where M, N is chosen in relation to the width of the hole, typically a window size of 32 x 32 is chosen.
Further, S2 includes:
s21: the background information in the video frame and the corresponding depth map are stored to form a background information block, the background block of the color image is recorded as S, and the corresponding depth information is recorded as G. Finding out the median value of the boundary points extracted in the last step as a threshold value for segmenting the foreground and the background;
s22: for each frame of new image, the information points with the depth values smaller than the threshold value are used as background information to be updated into the background sub-block, and for other pixel points, the points with the depth values larger than the threshold value and the cavity points are regarded as foreground information points and are not updated into the background sub-block.
Further, S22 includes that for the hole region, we regard it as an unreliable point, the boundary of the hole, and within the width of two pixels, we do not regard these points as background pixel points and do not include them into the background sub-block.
Specifically, for each frame of new image, the median of the boundary point obtained in the previous step is used as a threshold for segmenting the foreground and the background of each frame of image, so in step S23, if the depth value of the point is smaller than the threshold, the hollow area is filled with the background information block, and for the hollow area whose depth value is greater than the threshold, the processing is still performed according to the hollow. The result of filling by adopting the information of the adjacent frames is real and reliable, is the real value of the image of the hollow area, and is more practical than the speculative value obtained by the existing most filling algorithms. After the padding of the first frame image is completed, the background part in the newly padded information is updated into the background sub-block.
The purpose of step S22 is to avoid repeated padding of the same area to reduce the padding time.
Further, S3 includes:
s31: as shown in fig. 2, when the depth value of a known point in the target block is greater than the depth value of the background by a threshold, the point is marked as a foreground point, and the foreground point matching cost is not calculated. And classifying the known points in the target block into a background point set and a foreground point set. And judging whether the target block depth map is a foreground region or a background region through a known point in the target block depth map.
S32: after finding the optimal matching block corresponding to the target block, there are two corresponding target blocks to be processed, namely, the corresponding areas of the texture image hole and the depth image hole target blocks. And copying information from the point corresponding to the optimal matching block for all the hole points in the target block, and copying information from the depth map corresponding to the matching block for the hole points of the depth map corresponding to the target block to fill the holes in the block.
Further, S4 includes:
and after the current target block is filled, continuously filling other points in the hole. And updating the priority of the new hole boundary point in the neighborhood of the filling target block, so that the priority of the hole point does not need to be recalculated every time, and the calculation time is reduced.
Further, S5 includes:
and copying the optimal matching block information into the target block, wherein the depth map hole point also obtains a value from the position of the matching block corresponding to the depth map. Thereby obtaining a virtual viewpoint without a hole.
Compared with the prior art, the method for filling the cavity for converting the 2D video into the 3D video, disclosed by the invention, has the following beneficial effects:
according to the hole filling method for converting the 2D video into the 3D video, the filling priority of each hole point is calculated at first, and the hole point with the largest priority is found to serve as a target block to be filled in the iteration. Then, sampling image blocks in a current frame adjacent frame, finding an optimal matching block with the minimum matching cost with the target block by calculating the matching cost between the matching block and the target block, copying the information of the optimal matching block into the target block, and obtaining a value from the position of the depth map corresponding to the matching block by the depth map hole point. Thus, a virtual viewpoint without a hole is obtained. And the inter-frame consistency of the filling result of the hollow area is ensured to a certain extent.
Finally, it should be noted that: the above embodiments are only used to illustrate the technical solution of the present invention, and not to limit the same; while the invention has been described in detail and with reference to the foregoing embodiments, it will be understood by those skilled in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some or all of the technical features may be equivalently replaced; and the modifications or the substitutions do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present invention.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts.
FIG. 1 is a flowchart illustrating steps of a method for filling a hole in a 2D video to 3D video according to the present invention;
FIG. 2 is a schematic diagram of a priority depth item calculation neighborhood in the present invention;
fig. 3 illustrates the change of the images before and after step S5, where a is the virtual viewpoint hole image and b is the depth map hole filling result.
Claims (4)
1. A hole filling method for converting a 2D video into a 3D video is characterized by comprising the following steps:
and inputting information such as a virtual viewpoint containing a hole and a corresponding depth map, outputting a virtual viewpoint image after filling the hole, and judging whether the hole is processed.
And when the hole is not processed, extracting the boundary point of the hole area. And traversing all the hole points to obtain a hole area boundary point set.
Traversing the information source image and extracting a matching block.
And calculating the matching cost of the hole color image and the depth image target block of the central point and the source image block to be matched.
And obtaining the central point of the optimal matching block, and updating the hole image by the source image and the depth image of the neighborhood block.
2. The method for filling a hole from a 2D video to a 3D video according to claim 1, wherein the extracting the boundary point of the hole region comprises:
distributing priority to the edge pixels of the hollow space, and filling a target block where an edge point with higher priority is located;
generating discontinuous regions of depth values of the image for holes in the virtual viewpoint, taking information of the depth map into consideration in determining the priority.
3. The method for hole filling from 2D video to 3D video according to claim 2, wherein the prioritizing further comprises: the priority mainly comprises three items, namely a confidence coefficient item, a data item and a depth value constraint item.
4. The method for filling holes from 2D video to 3D video according to claim 1, wherein traversing the information source images to extract matching blocks comprises:
defining an image window for searching source information as a search area and searching an optimal matching block in the search area;
And calculating the matching cost of each block in the region and the target block according to a specific formula, wherein the block with the minimum cost is the most similar matching block with the target block.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011557582.5A CN114679579A (en) | 2020-12-25 | 2020-12-25 | Hole filling method for converting 2D video into 3D video |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011557582.5A CN114679579A (en) | 2020-12-25 | 2020-12-25 | Hole filling method for converting 2D video into 3D video |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114679579A true CN114679579A (en) | 2022-06-28 |
Family
ID=82070327
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011557582.5A Pending CN114679579A (en) | 2020-12-25 | 2020-12-25 | Hole filling method for converting 2D video into 3D video |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114679579A (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20160150208A1 (en) * | 2013-07-29 | 2016-05-26 | Peking University Shenzhen Graduate School | Virtual viewpoint synthesis method and system |
| CN109712067A (en) * | 2018-12-03 | 2019-05-03 | 北京航空航天大学 | A kind of virtual viewpoint rendering method based on depth image |
-
2020
- 2020-12-25 CN CN202011557582.5A patent/CN114679579A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20160150208A1 (en) * | 2013-07-29 | 2016-05-26 | Peking University Shenzhen Graduate School | Virtual viewpoint synthesis method and system |
| CN109712067A (en) * | 2018-12-03 | 2019-05-03 | 北京航空航天大学 | A kind of virtual viewpoint rendering method based on depth image |
Non-Patent Citations (4)
| Title |
|---|
| 丁焱: "基于深度图的虚拟视点绘制中空洞填补技术研究", 中国优秀硕士学位论文全文 数据库, vol. 2016, no. 02, pages 23 - 47 * |
| 丁焱: "基于深度图的虚拟视点绘制中空洞填补技术研究", 中国优秀硕士学位论文全文数据库, no. 2016, pages 23 - 47 * |
| 刘晓军: "2D 视频转3D 视 频的空洞填充方法研究", 中国优秀硕士学位论文全文 数据库, no. 2013, pages 18 - 42 * |
| 刘晓军: "2D视频转3D视频的空洞填充方法研究", 中国优秀硕士学位论文全文数据库 信息科技辑, no. 2013, pages 18 - 42 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11017586B2 (en) | 3D motion effect from a 2D image | |
| EP2230855B1 (en) | Synthesizing virtual images from texture and depth images | |
| JP5474061B2 (en) | Method and apparatus for filling in an area of occlusion in a depth or parallax map estimated based on at least two images | |
| CN109462747B (en) | DIBR system cavity filling method based on generation countermeasure network | |
| CN108596919B (en) | Automatic image segmentation method based on depth map | |
| Phan et al. | Semi-automatic 2D to 3D image conversion using scale-space random walks and a graph cuts based depth prior | |
| Choi et al. | Space-time hole filling with random walks in view extrapolation for 3D video | |
| US8355596B2 (en) | Image generation method and apparatus, program therefor, and storage medium which stores the program | |
| CN110706269B (en) | Binocular vision SLAM-based dynamic scene dense modeling method | |
| TW202037169A (en) | Method and apparatus of patch segmentation for video-based point cloud coding | |
| KR100987412B1 (en) | Multi-Frame Combined Video Object Matting System and Method Thereof | |
| CN104065946A (en) | Cavity filling method based on image sequence | |
| Pushpalwar et al. | Image inpainting approaches-a review | |
| CN110719453B (en) | Three-dimensional video clipping method | |
| Nguyen et al. | New hole-filling method using extrapolated spatio-temporal background information for a synthesized free-view | |
| CN112508821B (en) | Stereoscopic vision virtual image hole filling method based on directional regression loss function | |
| CN105791795A (en) | Three-dimensional image processing method and device and three-dimensional video display device | |
| Mukherjee et al. | A hybrid algorithm for disparity calculation from sparse disparity estimates based on stereo vision | |
| JP4600993B2 (en) | Free viewpoint video generation system | |
| Yao et al. | Fast and high-quality virtual view synthesis from multi-view plus depth videos | |
| Doulamis et al. | Unsupervised semantic object segmentation of stereoscopic video sequences | |
| CN114679579A (en) | Hole filling method for converting 2D video into 3D video | |
| CN110602476A (en) | Hole filling method of Gaussian mixture model based on depth information assistance | |
| JP4707093B2 (en) | Free viewpoint video generation system | |
| CN110490877B (en) | Target segmentation method for binocular stereo image based on Graph Cuts |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20220628 |