CN102075779B - Intermediate view synthesizing method based on block matching disparity estimation - Google Patents
Intermediate view synthesizing method based on block matching disparity estimation Download PDFInfo
- Publication number
- CN102075779B CN102075779B CN 201110041538 CN201110041538A CN102075779B CN 102075779 B CN102075779 B CN 102075779B CN 201110041538 CN201110041538 CN 201110041538 CN 201110041538 A CN201110041538 A CN 201110041538A CN 102075779 B CN102075779 B CN 102075779B
- Authority
- CN
- China
- Prior art keywords
- view
- piece
- parallax
- target image
- reference picture
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
Images
Landscapes
- Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)
Abstract
The invention discloses an intermediate view synthesizing method based on block matching disparity estimation. In order to carry out disparity estimation on two images selected from the same scene, the same time and different angles, a virtual view at any position on a straight line (baseline) between original views is obtained through image interpolation according to the acquired disparity image. The method comprises the following steps: firstly carrying out disparity estimation by taking a left view and a right view as target images respectively, and computing two disparity views from the left view to the right view as well as from the right view to the left view by adopting a disparity estimation algorithm based on the block matching; then synthesizing a preliminary intermediate view according to the disparity views; and finally carrying out hole filling on the synthetic view to obtain a final result. Aiming at the characteristic that the left view and the right view only have parallax in the horizontal direction because shooting is carried out by a camera on a horizontal line, search is only carried out in the horizontal direction, the amount of calculation is reduced, and the algorithm running time is shortened; and the experimental result indicates that a favorable synthetic result can be obtained under the condition of a farther distance of view foreground to the camera or smooth image gradation changes.
Description
Technical field
The present invention relates to a kind of processing method of binocular digital picture, particularly a kind of binocular digital picture intermediate-view synthetic method.
Background technology
Along with the development of multimedia technology, image and video technique also by two dimension to Three-dimensional Development, interactivity will become a principal character of Future Multimedia technology.One of key technology of interactive three-dimensional video system is exactly the virtual view synthetic technology.It is the indispensable modules of all three-dimensional display system terminals that virtual view is painted synthetic, also has vital effect in the high-end MultiMedia Fields such as Remote Video Conference, free view-point stereoscopic TV.For the user can be roamed in scene, realize " continuous looks around ", in the process of multi-view video collection, the quantity of video camera should be many as far as possible, but owing to placing a unlimited video camera to realize the unrealistic property of viewpoint seamless switching, in order to show the view of any viewpoint, must carry out the synthetic of virtual view in client, by the analysis to existing viewpoint, the viewpoint that synthetic user will observe.Therefore, the virtual view synthetic technology is very important emerging technology of MultiMedia Field.
The virtual view synthetic technology, be divided into two kinds of method for drafting (GBR) based on model and image-based method for drafting (IBR) on principle, the advantage of GBR is the geometrical model performance comprehensively and can satisfies flexible and changeable visual angle change demand, developed to get comparative maturity as traditional algorithm, interactivity is good.But the method has two major defects, namely creates the high accuracy three-dimensional object and the model of place amount of calculation is huge, may complete hardly; And calculate time-consumingly, and real-time is poor, and the drafting time is depended on the complexity of object.In addition, look if final purpose is only synthetic mesophase, it may be unwanted setting up so threedimensional model.Given this, can directly utilize real scene paid close attention to gradually and develop as the IBR method of the synthetic new view of image of reference viewpoint.The view sense of reality that the IBR method generates is stronger, better with the fusion of scene, and owing to having avoided process of reconstruction, make it be more convenient for carrying out the real-time rendering of complex scene, have computational complexity and do not change with scene changes, need not scene is carried out the characteristics such as Geometric Modeling.Based on the Intermediate View synthetic method of the parallax branch of IBR just.
In the middle of various IBR methods, virtually drawing (Depth-Image Based Rendering based on depth image, DIBR) develop to get comparative maturity, these class methods being by utilizing depth camera to take or utilize geometrical constraint and parallax information between each original view to obtain depth information corresponding to each colour element, and according to the virtual image of depth information drawing three-dimensional scene.For example, the piece that the smoothness of utilizing the reference view depth map that the people such as Jiang Gangyi propose becomes to vary in size with color images, then carry out the 3-D view conversion each piece is projected to virtual view, to complete method to the drafting of this viewpoint virtual view (referring to Jiang Gangyi, Zhu Bo, Yu Mei. a kind of method for drawing virtual view image based on gradable block [P]. Chinese invention patent: 200910153324.8,2009-10-14.).And for example the people such as Luo Kai proposes at first depth image to be carried out morphology and processes, then utilize the viewpoint change equation to generate target view, adopt the image repair algorithm to carry out the method for reprocessing (referring to Luo Kai to the target view that contains the cavity at last, Li Dongxiao, Feng Yamei, open bright. based on any drawing viewpoints [J] of DIBR and image repair. Journal of Image and Graphics, 2010,15 (3): 443-449.).The DIBR method is incorporated into the depth information of scene in the middle of the virtual view scene, reduced the required reference view number of virtual viewpoint rendering, but obtaining of depth image is very difficult, depth camera can directly be obtained depth information of scene, but involve great expense, the multi-lens camera group is demarcated difficulty; And the depth information that directly extracts each pixel from coloured image is very difficult, due to the synthetic precision that density and accuracy greatly affect virtual view of taking out of depth information, extracts depth information insecure often from original view.
Adopt method of geometry to obtain in the process of depth information, have a step to ask exactly optical parallax field, therefore, directly carrying out virtual viewpoint rendering with parallax is easily.In the middle of these class methods, the accuracy of disparity estimation is the key that obtains good medial view, and therefore, present algorithm mainly concentrates on how to obtain optical parallax field accurately.But optical parallax field needs very large amount of calculation accurately, and this is also impracticable in the middle of practical application.Classical Disparity estimation always can't average out aspect two of accuracy and rapidities, these algorithms are when the search parallax, in order to reach higher accuracy, often adopt complicated constraints and search strategy, but the raising effect to precision is also not obvious, makes on the contrary Riming time of algorithm lengthen; Employing can improve the efficient of disparity estimation at the optimal match point of local area search pixel, reduces than global search but cost is the disparity estimation precision.
In the middle of the disparity estimation link, the people such as Lv Chaohui have proposed a kind of method based on adaptive weight in disparity estimation, and introduce disparity smoothness bound term in the cost function of coupling, the optical parallax field that this method obtains is comparatively accurately, but because constraints is complicated, amount of calculation greatly increase (referring to Lv Chaohui, Yuan's Tun. based on the Intermediate View of disparity estimation synthetic [J]. photoelectron laser .2007,18 (7): 855-858).The people such as Luo Yan propose to add image segmentation algorithm in disparity estimation, after obtaining optical parallax field, input picture is carried out the image segmentation of intensity-based value, image are divided into have gray value different some zones, and the point of intra-zone has the gray scale similitude.After obtaining optical parallax field, has this hypothesis of same or analogous parallax value according to the pixel of same cut zone, optical parallax field is carried out level and smooth (referring to Luo Yan, Anping, Zhang Zhao raises. and the medial view picture based on disparity field calibration and Region Segmentation generates and interpolating method [J]. and journal .2004.25 (10) communicates by letter: 127-133.).Do like this parallax that can correct a part of erroneous matching, reach to a certain extent the level and smooth purpose of optical parallax field, but can cause the original correct parallax of another part to be endowed wrong parallax value, and, the increase of image segmentation module increases the amount of calculation of whole algorithm.
Summary of the invention
The technical problem to be solved in the present invention is: for overcoming the deficiencies in the prior art, the invention provides a kind of medial view synthetic method based on piece coupling disparity estimation, greatly reduce computational complexity when not affecting synthetic view quality, saved Riming time of algorithm.
The technical solution adopted for the present invention to solve the technical problems comprises: a kind of medial view synthetic method based on piece coupling disparity estimation is characterized in that comprising the following steps:
(1) input is taken from Same Scene, synchronization, and the position for video camera is in the left and right of same level height two width images, requires this two width image only to there are differences taking on the visual angle;
(2) if two width input pictures are coloured image, be translated into respectively gray level image; If two width input pictures are gray level image, execution in step (3);
(3) judge whether two width input image sizes are identical, if different, the prompting mistake is also jumped out; If identical, execution in step (4);
(4) be a kind of parallax estimation method based on the piece coupling, specifically take right view as target image, left view is reference picture, the piece that target view is divided into fixed size, for each piece in target image, search for one by one the most close with it piece in reference picture, calculate the displacement vector d between match block in each target image piece and reference picture
LR, be the piece parallax of left view in the right view; Take left view as target image, right view is reference picture again, and repeating step (4) is obtained the piece parallax d of right view in the left view
RL
(5) according to the piece parallax d of the left view of obtaining in step (4) to right view
LR, utilizing the binocular vision principle of parallax, can obtain through arrive the medial view I of right view piece parallax after preliminary View Synthesis based on left view
M L(wherein subscript M representative " medial view "; Subscript L represents " left view is to right view "); According to the piece parallax d of the right view of obtaining in step (4) to left view
RL, in like manner can obtain through arrive the medial view I of left view piece parallax after preliminary View Synthesis based on right view
M R(wherein subscript R representative " right view is to left view ") in theory, d
LRWith d
RL, I
M LWith I
M RShould equate, but because the error of parallax piece coupling can't be eliminated fully, so need further two medial view after preliminary View Synthesis to be processed;
(6) be a kind of method of seeking respectively view optimal match point to be synthesized in left view and right view, specifically for the I that obtains in step (5)
M LWith I
M RThese two preliminary synthetic medial view for each pixel wherein, are sought I one by one in left view and right view
M LWith I
M RIn the optimal match point I of each pixel
L(x
L, y) and I
R(x
R, y) (wherein, I
LAnd I
RRepresent the pixel gray value in left view and right view; The coordinate of (x, y) represent pixel point; x
LAnd x
RRepresent respectively the abscissa value of the optimal match point that finds in left view and right view, because binocular camera is in the same level height, so the ordinate value of the optimal match point that finds in left view and right view equates, represents with y), and according to the weighting of optimal match point to I
M LWith I
M RCarrying out the cavity filling (namely uses certain gray value to fill up I
M LWith I
M RIn be not mapped to get white space), finally obtain the synthetic I as a result of medial view
M
The parallax estimation method based on the piece coupling in step described above (4), adopt following steps to realize:
(i) reference picture being expanded the limit processes, increase respectively k pixel unit (k must satisfy the scene content that only is present in a width view in comprising all binocular views) on the left side of reference picture and right side, and the gray value that makes these pixels is 0, and the edge of augmenting is set to black;
(ii) target image is divided into the piece of M * N, wherein M is every width, and N is length;
(iii) obtain the SAD of target image and reference picture correspondence position piece, be used for initial value relatively during as search, wherein block size is M * N, upper left corner coordinate is (m, n) piece in target image and upper left corner coordinate are that absolute value error and the minimum absolute difference SAD between the reference image block of (p, q) is:
Wherein, (m, n) is the pixel coordinate in the piece upper left corner in target image; (p, q) is the pixel coordinate in the upper left corner in reference picture; I
1, I
2Be respectively target image and reference picture at the gray value of a certain coordinate points;
(iv) in reference picture, the search starting point is set to the upper left corner coordinate (m of target image piece, n), be (m-60 at abscissa zone, m+60) mate in scope, obtain the SAD of each matched position and target image interblock to be matched, make sad value obtain minimum reference picture correspondence position and be best matching blocks, and keep this SAD minimum value;
(v) the best matching blocks position that searches is recorded, and obtained displacement vector d between object block and best matching blocks, wherein d (i, j)=(m-p, n-q), i.e. parallax;
(vi) if what accept coupling is the piece that is positioned at the lower right corner in target image, do not comprise the edge of augmenting, namely upper left corner coordinate be (X+60-M, Y+60-N), finishes to mate; Otherwise, find target image piece next to be matched, return to step (iii);
(v) parallax take piece as unit is extended to take pixel as unit, i.e. d (i * M+m, j * N+n)=d (i, j); M ∈ [0, M-1] wherein, n ∈ [0, N-1].Take left view as reference picture, right view is target image, can calculate left view to the piece parallax d of right view
LRTake right view as reference picture, left view is target image again, can calculate right view to the piece parallax d of left view
RLThe present invention adopts this method to obtain d
LRAnd d
RL, be used for step (5) and step (6) etc.Obtain this two parallaxes, can proceed medial view synthetic.
In step described above (5), utilize the binocular vision principle of parallax to obtain the method for the medial view that preliminary View Synthesis obtains as follows:
Be positioned at two video cameras of same level height, simultaneously scene taken, according to the binocular parallax principle, have:
x
R＝x
L+d
LR(x，y)
I
R(x，y)＝I
L(x+d
LR(x，y)，y)
Wherein, x
LFor some abscissa in camera coordinate system of left view, take pixel as unit; X
RAbscissa for these corresponding points in right view; d
LRFor take left view as target image, right view is the parallax value that reference picture is obtained,
The following luminance weighted acquisition of the synthetic use at virtual visual angle:
I
M(x
L，y)＝(1-α)I
L(x
L，y)+αI
R(x
R，y)
If α is location parameter, make α=0 be the left view position, α=1 is the right view position, and α is (0,1) can represent optional position on straight line between original view (baseline) in the interval, and Intermediate View to be synthesized and left and right original view Relations Among are expressed as:
x
M＝x
L+αd
LR(x，y)＝x
R+(1-α)d
RL(x，y)
To can obtain respectively using d with co-relation substitution weighting formula
LRAnd d
RLThe I that obtains
M LWith I
M RAs follows:
I
M L(x
L+αd
LR，y)＝(1-α)I
L(x
L，y)+αI
R(x
L+d
LR，y)
I
M R(x
R+(1-α)d
RL，y)＝(1-α)I
L(x
R+d
RL，y)+αI
R(x
R，y)
Wherein, I
M LAnd I
M RThe medial view that obtains for preliminary View Synthesis; I
LAnd I
RBe respectively the left and right original view; x
LAnd x
RAll can travel through entire image.
In step described above (6), the method for seeking respectively view optimal match point to be synthesized in left view and right view is embodied as:
(i) obtain left view to the disparity map d of right view
LRIn maximum d
LR max
(ii) obtain function g
L(x)=| x
I-(x+ α d
RL(x, y)) | at x ∈ [x
I-α d
RL max, x
I+ α d
RL max] interval interior minimum value min (g
L(x))=g
L(x
L), and preservation makes g
L(x) obtain minimum x value, be designated as x
L, wherein, x
IBe the abscissa of medial view mid point, value is x
I∈ [0, picture traverse-1];
(iii) obtain right view to the disparity map d of left view
RLIn maximum d
RL max
(iv) obtain decision function g
R(x)=| x
I-(x+ α d
RL(x, y)) | at x ∈ [x
I-α d
RL, x
I+ α d
RL] interval interior minimum value min (g
R(x))=g
R(x
R), and preserve x
R
(v) according to luminance weighted formula I
M(x, y)=(1-α) I
L(x
L, y)+α I
R(x
R, y), finally obtain the synthetic I as a result of medial view
M
The present invention compared with prior art has advantages of:
(1) the present invention is at the little step length searching of disparity estimation link employing in a big way, and search precision is equal to the global search of exhaustive type substantially, and search precision is higher than the Local Search scheme of classical Disparity estimation;
(2) the present invention in disparity estimation according to for the synthesis of the original view of Intermediate View, according to the two width images of binocular parallax principle shooting from the same level height, there is not in theory the parallax of vertical direction, therefore only horizontal direction is carried out match search, greatly reduce operand, reduced Riming time of algorithm;
(3) experiment shows, when the method that the present invention adopts can be synthesized the medial view with good visual effect, particularly prospect far away or grey scale change scene is milder apart from video camera; And the view precision is substantially constant has effective improvement to the view aggregate velocity in the situation that synthesize.
Description of drawings
Fig. 1 is the medial view composition algorithm flow chart that the present invention is based on piece coupling disparity estimation;
Fig. 2 is disparity estimation part flow chart in the present invention;
Fig. 3 is medial view composite part flow chart in the present invention;
Fig. 4 is the left view in original view;
Fig. 5 is the right view in original view;
Fig. 6 is the left view in preliminary synthetic result in the present invention;
Fig. 7 is the right view in preliminary synthetic result in the present invention;
Fig. 8 is the synthetic result of medial view final in the present invention, and location parameter is α=0.25;
Fig. 9 is the synthetic result of medial view final in the present invention, and location parameter is α=0.5;
Figure 10 is the synthetic result of medial view final in the present invention, and location parameter is α=0.75.
Embodiment
medial view synthetic method based on piece coupling disparity estimation of the present invention, the disparity estimation that use is mated based on piece obtains respectively the optical parallax field from the left view to the right view and from the right view to the left view, and carry out Intermediate View according to these two optical parallax fields respectively and synthesize, utilize at last optical parallax field and preliminary synthetic view computation to go out the optimal match point of each pixel in the view of left and right of view to be synthesized, be that every bit in view to be synthesized is composed brightness value according to correspondence position optimal match point luminance weighted, synthetic with the view of completing any one position on two view lines.
In the present embodiment, all " views ", " images " all refer to digital bitmap, and abscissa is for from left to right, and ordinate is for from top to bottom, all count since 0, and the pixel representation is (X, Y).Figure 1 shows that the medial view composition algorithm flow chart based on piece coupling disparity estimation of the present invention; Figure 2 shows that the flow process of disparity estimation part in the present invention; Figure 3 shows that medial view composite part flow chart, concrete steps are as follows:
(1) input is taken from Same Scene, synchronization, and the position for video camera is in two width images of same level height, requires this two width image only to there are differences taking on the visual angle.The binocular principle is satisfied in above requirement to one group of input picture, is equivalent to respectively with " right and left eyes ", scene be observed and record.And the work that the present invention will complete is namely synthesize and export between " right and left eyes " any point place scene is observed resulting result;
(2) if two width input pictures are coloured image, be translated into respectively gray level image; If two width input pictures are gray level image, execution in step (3).When coloured image is converted into gray-scale map, be respectively 0.11,0.59,0.30 according to three color component RGB proportions and be weighted, acquired results is the respective pixel gray value;
(3) judge whether two width input image sizes are identical, if different, the prompting mistake is also jumped out program; If identical, execution in step (4).If the left and right picture size for the synthesis of medial view is different, the respective pixel of two width figure can't be carried out view interpolation, thereby obtain final result;
(4) take right view as target image, left view is reference picture, the piece that target view is divided into fixed size, search for the most close with it piece in reference picture, calculate the displacement vector between match block in each target image piece and reference picture, i.e. the piece parallax d of left view in the right view
LRTake left view as target image, right view is reference picture again, and repeating step (4) is obtained the piece parallax d of right view in the left view
RL, idiographic flow as shown in Figure 2;
(5) according to the d that obtains in step (4)
LR, obtain through arrive the medial view I of right view piece parallax after preliminary View Synthesis based on left view according to the binocular vision principle of parallax
M LAccording to the d that obtains in step (4)
RL, obtain through arrive the medial view I of left view piece parallax after preliminary View Synthesis based on right view
M RDue to I
M LAnd I
M RThe parallax information that utilizes of solution procedure incomplete, ask I
M LThe time only use d
LR, and ask I
M RThe time only use d
RL, and weighting formula employing forward mapping, therefore I
M LAnd I
M RMiddle meeting produces the cavity, namely there is no the zone of filling, need to be further processed;
(6) for the I that obtains
M LWith I
M RThese two preliminary synthetic medial view for each pixel wherein, according to flow process shown in Figure 3, are sought respectively I one by one in left view and right view
M LWith I
M RIn the optimal match point I of each pixel
L(x
L, y) and I
R(x
R, y), and according to the gray value weighting of optimal match point, preliminary synthetic view is carried out the cavity and fill, finally obtain the synthetic I as a result of medial view
M
Wherein, the parallax estimation method based on the piece coupling in step described above (4), adopt following steps to realize:
(i) reference picture being expanded the limit processes, increase respectively k pixel unit (k must satisfy the scene content that only is present in a width view in comprising all binocular views on the left side of reference picture and right side, through experiment, the present invention gets k=60), and to make it be 0, and the part of augmenting is set to black.These black regions of augmenting will be for search during near the piece of target image left and right edges, makes the edge of reference picture be no more than the hunting zone;
(ii) target image is divided into the piece of M * N.General easy in order to explain when processing image, usually make M=N, in the middle of the example in Figure of description of the present invention, adopting size is that 16 * 16 piece mates;
(iii) obtain the SAD of target image and reference picture correspondence position piece, be used for initial value relatively during as search.
SAD is defined as the absolute value sum that the two the same number of array corresponding elements of group element subtract each other.In the present invention, block size is M * N, and upper left corner coordinate is that piece and the upper left corner coordinate in the target image of (m, n) is for absolute value error and SAD between the reference image block of (p, q):
Wherein, (m, n) is the pixel coordinate in the piece upper left corner in target image; (p, q) is the pixel coordinate in the upper left corner in reference picture; I
1, I
2Be respectively target image and reference picture at the gray value of a certain coordinate points;
This definition is to hereinafter all SAD are all applicable;
(iv) in reference picture, the search starting point is set to the upper left corner coordinate (m of target image piece, n), mate in abscissa zone is (m-60, m+60) scope, obtain the SAD of each matched position and target image interblock to be matched, make sad value obtain minimum reference picture correspondence position and be best matching blocks, if the coordinate of this best matching blocks is (p, q), and keep this SAD minimum value;
(v) the best matching blocks position (p, q) that searches is recorded, and obtained displacement vector d between object block and best matching blocks, take pixel as unit, wherein d (i, j)=(m-p, n-q), i.e. difference vector.In the present invention, in same level height photographs, so the component on the vertical direction of difference vector is zero, only has the component on horizontal direction due to getting input picture;
(vi) if what accept coupling is to be positioned at significant last piece of last column in target image, the piece in the target image lower right corner (not comprising the edge of augmenting) namely, its upper left corner coordinate be (X+60-M, Y+60-N), finishes to mate; Otherwise, find target image piece next to be matched, return to step (iii).Adopt this method that the target image that is divided into some is traveled through, obtain the most at last the parallax value of all pieces in target image;
(v) parallax take piece as unit is extended to take pixel as unit, i.e. d (i * M+m, j * N+n)=d (i, j); M ∈ [0, M-1] wherein, n ∈ [0, N-1].Take left view as reference picture, right view is target image, can calculate left view to the piece parallax d of right view
LRTake right view as reference picture, left view is target image again, can calculate right view to the piece parallax d of left view
RL
In step described above (5), the method for obtaining preliminary medial view according to the binocular vision principle of parallax is specific as follows:
Use is positioned at two video cameras of same level height, simultaneously scene is taken, and according to the binocular parallax principle, takes two views that obtain and has following geometrical relationship:
x
R＝x
L+d
LR(x，y)
I
R(x，y)＝I
L(x+d
LR(x，y)，y)
Wherein, x
LFor some abscissa in camera coordinate system of left view, take pixel as unit; x
RAbscissa for these corresponding points in right view; d
LRFor take left view as target image, right view is the parallax value that reference picture is obtained.
On this two video cameras line, virtual visual angle, optional position is synthetic, can use following luminance weighted formula to obtain:
I
M(x
L，y)＝(1-α)I
L(x
L，y)+αI
R(x
R，y)
Wherein, I
MBe the medial view that will synthesize, α is location parameter, makes α=0 be the left view position, and α=1 is the right view position, and α can represent optional position on straight line between original view (baseline) in (0,1) interval.And between Intermediate View to be synthesized and left and right original view, available following relation represents:
x
M＝x
L+αd
LR(x，y)＝x
R+(1-α)d
RL(x，y)
To can obtain respectively using d with co-relation substitution weighting formula
LRAnd d
RLThe I that obtains
M LWith I
M RAs follows:
I
M L(x
L+αd
LR，y)＝(1-α)I
L(x
L，y)+αI
R(x
L+d
LR，y)
I
M R(x
R+(1-α)d
RL，y)＝(1-α)I
L(x
R+d
RL，y)+αI
R(x
R，y)
Wherein, I
M LAnd I
M RThe medial view that obtains for preliminary View Synthesis; I
LAnd I
RBe respectively the left and right original view; x
LAnd x
RAll can travel through entire image.
Adopt these two formula to carry out medial view to Fig. 4, Fig. 5 and synthesize, synthetic view such as Fig. 6, shown in Figure 7.The view synthetic due to every width only utilized an optical parallax field, and namely synthetic view has only been used d take left view as target image
RL, and synthetic view has only been used d take right view as target image
LR, the shortage of parallax information makes synthetic result not accurate enough, exists than multiple error; More be apparent that, the composite formula that adopts in this step uses the forward mapping method, and namely the I of assignment is treated on the equation left side
M, coordinate can not traverse entire image, has a lot of blank zones that are not mapped in the middle of the view that therefore causes synthesizing, and very impact is viewed and admired, and can not be taken as final synthetic result.
In step described above (6), seek respectively view optimal match point to be synthesized in left view and right view, because the medial view of obtaining in step described above (5) lacks parallax information, and composite formula adopts the forward mapping method, there are error and a lot of white space in the middle of the view that causes synthesizing, very impact is viewed and admired, can not be taken as final synthetic result, so must further process, to not being mapped to such an extent that hole region is filled, method specifically adopts following steps to realize referring to the flow chart of Fig. 3:
(i) the traversal left view is to the disparity map d of right view
LR, obtain d
LRIn maximum d
LR max
(ii) in right image, obtain decision function g
L(x)=| x
I-(x+ α d
RL(x, y)) | at x ∈ [x
I-α d
RL max, x
I+ α d
RL max] interval interior minimum value min (g
L(x))=g
L(x
L), and preservation makes g
L(x) obtain minimum x value, be designated as x
L, x
LBe the optimal match point of this point.Wherein, x
IBe the abscissa of medial view mid point, value is x
I∈ [0, picture traverse-1]; Traversal view picture reference picture can obtain left view all optimal match points in right view;
(iii) obtain right view to the disparity map d of left view
RLIn maximum d
RL max
(iv) in left image, obtain decision function g
R(x)=| x
I-(x+ α d
RL(x, y)) | at x ∈ [x
I-α d
RL, x
I+ α d
RL] interval interior minimum value min (g
R(x))=g
R(x
R), and preservation makes g
R(x) obtain minimum x value, be designated as x
R, x
RBe the optimal match point of this point;
(v) according to luminance weighted formula I
M(x, y)=(1-α) I
L(x
L, y)+α I
R(x
R, y), finally obtain the synthetic I as a result of medial view
M, as Fig. 8, Fig. 9, Figure 10, its location parameter is respectively α=0.25, α=0.5, α=0.75.This formula takes full advantage of the geological information of left view, right view, and left view is to right view, and right view is to two optical parallax fields of left view, and the result that therefore obtains is more accurately.From the accompanying drawing examples prove, the present invention has obtained good visual effect really.
The content that is not described in detail in specification of the present invention belongs to the known prior art of this area professional and technical personnel.
Claims (1)
1. medial view synthetic method based on piece coupling disparity estimation is characterized in that performing step is as follows:
(1) input is taken from Same Scene, synchronization, and the position for video camera is in the left and right of same level height two width images, requires this two width image only to there are differences taking on the visual angle;
(2) if two width input pictures are coloured image, be translated into respectively gray level image; If two width input pictures are gray level image, execution in step (3);
(3) judge whether two width input image sizes are identical, if different, the prompting mistake is also jumped out; If identical, execution in step (4);
(4) take right view as target image, left view is reference picture, the piece that target image is divided into fixed size, the most close piece of each piece in search and target image respectively one by one in reference picture calculates the displacement vector d between match block in each target image piece and reference picture
LR, be the piece parallax of left view in the right view; Take left view as target image, right view is reference picture again, and repeating step (4) is obtained the piece parallax d of right view in the left view
RL
(5) according to the piece parallax d of the left view of obtaining in step (4) to right view
LR, utilize the binocular vision principle of parallax, obtain through arrive the medial view I of right view piece parallax after preliminary View Synthesis based on left view
M L, wherein subscript M represents medial view, subscript L represents left view to right view, according to the piece parallax d of the right view of obtaining in step (4) to left view
RL, in like manner obtain through arrive the medial view I of left view piece parallax after preliminary View Synthesis based on right view
M R, wherein subscript R represents that right view is to left view;
(6) for the I that obtains in step (5)
M LWith I
M RThese two preliminary synthetic medial view for each pixel wherein, are sought I one by one in left view and right view
M LWith I
M RIn the optimal match point I of each pixel
L(x
L, y) and I
R(x
R, y), I wherein
LAnd I
RRepresent the pixel gray value in left view and right view, the coordinate of (x, y) represent pixel point, x
LAnd x
RRepresent respectively the abscissa value of the optimal match point that finds in left view and right view, because binocular camera is in the same level height, so the ordinate value of the optimal match point that finds in left view and right view equates, represent with y, and according to the weighting of optimal match point to I
M LWith I
M RCarry out the cavity and fill, namely use gray value to fill up I
M LWith I
M RIn the white space that is not mapped to, finally obtain the synthetic I as a result of medial view
M
Being implemented as follows in described step (4):
(ⅰ) reference picture being expanded the limit processes, left side and right side at reference picture increase respectively k pixel unit, k must satisfy to comprise in all binocular views and only is present in the scene content of a width view, and to make the gray value of these pixels be 0, and the edge of augmenting is set to black;
(ⅱ) target image is divided into the piece of M * N, wherein M is every width, and N is length;
(ⅲ) obtain the SAD of target image and reference picture correspondence position piece, be used for initial value relatively during as search, wherein block size is M * N, upper left corner coordinate is (m, n) piece in target image and upper left corner coordinate are that absolute value error and the SAD between the reference image block of (p, q) is:
Wherein, (m, n) is the pixel coordinate in the piece upper left corner in target image, and (p, q) is the pixel coordinate in the upper left corner in reference picture, I
1, I
2Be respectively target image and reference picture at the gray value of a certain coordinate points; I gets respectively the integer from 0 to M-1; J gets respectively the integer from 0 to N-1;
(ⅳ) in reference picture, the search starting point is set to the upper left corner coordinate (m of target image piece, n), be (m-60 at abscissa zone, m+60) mate in scope, obtain the SAD of each matched position and target image interblock to be matched, make sad value obtain minimum reference picture correspondence position and be best matching blocks, and keep this SAD minimum value;
(ⅴ) the best matching blocks position that searches is recorded, and obtained displacement vector d between object block and best matching blocks, wherein d (i, j)=(m-p, n-q), i.e. parallax;
If (ⅵ) acceptance coupling is the piece that is positioned at the lower right corner in target image, do not comprise the edge of augmenting, namely upper left corner coordinate is (X+60-M, Y+60-N), finishes coupling; Otherwise, find target image piece next to be matched, return to step (ⅲ);
(ⅴ) parallax take piece as unit is extended to take pixel as unit, i.e. d (i * M+m, j * N+n)=d (i, j), m ∈ [0, M-1] wherein, n ∈ [0, N-1], take left view as reference picture, right view is target image, calculates left view to the piece parallax d of right view
LRTake right view as reference picture, left view is target image again, calculates right view to the piece parallax d of left view
RL
Utilize the binocular vision principle of parallax to obtain the method for the medial view that preliminary View Synthesis obtains in described step (5) as follows:
Be positioned at two video cameras of same level height, simultaneously scene taken, according to the binocular parallax principle, have:
x
R＝x
L+d
LR(x,y)
I
R(x,y)＝I
L(x+d
LR(x,y),y)
Wherein, x
LFor some abscissa in camera coordinate system of left view, take pixel as unit; x
RAbscissa for these corresponding points in right view; d
LRFor take left view as reference picture, right view is the parallax value that target image is obtained,
The following luminance weighted acquisition of the synthetic use at virtual visual angle:
I
M(x
L,y)＝(1-α)I
L(x
L,y)+αI
R(x
R,y)
If α is location parameter, make α=0 be the left view position, α=1 is the right view position, α is (0,1) can represent straight line between original view in interval, i.e. optional position on baseline, and medial view to be synthesized and left and right original view Relations Among are expressed as:
x
M＝x
L+αd
LR(x,y)＝x
R+(1-α)d
RL(x,y)
To can obtain respectively using d with co-relation substitution weighting formula
LRAnd d
RLThe I that obtains
M LWith I
M RAs follows:
I
M L(x
L+αd
LR,y)＝(1-α)I
L(x
L,y)+αI
R(x
L+d
LR,y)
I
M R(x
R+(1-α)d
RL,y)＝(1-α)I
L(x
R+d
RL,y)+αI
R(x
R,y)
Wherein, I
M LAnd I
M RThe medial view that obtains for preliminary View Synthesis; I
LAnd I
RBe respectively the left and right original view; x
LAnd x
RAll can travel through entire image;
Described step (6) is implemented as follows:
(ⅰ) obtain left view to the disparity map d of right view
LRIn maximum d
LR max
(ⅱ) obtain function g
L(x)=| x
I-(x+ α d
RL(x, y)) | at x ∈ [x
I-α d
RL max, x
I+ α d
RL max] interval interior minimum value min (g
L(x))=g
L(x
L), and preservation makes g
L(x) obtain minimum x value, be designated as x
L, wherein, x
IBe the abscissa of medial view mid point, value is x
I∈ [0, picture traverse-1];
(ⅲ) obtain right view to the disparity map d of left view
RLIn maximum d
RL max
(ⅳ) obtain decision function g
R(x)=| x
I-(x+ α d
RL(x, y)) | at x ∈ [x
I-α d
RL, x
I+ α d
RL] interval interior minimum value min (g
R(x))=g
R(x
R), and preserve x
R
(ⅴ) according to luminance weighted formula I
M(x, y)=(1-α) I
L(x
L, y)+α I
R(x
R, y), finally obtain the synthetic I as a result of medial view
M
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN 201110041538 CN102075779B (en) | 2011-02-21 | 2011-02-21 | Intermediate view synthesizing method based on block matching disparity estimation |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN 201110041538 CN102075779B (en) | 2011-02-21 | 2011-02-21 | Intermediate view synthesizing method based on block matching disparity estimation |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN102075779A CN102075779A (en) | 2011-05-25 |
| CN102075779B true CN102075779B (en) | 2013-05-08 |
Family
ID=44034099
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN 201110041538 Expired - Fee Related CN102075779B (en) | 2011-02-21 | 2011-02-21 | Intermediate view synthesizing method based on block matching disparity estimation |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN102075779B (en) |
Families Citing this family (20)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102254348B (en) * | 2011-07-25 | 2013-09-18 | 北京航空航天大学 | Virtual viewpoint mapping method based o adaptive disparity estimation |
| CN102263957B (en) * | 2011-07-25 | 2013-07-03 | 北京航空航天大学 | Search-window adaptive parallax estimation method |
| JP2013089981A (en) * | 2011-10-13 | 2013-05-13 | Sony Corp | Image processing apparatus, image processing method and program |
| CN103108200B (en) * | 2011-11-09 | 2016-03-02 | 晨星软件研发(深圳)有限公司 | To the processing method of stereopsis |
| JP5953916B2 (en) * | 2012-05-02 | 2016-07-20 | ソニー株式会社 | Image processing apparatus and method, and program |
| TWI547904B (en) * | 2012-05-31 | 2016-09-01 | 財團法人工業技術研究院 | Hole filling method for multi-view disparity map |
| US9076249B2 (en) | 2012-05-31 | 2015-07-07 | Industrial Technology Research Institute | Hole filling method for multi-view disparity maps |
| CN104065954B (en) * | 2014-07-03 | 2016-05-18 | 中国传媒大学 | A kind of disparity range method for quick of high definition three-dimensional video-frequency |
| WO2016172385A1 (en) * | 2015-04-23 | 2016-10-27 | Ostendo Technologies, Inc. | Methods for full parallax compressed light field synthesis utilizing depth information |
| CN105007478A (en) * | 2015-07-08 | 2015-10-28 | 南京信息工程大学 | View synthesis method based on mean shift stereo matching |
| CN105657401B (en) * | 2016-01-13 | 2017-10-24 | 深圳创维－Rgb电子有限公司 | A kind of bore hole 3D display methods, system and bore hole 3D display device |
| WO2018014324A1 (en) * | 2016-07-22 | 2018-01-25 | 北京大学深圳研究生院 | Method and device for synthesizing virtual viewpoints in real time |
| CN107392965B (en) * | 2017-07-24 | 2020-10-30 | 王华锋 | Range finding method based on combination of deep learning and binocular stereo vision |
| CN107493465B (en) * | 2017-09-18 | 2019-06-07 | 郑州轻工业学院 | A kind of virtual multi-view point video generation method |
| TWI701639B (en) * | 2018-12-11 | 2020-08-11 | 緯創資通股份有限公司 | Method of identifying foreground object in image and electronic device using the same |
| CN110177216B (en) * | 2019-06-28 | 2021-06-15 | Oppo广东移动通信有限公司 | Image processing method, image processing device, mobile terminal and storage medium |
| CN110473247A (en) * | 2019-07-30 | 2019-11-19 | 中国科学院空间应用工程与技术中心 | Solid matching method, device and storage medium |
| CN111464804A (en) | 2020-04-08 | 2020-07-28 | 北京小米松果电子有限公司 | Omnidirectional parallax view synthesis method and device and storage medium |
| CN112365400B (en) * | 2020-10-27 | 2024-05-28 | 深圳大学 | Rapid light field angle super-resolution reconstruction method |
| CN113132706A (en) * | 2021-03-05 | 2021-07-16 | 北京邮电大学 | Controllable position virtual viewpoint generation method and device based on reverse mapping |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1688168A (en) * | 2005-06-24 | 2005-10-26 | 清华大学 | Visual difference calculating method for video image in multi-vision point system |
| CN1767655A (en) * | 2005-10-18 | 2006-05-03 | 宁波大学 | Multi view point video image parallax difference estimating method |
| CN101312540A (en) * | 2008-07-03 | 2008-11-26 | 浙江大学 | Virtual visual point synthesizing method based on depth and block information |
| CN101873508A (en) * | 2010-05-25 | 2010-10-27 | 东南大学 | Intermediate view synthesis method based on improved BP (Belief Propagation) algorithm |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20070159476A1 (en) * | 2003-09-15 | 2007-07-12 | Armin Grasnick | Method for creating a stereoscopic image master for imaging methods with three-dimensional depth rendition and device for displaying a steroscopic image master |
-
2011
- 2011-02-21 CN CN 201110041538 patent/CN102075779B/en not_active Expired - Fee Related
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1688168A (en) * | 2005-06-24 | 2005-10-26 | 清华大学 | Visual difference calculating method for video image in multi-vision point system |
| CN1767655A (en) * | 2005-10-18 | 2006-05-03 | 宁波大学 | Multi view point video image parallax difference estimating method |
| CN101312540A (en) * | 2008-07-03 | 2008-11-26 | 浙江大学 | Virtual visual point synthesizing method based on depth and block information |
| CN101873508A (en) * | 2010-05-25 | 2010-10-27 | 东南大学 | Intermediate view synthesis method based on improved BP (Belief Propagation) algorithm |
Non-Patent Citations (6)
| Title |
|---|
| 吕朝辉等.基于图像的中间视合成方法综述.《电视技术》.2009,第33卷(第6期),26-29. |
| 基于图像的中间视合成方法综述;吕朝辉等;《电视技术》;20090630;第33卷(第6期);26-29 * |
| 应用于多视点视频编码的基于深度特征的图像区域分割与区域视差估计;杨海涛等;《光学学报》;20080630;第28卷(第6期);1073-1078 * |
| 杨海涛等.应用于多视点视频编码的基于深度特征的图像区域分割与区域视差估计.《光学学报》.2008,第28卷(第6期),1073-1078. |
| 梁广民.立体视频会议中基于对象的中间视生成.《计算机工程与应用》.2005,73-75. |
| 立体视频会议中基于对象的中间视生成;梁广民;《计算机工程与应用》;20050930;73-75 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN102075779A (en) | 2011-05-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102075779B (en) | Intermediate view synthesizing method based on block matching disparity estimation | |
| CN102254348B (en) | Virtual viewpoint mapping method based o adaptive disparity estimation | |
| CN112435325B (en) | VI-SLAM and depth estimation network-based unmanned aerial vehicle scene density reconstruction method | |
| CN103400409B (en) | A kind of coverage 3D method for visualizing based on photographic head attitude Fast estimation | |
| US7505623B2 (en) | Image processing | |
| CN102665086B (en) | Method for obtaining parallax by using region-based local stereo matching | |
| CN111047510A (en) | Large-field-angle image real-time splicing method based on calibration | |
| CN104780355A (en) | Depth-based cavity repairing method in viewpoint synthesis | |
| CN102263957B (en) | Search-window adaptive parallax estimation method | |
| CN103702103B (en) | Based on the grating stereo printing images synthetic method of binocular camera | |
| CN108776989A (en) | Low texture plane scene reconstruction method based on sparse SLAM frames | |
| US20140147031A1 (en) | Disparity Estimation for Misaligned Stereo Image Pairs | |
| CN109769109A (en) | Method and system based on virtual view synthesis drawing three-dimensional object | |
| CN110245199A (en) | A kind of fusion method of high inclination-angle video and 2D map | |
| CN103634584B (en) | A kind of multiple views 3D image synthesizing method | |
| CN104065946A (en) | Cavity filling method based on image sequence | |
| CN114693720A (en) | Design method of monocular vision odometer based on unsupervised deep learning | |
| CN103679739A (en) | Virtual view generating method based on shielding region detection | |
| Tzovaras et al. | 3D object articulation and motion estimation in model-based stereoscopic videoconference image sequence analysis and coding | |
| Bleyer et al. | Temporally consistent disparity maps from uncalibrated stereo videos | |
| CN113034681B (en) | Three-dimensional reconstruction method and device for spatial plane relation constraint | |
| CN107767393B (en) | Scene flow estimation method for mobile hardware | |
| CN112927348A (en) | High-resolution human body three-dimensional reconstruction method based on multi-viewpoint RGBD camera | |
| CN104284177A (en) | Convergence stereo image parallax control method | |
| Agouris et al. | Automation and digital photogrammetric workstations |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| C17 | Cessation of patent right | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20130508 Termination date: 20140221 |