CN100576934C - Virtual visual point synthesizing method based on the degree of depth and block information - Google Patents
Virtual visual point synthesizing method based on the degree of depth and block information Download PDFInfo
- Publication number
- CN100576934C CN100576934C CN 200810062811 CN200810062811A CN100576934C CN 100576934 C CN100576934 C CN 100576934C CN 200810062811 CN200810062811 CN 200810062811 CN 200810062811 A CN200810062811 A CN 200810062811A CN 100576934 C CN100576934 C CN 100576934C
- Authority
- CN
- China
- Prior art keywords
- image
- virtual
- view
- visual point
- viewpoint
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Landscapes
- Processing Or Creating Images (AREA)
Abstract
The invention discloses a kind of virtual visual point synthesizing method based on the degree of depth and block information.May further comprise the steps: 1) determine 3D drafting equation based on the method for drafting of depth image; 2) according to the locus of reference view and virtual view, formulate the pixel drawing order; 3) utilization with reference to main viewpoint, is drawn the image that obtains virtual view based on the horizontal half-pix method for drafting of depth image; 4) according to the block information of auxilliary viewpoint, virtual visual point image is blocked compensation, obtain high-resolution virtual visual point image; 5) high-resolution image is carried out conversion, obtain the image of original resolution; 6) utilize a kind of asymmetrical linear interpolation method to carry out remaining cavity compensation; 7) it is level and smooth to carry out blocking effect, obtains high-quality virtual view composograph.The present invention is applicable to parallel optical axis camera array model, and virtual view to be synthesized can synthesize high-quality virtual visual point image efficiently between main viewpoint and auxilliary viewpoint.
Description
Technical field
The present invention relates to image processing field, relate in particular to a kind of virtual visual point synthesizing method based on the degree of depth and block information.
Background technology
Along with the video development of technologies, people pursue free view-point and stereoscopic vision gradually for Video Applications of future generation, and this requirement can the corresponding three-dimensional scenic of complete reconstruction at display end.
To obtain the free stereo vision at display end, rebuild corresponding three-dimensional scenic, promptly need to obtain many viewpoints of this three-dimensional scenic so that the video image of any viewpoint.At this purpose, mainly contain two kinds of approach now and realize: a kind of 3D modeling method that is based on model; Another kind is based on the synthetic method of image.The view-point image quality of rebuilding based on the 3D modeling method of model is higher, but computational complexity is also very high simultaneously, the marked change of its complexity along with scene content, and generally need special image processing hardware to realize; Complexity is lower comparatively speaking based on the synthetic method of image, and the viewpoint that is more suitable for the nature three-dimensional scenic is rebuild.Based on image the viewpoint method for reconstructing be the image of rebuilding other viewpoints (virtual view) by one or more viewpoints (reference view) image, general reference view is many more, the image of virtual view is also good more.Increasing of reference view number brought great challenge to the transmission bandwidth of network.Therefore,, experience in order to obtain real free stereo vision on the other hand, need a kind of virtual visual point synthesizing method efficiently in a hurry on the one hand in order to save the network bandwidth.
Based on image virtual visual point synthesizing method, present realization approach mainly contains two kinds.A kind of multiple view video coding (Multi-view Video Coding, virtual visual point synthesizing method MVC) of being based on.MVC, it utilizes between each viewpoint correlation to compress, its encode complete two-dimensional video frame of main viewpoint, and the difference vector of all the other each road viewpoints and residual error, its code check can generate virtual visual point image along with the increase linear growth of viewpoint number by the image of interpolation adjacent viewpoint in decoding end.Another kind method is the virtual visual point synthesizing method that the two-dimensional video of ATTEST employing adds its degree of depth.The ATTEST system is at coding side only encode one tunnel reference video and depth information thereof, and (depth image based rendering DIBR), can reconstruct the two dimensional image of any viewpoint to adopt method for drafting based on depth image in decoding end.Compare with virtual visual point synthesizing method based on MVC, the method that this two-dimensional video adds its degree of depth only need encode one road two-dimensional video and one road depth map, the Network Transmission burden is smaller, and can recover the virtual visual point image of arbitrary position in decoding end; But because object blocks the existence of phenomenon mutually, significantly cavity appears in the inner meeting of virtual visual point image that utilizes depth information to recover to obtain, and along with the position of virtual view is departed from reference view gradually, cavitation can be more serious, causes the quality of virtual view to descend gradually.
In order to control code check, the MVC coding method has bigger restriction to the viewpoint number that is encoded, and causes reference view fewer, can only interpolation goes out to have the virtual view of extreme position in decoding end; And because object is subjected to the influence of its depth of field very big at the image space of different points of view, and general its depth of field possibility difference of different objects is very big, therefore be difficult to provide a kind of efficiently based on the viewpoint interpolation method that is close to two dimensional image, with the more accurate virtual view that recovers.
The method that the two-dimensional video that ATTEST adopts adds its degree of depth can recover the virtual visual point image of optional position, and its key technology is based on the method for drafting of depth image.Method for drafting based on depth image is a kind of process of utilizing reference picture and corresponding depth information thereof to synthesize virtual view in the scene.Method for drafting based on depth image mainly can be divided into following two steps: (1) utilizes depth information, with the position of pixel re-projections all on the reference picture in 3 dimension spaces of their correspondences; (2) these 3 dimension space points are projected to the plane of delineation of virtual view again.This is tieed up the re-projections of 3 dimensions and is called the 3 d image conversion to the projections of 2 dimensions again from 3 dimensions from 2, and its core is that it draws equation, can be expressed as:
Wherein, m and
It is respectively the pixel coordinate in reference picture and the drawn image; Z and
Be respectively the depth value of three dimensions point under reference camera and drawn camera coordinates system; Matrix A and
It is respectively the confidential reference items matrix of reference camera and drawn camera; R and T refer to spin matrix and the translation matrix of drawn camera with respect to reference camera respectively.Existing method for drafting based on depth image, its main shortcoming are effectively to solve occlusion issue, and it is of low quality to cause drawing the virtual visual point image that obtains.
Block mechanism that phenomenon produces as shown in Figure 4.Its basic reason is that the degree of depth is not uniformly continuous in the scene, in some place is to have sudden change, and the general position that occurs is the boundary vicinity of object.Occlusion issue can be divided into two classifications: (1) overlap problem; (2) empty problem; The cavity problem is segmented and can also be divided into local cavity and overall situation cavity.Overlap problem and empty problem all are comparatively speaking, and viewpoint is chosen when changing, and overlap problem originally can be transformed into empty problem, and original empty problem also might be transformed into overlap problem.Overlap problem is meant: at the reference view imaging plane, the two or several objects that are imaged on diverse location are imaged on same position in new viewpoint, occur overlappingly, must judge at this moment which object is visible in new viewpoint.The cavity problem is meant: become in new viewpoint as seen at the sightless object of reference view, when according to reference view prediction new viewpoint, these pixels can not obtain by arbitrary pixel prediction of reference view, so-called cavity just occurs in these positions.Solving now the fundamental method of overlap problem is the Z-buffer algorithm.This method can reasonable elimination eclipsing effects, and just, though this algorithm is general to any camera array, the cost that realizes is than higher.Solve empty problem, mainly contain two kinds of methods now: a kind of is to utilize background, texture, neighbor etc. to filling in the cavity; Another kind is that depth map is carried out preliminary treatment, mainly it is carried out smoothing processing, and the discontinuity of depth of smoothness is drawn the cavity that the back produces to reduce image.Filling technique can not be from solving empty problem in essence, and is particularly undesirable to blocking the more serious video sequence effect of phenomenon; And smoothly preconditioning technique is owing to be out of shape object, and the visual quality of the virtual view that obtains has apparent in view decline.
Recover more accurate virtual view, the decrease in image quality of avoiding occlusion issue to bring, remedy based on MVC, two-dimensional video and add these two kinds of deficiencies that virtual visual point synthesizing method brings of its degree of depth, the method that also has now is to adopt the virtual visual point synthesizing method of the many depth maps of many reference pictures, but must transmit the more degree of depth and supplementary like this, code check significantly increases, and the burden to transmission network is not suitable for practical application.
One side will solve based on MVC, two-dimensional video and add these two kinds of deficiencies that virtual visual point synthesizing method brings of its degree of depth, will prevent the aggravation of code check and complexity on the other hand.Based on the consideration of code efficiency and virtual view quality, in the building-up process of virtual view, we propose to add on the basis of its degree of depth in two-dimentional reference video, and the block information in conjunction with auxilliary viewpoint can recover mass ratio virtual visual point image preferably like this.The position of reference view, auxilliary viewpoint, virtual view as shown in Figure 2, the block information of auxilliary viewpoint comprises the pixel value and the corresponding depth value thereof in the cavity that obtains according to the auxilliary viewpoint of reference view prediction, as shown in Figure 8, this part block information can pass through auxiliary data transmission at coding side, the synthetic general principles block diagram of virtual view as shown in Figure 3 (since block information be with viewpoint position one to one, if each road virtual view that needs are recovered all passes one road block information, along with the increase of virtual view number, data volume also can linear growth.Therefore, generally only pass one the road or the block information of several roads viewpoint, have the viewpoint of block information to be called auxilliary viewpoint) at coding side.The main shortcoming of method that two-dimensional video adds its depth recovery virtual view is that empty problem is difficult to solve, and the appearance of empty problem is because some object area is invisible at reference view, and at virtual view as seen, these zones just can not obtain from the reference view prediction, cause occurring in the virtual view relevant position empty.Consider that these zones can be at virtual view as seen, invisible at reference view, may be again visible so in some viewpoint outside this two viewpoint, so just can solve the empty problem of virtual view according to the block information of another one viewpoint.As shown in Figure 2, when the virtual view position in the middle of reference view and auxilliary viewpoint, the parallax range of reference view and auxilliary viewpoint is within the specific limits the time, so almost so to say that: at reference view as seen in the visible zone of virtual view is not, be exactly as seen in auxilliary viewpoint, perhaps in two viewpoints all as seen.Like this, we propose: according to video image and its depth map of reference view, and auxilliary viewpoint is with respect to the block information of reference view, the virtual visual point image of the optional position of recovery position that just can be more accurate between reference view and auxilliary viewpoint.
This virtual visual point synthesizing method that we propose based on the degree of depth and block information, with add these two kinds of virtual visual point synthesizing methods of its degree of depth based on MVC with two-dimensional video and compare, on the one hand because it has solved occlusion issue, therefore can synthesize the virtual visual point image of better quality, it is lower to the burden of transmission network on the other hand, it only needs one road reference video image, one tunnel corresponding with it depth information, and one the tunnel auxilliary viewpoint with respect to the block information of reference view, this block information comprises the pixel value and the corresponding depth value thereof in the cavity that obtains after according to reference view auxilliary viewpoint being predicted.Because general empty coverage is smaller, so the data volume of this road block information is very little than depth map or video image.Simultaneously, under parallel optical axis camera array model, in the building-up process of virtual view, we have proposed a kind ofly have certain pixel drawing order, based on the horizontal half-pix method for drafting of depth image, with precision and the basic solution overlap problem that improves coupling, thereby can synthesize virtual visual point image more accurately.
Summary of the invention
The objective of the invention is to overcome the deficiencies in the prior art, a kind of virtual visual point synthesizing method based on the degree of depth and block information is provided.
May further comprise the steps:
1) under parallel optical axis camera array model, determines 3D drafting equation based on the method for drafting of depth image;
2) according to the locus of reference view and virtual view, formulate the pixel drawing order;
3) according to the drawing order of formulating, utilize horizontal half-pix method for drafting based on depth image, with reference to main viewpoint, draw the image that obtains virtual view;
4) according to the block information of auxilliary viewpoint, utilize horizontal half-pix method for drafting, virtual visual point image is blocked compensation, obtain high-resolution virtual visual point image based on depth image;
5) high-resolution virtual visual point image is carried out conversion, obtain the virtual visual point image of original resolution;
6) utilize asymmetrical linear interpolation method to carry out remaining cavity compensation;
7) it is level and smooth to carry out blocking effect, obtains high-quality virtual view composograph.
Described locus according to reference view and virtual view, formulate pixel drawing order step:
A) under parallel optical axis camera array model, when overlapping generation, judge their hiding relations in the position of reference view at virtual view according to pixel;
Virtual view is before drawn, and all pixel values all are zero; At reference view x
1The pixel of position is through drawing the x of back at virtual view
2The position is as if x this moment
2The pixel value of position is non-vanishing, then overlapping generation.According to given parallel camera model, according to
F is a camera focus, and c is the horizontal displacement of virtual view with respect to reference view, and z is the depth of field, and regulation object a blocks object b should be satisfied:
x
1(a), x
1(b) be object a, b is imaged on the horizontal coordinate of reference view;
x
2(a), x
2(b) be object a, b is imaged on the horizontal coordinate of drawn viewpoint
B) block the relation that should satisfy according to the object that obtains in a), locus in conjunction with reference view and virtual view, formulate a kind of image drawing order that under given camera model, carries out to solve substantially when image is drawn overlap problem, if virtual view is on the reference view right side, then for the pixel in the reference view, draw according to from left to right reference sequence line by line; If virtual view is in reference view left side,, draw according to dextrosinistral reference sequence line by line then for the pixel in the reference view.
Described according to the drawing order of formulating, utilize horizontal half-pix method for drafting based on depth image, with reference to main viewpoint, draw the image step that obtains virtual view:
C) raising obtains a kind of horizontal half-pix method for drafting based on depth image based on the matching precision of the method for drafting of depth image;
1. the depth image that obtains according to the first step is drawn equation, calculate object the accurate coordinates of virtual view (x, y);
2. be the virtual visual point image coordinate transformation of H*V resolution the image coordinate of 2H*V resolution, the span of horizontal coordinate x expands to [0,2H] from [0, H];
3. (x y) carries out horizontal half-pix correction, changes (x into to coordinate
2, y), x wherein
2=2x is again to x
2Round nearby;
D) utilization according to the drawing order of formulating, with reference to main viewpoint, is drawn the image that obtains virtual view based on the horizontal half-pix method for drafting of depth image;
4. according to the drawing order of formulating, for arbitrary pixel (x of main viewpoint
1, y), utilize horizontal half-pix method for drafting based on depth image, calculate its coordinate (x in virtual view
2, y) overlapping as if not detecting, be (x then with coordinate in the virtual view
2, coordinate is (x in pixel value y) and the main viewpoint
1, pixel value unanimity y);
5. if detect overlappingly,, directly cover coordinate (x when virtual view during on main viewpoint right side
2, pixel value y) when virtual view is on the left of reference view, keeps coordinate (x
2, y) original pixel value;
6. repeating step 4. 5., all pixels traversal is finished in main visual point image, generates the image of high-resolution virtual view at last.
The block information of the auxilliary viewpoint of described basis utilizes the horizontal half-pix method for drafting based on depth image, and virtual visual point image is blocked compensation, obtains high-resolution virtual visual point image step:
E) obtain the block information of auxilliary viewpoint, the locus of virtual view, auxilliary viewpoint;
Block information comprises the pixel value and the depth value in auxilliary viewpoint cavity; According to parallel optical axis camera array model, virtual view and auxilliary viewpoint also have only horizontal displacement; The locus of block information and virtual view, auxilliary viewpoint can obtain at the auxiliary data partial decoding of h of decoding end from encoded data stream in coding side these information of encoding;
F) according to step 2) drawing order formulated, utilize horizontal half-pix method for drafting method based on depth image, the block information based on auxilliary viewpoint carries out the drafting of block information to virtual visual point image, realizes blocking compensation;
Described high-resolution virtual visual point image is carried out conversion, obtains the virtual visual point image step of original resolution:
G) determine a kind of transform method that changes low resolution from high-resolution into;
7. make that (x y) is the pixel coordinate of image; Function f (x, y), g (x, y) be respectively the H*V image coordinate for (x, y) and the 2H*V image coordinate be (x, y) pixel value of position; Function h (x, y) expression: when g (x, when y) being not equal to zero, h (x, y)=1; When g (x, when y) equalling zero, h (x, y)=0;
8. function f (x, y) with g (x, mapping relations y) are:
H) utilize step mapping relations 8., high-resolution virtual visual point image is carried out conversion, generate virtual visual point image with original resolution.
The described empty compensation process that utilizes asymmetrical linear interpolation method to carry out remnants:
I), adopt and a kind ofly carry out the cavity at vertical and the asymmetrical linear interpolation method of horizontal direction and compensate to the cavity of remnants;
9. detect cavity position, be made as (x, y); Function f (x, y) denotation coordination is (x, y) pixel value; Function h (x, y) expression: when f (x, when y) being not equal to zero, h (x, y)=1; When f (x, when y) equalling zero, h (x, y)=0; Threshold value t
1, t
2, t wherein
1＞t
2
10. empty (x, compensation equation y) is:
Utilize the compensation equation of step in 10. that virtual visual point image is carried out remaining cavity compensation, can obtain the better virtual visual point image of visual quality.
The present invention is applicable to parallel optical axis camera array model, and virtual view to be synthesized can synthesize high-quality virtual visual point image efficiently between main viewpoint and auxilliary viewpoint.Than other virtual visual point synthesizing methods, the present invention has improved the precision of drawing by existing method for drafting based on depth image is optimized; In drawing process, adopt specific pixel drawing order, solved the overlap problem in the drawing process substantially; Block information according to reference view and auxilliary viewpoint carries out dual drafting, has solved occlusion issue, improves the picture quality of synthetic virtual view greatly.
Description of drawings
Fig. 1 is a parallel optical axis camera model schematic diagram;
Fig. 2 is main viewpoint, virtual view, auxilliary viewpoint position schematic diagram;
Fig. 3 is the general principles schematic diagram of synthetic virtual view;
Fig. 4 is for producing the principle schematic of blocking;
Fig. 5 is the result schematic diagram based on the method for drafting of general depth image;
Fig. 6 is a high-resolution virtual visual point image schematic diagram;
Fig. 7 is an original resolution virtual visual point image schematic diagram;
Fig. 8 is a block information content schematic diagram.
Embodiment
Virtual visual point synthesizing method based on the degree of depth and block information may further comprise the steps:
1) under parallel optical axis camera array model, determines 3D drafting equation based on the method for drafting of depth image;
The camera of reference view and drawn viewpoint meets parallel optical axis camera array model, as shown in Figure 1, can simplify the drafting equation based on the method for drafting of depth image like this.
With reference to parallel optical axis camera array model shown in Figure 1, camera O
1Be reference view, camera O
2Be drawn viewpoint, camera O
1Coordinate system overlap with world coordinate system, camera focus all is f, drawn viewpoint O
2With respect to reference view O
1Have only horizontal direction displacement c, not rotation, same object is the same with respect to reference view with the depth z of drawn viewpoint, it is imaged on the displacement of also having only horizontal direction in the coordinate system of two viewpoints so.For the object A with depth of field z, its world coordinates is that (z), then object A is imaged on reference view O for x, y
1The horizontal coordinate of imaging plane is:
Be imaged on drawn viewpoint O
2The horizontal coordinate of imaging plane is:
Coordinate (the x of known object in reference view
1, y
1), its coordinate that is plotted in the drawn viewpoint is (x
2, y
2), then the drafting equation based on the method for drafting of depth image can abbreviation be:
Promptly under parallel optical axis camera model condition, the method for drafting based on depth image of virtual view is:
Arbitrary pixel (x for reference view
1, y), utilize formula 1, calculate its new coordinate (x in drawn viewpoint
2, y), be (x with coordinate in the drawn viewpoint then
2, coordinate is (x in pixel value y) and the reference view
1, the value unanimity of pixel y) so just can be finished image and draw, and obtains the image of new viewpoint.
2) according to the locus of reference view and virtual view, formulate the pixel drawing order;
3) according to the drawing order of formulating, utilize horizontal half-pix method for drafting based on depth image, with reference to main viewpoint, draw the image that obtains virtual view;
4) according to the block information of auxilliary viewpoint, utilize horizontal half-pix method for drafting, virtual visual point image is blocked compensation, obtain high-resolution virtual visual point image based on depth image;
5) high-resolution virtual visual point image is carried out conversion, obtain the virtual visual point image of original resolution;
6) utilize asymmetrical linear interpolation method to carry out remaining cavity compensation;
7) it is level and smooth to carry out blocking effect, obtains high-quality virtual view composograph.
Described locus according to reference view and virtual view, formulate pixel drawing order step:
A) under parallel optical axis camera array model, when overlapping generation, judge their hiding relations in the position of reference view at virtual view according to pixel;
Virtual view is before drawn, and all pixel values all are zero; At reference view x
1The pixel of position is through drawing the x of back at virtual view
2The position is as if x this moment
2The pixel value of position is non-vanishing, then overlapping generation.According to given parallel camera model, according to
F is a camera focus, and c is the horizontal displacement of virtual view with respect to reference view, and z is the depth of field, and regulation object a blocks object b should be satisfied:
x
1(a), x
1(b) be object a, b is imaged on the horizontal coordinate of reference view;
x
2(a), x
2(b) be object a, b is imaged on the horizontal coordinate of drawn viewpoint
B) block the relation that should satisfy according to the object that obtains in a), locus in conjunction with reference view and virtual view, formulate a kind of image drawing order that under given camera model, carries out to solve substantially when image is drawn overlap problem, if virtual view is on the reference view right side, then for the pixel in the reference view, draw according to from left to right reference sequence line by line; If virtual view is in reference view left side,, draw according to dextrosinistral reference sequence line by line then for the pixel in the reference view.
When overlapping generation, if virtual view on the reference view right side, then must be that the object that horizontal coordinate is bigger in reference view blocks the smaller object of horizontal coordinate; Otherwise, if virtual view then must be that the smaller object of horizontal coordinate blocks the bigger object of horizontal coordinate in reference view in the reference view left side.Reference view (main viewpoint), virtual view, the position of auxilliary viewpoint is as shown in Figure 2 the time, then for the pixel in the reference view, draws according to from left to right reference sequence line by line;
Described according to the drawing order of formulating, utilize horizontal half-pix method for drafting based on depth image, with reference to main viewpoint, draw the image step that obtains virtual view:
Draw equation according to the method for drafting based on depth image of formula 1 expression image is drawn the drawing image that obtains as shown in Figure 5.The new locations of pixels major part that obtains with the general method for drafting based on depth image can drop on the non-whole location of pixels as can be seen, if directly the value of pixel is nearby composed to nearest whole pixel, the error ratio that pixel causes when should whole pixel distance distant nearby is bigger.A kind of simple but effective method is: the depth image that the image level direction is carried out half-pix is drawn.
C) raising obtains a kind of horizontal half-pix method for drafting based on depth image based on the matching precision of the method for drafting of depth image;
1. the depth image that obtains according to the first step is drawn equation, calculate object the accurate coordinates of virtual view (x, y);
2. be the virtual visual point image coordinate transformation of H*V resolution the image coordinate of 2H*V resolution, the span of horizontal coordinate x expands to [0,2H] from [0, H];
3. (x y) carries out horizontal half-pix correction, changes (x into to coordinate
2, y), x wherein
2=2x is again to x
2Round nearby;
D) utilization according to the drawing order of formulating, with reference to main viewpoint, is drawn the image that obtains virtual view based on the horizontal half-pix method for drafting of depth image;
4. according to the drawing order of formulating, for arbitrary pixel (x of main viewpoint
1, y), utilize horizontal half-pix method for drafting based on depth image, calculate its coordinate (x in virtual view
2, y) overlapping as if not detecting, be (x then with coordinate in the virtual view
2, coordinate is (x in pixel value y) and the main viewpoint
1, pixel value unanimity y);
5. if detect overlappingly,, directly cover coordinate (x when virtual view during on main viewpoint right side
2, pixel value y) when virtual view is on the left of reference view, keeps coordinate (x
2, y) original pixel value;
6. repeating step 4. 5., all pixels traversal is finished in main visual point image, generates the image of high-resolution virtual view at last.
The block information of the auxilliary viewpoint of described basis utilizes the horizontal half-pix method for drafting based on depth image, and virtual visual point image is blocked compensation, obtains high-resolution virtual visual point image step:
E) obtain the block information of auxilliary viewpoint, the locus of virtual view, auxilliary viewpoint;
Block information comprises the pixel value and the depth value in auxilliary viewpoint cavity; According to parallel optical axis camera array model, virtual view and auxilliary viewpoint also have only horizontal displacement; The locus of block information and virtual view, auxilliary viewpoint can obtain at the auxiliary data partial decoding of h of decoding end from encoded data stream in coding side these information of encoding;
Information among the block information figure of auxilliary viewpoint in the middle heavy line usefully shown in Figure 9, pixel value that heavy line is outer and depth value thereof all are zero, when carrying out the block information drafting, if pixel value is zero, then directly skip, do not need to use horizontal half-pix to draw based on depth image.
F) according to step 2) drawing order formulated, utilize horizontal half-pix method for drafting method based on depth image, the block information based on auxilliary viewpoint carries out the drafting of block information to virtual visual point image, realizes blocking compensation;
Arbitrary zone in the virtual visual point image general otherwise in main viewpoint as seen, in auxilliary viewpoint as seen, promptly as shown in Figure 3: the virtual view oblique line represents that part can be obtained by main drawing viewpoints, and white hollow sectors can be obtained by the block information drafting that auxilliary viewpoint provides basically;
If the position of main viewpoint, virtual view, auxilliary viewpoint is as shown in Figure 2, draws equation and be:
The solution of overlap problem should be to draw from right to left line by line; Like this,, can obtain the cavity seldom, even not have the virtual view composograph in cavity through after blocking compensation.
Described high-resolution virtual visual point image is carried out conversion, obtains the virtual visual point image step of original resolution:
To be resolution that the image of 2H*V is remapped to the image that resolution is H*V now, as shown in Figure 7;
G) determine a kind of transform method that changes low resolution from high-resolution into;
7. make that (x y) is the pixel coordinate of image; Function f (x, y), g (x, y) be respectively the H*V image coordinate for (x, y) and the 2H*V image coordinate be (x, y) pixel value of position; Function h (x, y) expression: when g (x, when y) being not equal to zero, h (x, y)=1; When g (x, when y) equalling zero, h (x, y)=0;
8. function f (x, y) with g (x, mapping relations y) are:
H) utilize step mapping relations 8., high-resolution virtual visual point image is carried out conversion, generate virtual visual point image with original resolution.
Three adjacent pixels of level are done linear interpolation in the high-resolution virtual visual point image that step f) is obtained, and promptly utilize formula (2) to obtain Fig. 7 (x, pixel value y).
The described empty compensation process that utilizes asymmetrical linear interpolation method to carry out remnants:
Generally be the partial occlusion problem through blocking residual cavity, compensation back on the one hand, promptly can obtain according to the adjacent pixels interpolation;
On the other hand, the reason that is formed by the cavity is the discontinuity of the object depth of field, and for the displacement of having only level between the parallel camera array model viewpoint, therefore in this specific camera array, the generation in cavity mainly is because the discontinuous of the horizontal direction degree of depth causes in the image rendering process; Moreover the discontinuous main existence of the degree of depth and the boundary of object, the discontinuous also main existence of pixel value and the boundary of object.
Can draw from this two aspect: in the time of partial occlusion, the pixel value in cavity is more close with some of surrounding pixel points, can recover more accurately according to certain several neighbor; But near the cavity, horizontal direction probably is the border of object, and the depth of field and brightness, chroma pixel value change greatly;
Therefore, draw a kind of at the vertical and asymmetrical linear interpolation method of horizontal direction, with the empty problem of compensate for residual.
I), adopt and a kind ofly carry out the cavity at vertical and the asymmetrical linear interpolation method of horizontal direction and compensate to the cavity of remnants;
9. detect cavity position, be made as (x, y); Function f (x, y) denotation coordination is (x, y) pixel value; Function h (x, y) expression: when f (x, when y) being not equal to zero, h (x, y)=1; When f (x, when y) equalling zero, h (x, y)=0; Threshold value t
1, t
2, t wherein
1＞t
2
10. empty (x, compensation equation y) is:
Utilize the compensation equation of step in 10. that virtual visual point image is carried out remaining cavity compensation, can obtain the better virtual visual point image of visual quality.
Claims (4)
1. virtual visual point synthesizing method based on the degree of depth and block information is characterized in that may further comprise the steps:
1) under parallel optical axis camera array model, determines 3D drafting equation based on the method for drafting of depth image;
2) according to the locus of reference view and virtual view, formulate the pixel drawing order;
3) according to the drawing order of formulating, utilize horizontal half-pix method for drafting based on depth image, with reference to main viewpoint, draw the image that obtains virtual view;
4) according to the block information of assisting viewpoint, utilization is based on the horizontal half-pix method for drafting of depth image, virtual visual point image is blocked compensation, obtain high-resolution virtual visual point image, the block information of auxilliary viewpoint comprises the pixel value and the corresponding depth value thereof in the cavity that obtains according to the auxilliary viewpoint of reference view prediction;
5) high-resolution virtual visual point image is carried out conversion, obtain the virtual visual point image of original resolution;
6) utilize asymmetrical linear interpolation method to carry out remaining cavity compensation;
7) it is level and smooth to carry out blocking effect, obtains high-quality virtual view composograph;
Described locus according to reference view and virtual view, formulate pixel drawing order step and comprise:
A) under parallel optical axis camera array model, when overlapping generation, judge their hiding relations in the position of reference view at virtual view according to pixel;
Virtual view is before drawn, and all pixel values all are zero; At reference view x
1The pixel of position is through drawing the x of back at virtual view
2The position is as if x this moment
2The pixel value of position is non-vanishing, then overlapping generation, and according to given parallel camera model, according to
F is a camera focus, and c is the horizontal displacement of virtual view with respect to reference view, and z is the depth of field, and regulation object a blocks object b should be satisfied:
B) block the relation that should satisfy according to the object that obtains in a), locus in conjunction with reference view and virtual view, formulate a kind of image drawing order that under given camera model, carries out to solve substantially when image is drawn overlap problem, if virtual view is on the reference view right side, then for the pixel in the reference view, draw according to from left to right reference sequence line by line; If virtual view is in reference view left side,, draw according to dextrosinistral reference sequence line by line then for the pixel in the reference view;
Described according to the drawing order of formulating, utilize horizontal half-pix method for drafting based on depth image, with reference to main viewpoint, draw the image step that obtains virtual view and comprise:
C) raising obtains a kind of horizontal half-pix method for drafting based on depth image based on the matching precision of the method for drafting of depth image:
1. the depth image that obtains according to the first step is drawn equation, calculate object the accurate coordinates of virtual view (x, y);
2. be the virtual visual point image coordinate transformation of H*V resolution the image coordinate of 2H*V resolution, the span of horizontal coordinate x expands to [0,2H] from [0, H];
3. (x y) carries out horizontal half-pix correction, changes (x into to coordinate
2, y), x wherein
2=2x is again to x
2Round nearby;
D) utilization according to the drawing order of formulating, with reference to main viewpoint, is drawn the image that obtains virtual view based on the horizontal half-pix method for drafting of depth image:
4. according to the drawing order of formulating, for arbitrary pixel (x of main viewpoint
1, y), utilize horizontal half-pix method for drafting based on depth image, calculate its coordinate (x in virtual view
2, y) overlapping as if not detecting, be (x then with coordinate in the virtual view
2, coordinate is (x in pixel value y) and the main viewpoint
1, pixel value unanimity y);
5. if detect overlappingly,, directly cover coordinate (x when virtual view during on main viewpoint right side
2, pixel value y) when virtual view is on the left of reference view, keeps coordinate (x
2, y) original pixel value;
6. repeating step 4. 5., all pixels traversal is finished in main visual point image, generates the image of high-resolution virtual view at last.
2. a kind of virtual visual point synthesizing method according to claim 1 based on the degree of depth and block information, the block information that it is characterized in that the auxilliary viewpoint of described basis, utilization is based on the horizontal half-pix method for drafting of depth image, virtual visual point image is blocked compensation, obtains high-resolution virtual visual point image step and comprise:
E) obtain the block information of auxilliary viewpoint, the locus of virtual view, auxilliary viewpoint;
According to parallel optical axis camera array model, virtual view and auxilliary viewpoint also have only horizontal displacement; The locus of block information and virtual view, auxilliary viewpoint can obtain at the auxiliary data partial decoding of h of decoding end from encoded data stream in coding side these information of encoding;
F) according to step 2) drawing order formulated, utilize horizontal half-pix method for drafting based on depth image, the block information based on auxilliary viewpoint carries out the drafting of block information to virtual visual point image, realizes blocking compensation.
3. a kind of virtual visual point synthesizing method based on the degree of depth and block information according to claim 1 is characterized in that described high-resolution virtual visual point image being carried out conversion, and the virtual visual point image step that obtains original resolution comprises:
G) determine a kind of transform method that changes low resolution from high-resolution into:
7. make that (x y) is the pixel coordinate of image; Function f (x, y), g (x, y) be respectively the H*V image coordinate for (x, y) and the 2H*V image coordinate be (x, y) pixel value of position; Function h (x, y) expression: when g (x, when y) being not equal to zero, h (x, y)=1; When g (x, when y) equalling zero, h (x, y)=0;
8. function f (x, y) with g (x, mapping relations y) are:
H) utilize step mapping relations 8., high-resolution virtual visual point image is carried out conversion, generate virtual visual point image with original resolution.
4. a kind of virtual visual point synthesizing method based on the degree of depth and block information according to claim 1 is characterized in that the described empty compensation process that utilizes asymmetrical linear interpolation method to carry out remnants comprises:
I), adopt and a kind ofly carry out the cavity at vertical and the asymmetrical linear interpolation method of horizontal direction and compensate to the cavity of remnants:
9. detect cavity position, be made as (x, y); Function f (x, y) denotation coordination is (x, y) pixel value; Function h (x, y) expression: when f (x, when y) being not equal to zero, h (x, y)=1; When f (x, when y) equalling zero, h (x, y)=0; Threshold value t
1, t
2, t wherein
1＞t
2
10. empty (x, compensation equation y) is:
Utilize the compensation equation of step in 10. that virtual visual point image is carried out remaining cavity compensation, can obtain the better virtual visual point image of visual quality.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN 200810062811 CN100576934C (en) | 2008-07-03 | 2008-07-03 | Virtual visual point synthesizing method based on the degree of depth and block information |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN 200810062811 CN100576934C (en) | 2008-07-03 | 2008-07-03 | Virtual visual point synthesizing method based on the degree of depth and block information |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN101312540A CN101312540A (en) | 2008-11-26 |
| CN100576934C true CN100576934C (en) | 2009-12-30 |
Family
ID=40100932
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN 200810062811 Active CN100576934C (en) | 2008-07-03 | 2008-07-03 | Virtual visual point synthesizing method based on the degree of depth and block information |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN100576934C (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102821293A (en) * | 2011-06-10 | 2012-12-12 | 三星电子株式会社 | Image processing method and apparatus |
Families Citing this family (47)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8866920B2 (en) | 2008-05-20 | 2014-10-21 | Pelican Imaging Corporation | Capturing and processing of images using monolithic camera array with heterogeneous imagers |
| US11792538B2 (en) | 2008-05-20 | 2023-10-17 | Adeia Imaging Llc | Capturing and processing of images including occlusions focused on an image sensor by a lens stack array |
| CN102074004B (en) * | 2010-01-07 | 2013-09-25 | 董福田 | Method and device for determining type of barrier of spatial entity |
| CN101808251B (en) * | 2010-04-06 | 2011-07-20 | 浙江大学 | Method for extracting blocking information in stereo image pair |
| CN101873508B (en) * | 2010-05-25 | 2012-07-04 | 东南大学 | Intermediate view synthesis method based on improved BP (Belief Propagation) algorithm |
| US8878950B2 (en) | 2010-12-14 | 2014-11-04 | Pelican Imaging Corporation | Systems and methods for synthesizing high resolution images using super-resolution processes |
| FR2964775A1 (en) | 2011-02-18 | 2012-03-16 | Thomson Licensing | METHOD FOR ESTIMATING OCCULTATION IN A VIRTUAL ENVIRONMENT |
| CN102075779B (en) * | 2011-02-21 | 2013-05-08 | 北京航空航天大学 | Intermediate view synthesizing method based on block matching disparity estimation |
| EP2679014B1 (en) * | 2011-02-23 | 2015-01-07 | Koninklijke Philips N.V. | Processing depth data of a three-dimensional scene |
| CN103404156A (en) * | 2011-02-24 | 2013-11-20 | 索尼公司 | Image processing apparatus and image processing method |
| CN102325259A (en) * | 2011-09-09 | 2012-01-18 | 青岛海信数字多媒体技术国家重点实验室有限公司 | Method and device for synthesizing virtual viewpoints in multi-viewpoint video |
| CN102447925B (en) * | 2011-09-09 | 2014-09-10 | 海信集团有限公司 | Method and device for synthesizing virtual viewpoint image |
| US8542933B2 (en) | 2011-09-28 | 2013-09-24 | Pelican Imaging Corporation | Systems and methods for decoding light field image files |
| TWI456526B (en) * | 2011-11-03 | 2014-10-11 | Au Optronics Corp | Ulti-view stereoscopic image generating method and multi-view stereoscopic image generating apparatus applying the same method |
| CN102523454B (en) * | 2012-01-02 | 2014-06-04 | 西安电子科技大学 | Method for utilizing 3D (three dimensional) dictionary to eliminate block effect in 3D display system |
| JP6016061B2 (en) * | 2012-04-20 | 2016-10-26 | Ｎｌｔテクノロジー株式会社 | Image generation apparatus, image display apparatus, image generation method, and image generation program |
| US9076249B2 (en) | 2012-05-31 | 2015-07-07 | Industrial Technology Research Institute | Hole filling method for multi-view disparity maps |
| TWI547904B (en) * | 2012-05-31 | 2016-09-01 | 財團法人工業技術研究院 | Hole filling method for multi-view disparity map |
| CN102957936B (en) * | 2012-07-25 | 2015-07-08 | 中山大学 | Virtual viewpoint generation method from video single viewpoint to multiple viewpoints |
| EP2888720B1 (en) | 2012-08-21 | 2021-03-17 | FotoNation Limited | System and method for depth estimation from images captured using array cameras |
| KR20150046154A (en) * | 2012-09-25 | 2015-04-29 | 니폰 덴신 덴와 가부시끼가이샤 | Image encoding method, image decoding method, image encoding device, image decoding device, image encoding program, image decoding program, and recording medium |
| US8866912B2 (en) | 2013-03-10 | 2014-10-21 | Pelican Imaging Corporation | System and methods for calibration of an array camera using a single captured image |
| JP6097150B2 (en) * | 2013-05-24 | 2017-03-15 | ソニーセミコンダクタソリューションズ株式会社 | Image processing apparatus, image processing method, and program |
| US10027947B2 (en) * | 2013-06-05 | 2018-07-17 | Sony Corporation | Image processing apparatus and image processing method |
| KR102156402B1 (en) * | 2013-11-05 | 2020-09-16 | 삼성전자주식회사 | Method and apparatus for image processing |
| CN103785174B (en) * | 2014-02-26 | 2016-08-17 | 北京智明星通科技有限公司 | A kind of same screen of playing shows the method and system of ten thousand people |
| CN104378618B (en) * | 2014-11-11 | 2016-06-29 | 清华大学深圳研究生院 | A kind of parallel View Synthesis load-balancing method based on depth map |
| CN105611373B (en) * | 2015-12-23 | 2019-03-08 | 小米科技有限责任公司 | Method of video image processing and device |
| CN106210696B (en) * | 2016-07-22 | 2018-07-06 | 北京大学深圳研究生院 | A kind of method and device of real-time virtual View Synthesis |
| KR102378860B1 (en) * | 2016-07-29 | 2022-03-28 | 소니그룹주식회사 | Image processing apparatus and image processing method |
| CN107509067B (en) * | 2016-12-28 | 2019-07-30 | 浙江工业大学 | A kind of free view-point image composition method of high-speed high-quality amount |
| US11074679B2 (en) | 2017-02-06 | 2021-07-27 | Huawei Technologies Co., Ltd. | Image correction and display method and device |
| JP6824579B2 (en) * | 2017-02-17 | 2021-02-03 | 株式会社ソニー・インタラクティブエンタテインメント | Image generator and image generation method |
| CN107437268A (en) * | 2017-07-31 | 2017-12-05 | 广东欧珀移动通信有限公司 | Photographic method, device, mobile terminal and computer-readable storage medium |
| EP3598389A1 (en) * | 2018-07-19 | 2020-01-22 | Thomson Licensing | Method for detecting occlusions in an image, corresponding device and computer program product |
| CN111383262B (en) * | 2018-12-28 | 2023-05-12 | 曜科智能科技（上海）有限公司 | Occlusion detection method, occlusion detection system, electronic terminal and storage medium |
| MX2022003020A (en) | 2019-09-17 | 2022-06-14 | Boston Polarimetrics Inc | Systems and methods for surface modeling using polarization cues. |
| EP4042101A4 (en) | 2019-10-07 | 2023-11-22 | Boston Polarimetrics, Inc. | Systems and methods for surface normals sensing with polarization |
| JP7329143B2 (en) | 2019-11-30 | 2023-08-17 | ボストン ポーラリメトリックス，インコーポレイティド | Systems and methods for segmentation of transparent objects using polarization cues |
| CN111225217B (en) * | 2019-12-16 | 2022-05-10 | 功智域控科技(苏州)有限公司 | 3D-HEVC error concealment method based on virtual viewpoint rendering |
| US11195303B2 (en) | 2020-01-29 | 2021-12-07 | Boston Polarimetrics, Inc. | Systems and methods for characterizing object pose detection and measurement systems |
| CN115428028A (en) | 2020-01-30 | 2022-12-02 | 因思创新有限责任公司 | System and method for synthesizing data for training statistical models in different imaging modalities including polarized images |
| US11953700B2 (en) | 2020-05-27 | 2024-04-09 | Intrinsic Innovation Llc | Multi-aperture polarization optical systems using beam splitters |
| US11290658B1 (en) | 2021-04-15 | 2022-03-29 | Boston Polarimetrics, Inc. | Systems and methods for camera exposure control |
| US11954886B2 (en) | 2021-04-15 | 2024-04-09 | Intrinsic Innovation Llc | Systems and methods for six-degree of freedom pose estimation of deformable objects |
| US11689813B2 (en) | 2021-07-01 | 2023-06-27 | Intrinsic Innovation Llc | Systems and methods for high dynamic range imaging using crossed polarizers |
| CN115202174B (en) * | 2022-07-14 | 2023-11-10 | 浙江理工大学 | Holographic view acquisition method, system and application based on light field image |
-
2008
- 2008-07-03 CN CN 200810062811 patent/CN100576934C/en active Active
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102821293A (en) * | 2011-06-10 | 2012-12-12 | 三星电子株式会社 | Image processing method and apparatus |
| CN102821293B (en) * | 2011-06-10 | 2016-11-02 | 三星电子株式会社 | Image processing method and equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN101312540A (en) | 2008-11-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN100576934C (en) | Virtual visual point synthesizing method based on the degree of depth and block information | |
| CN101902657B (en) | Method for generating virtual multi-viewpoint images based on depth image layering | |
| EP2150065B1 (en) | Method and system for video rendering, computer program product therefor | |
| CN102609974B (en) | Virtual viewpoint image generation process on basis of depth map segmentation and rendering | |
| CN103402097B (en) | A kind of free viewpoint video depth map encoding method and distortion prediction method thereof | |
| CN103828359A (en) | Representation and coding of multi-view images using tapestry encoding | |
| CN103581650B (en) | Binocular 3D video turns the method for many orders 3D video | |
| CN102438167B (en) | Three-dimensional video encoding method based on depth image rendering | |
| AU2017331736B2 (en) | Base anchored models and inference for the compression and upsampling of video and multiview imagery | |
| CN104754359A (en) | Depth map coding distortion forecasting method for two-dimensional free viewpoint video | |
| CN112019828B (en) | Method for converting 2D (two-dimensional) video into 3D video | |
| Jantet et al. | Object-based layered depth images for improved virtual view synthesis in rate-constrained context | |
| CN110113593A (en) | Wide baseline multi-view point video synthetic method based on convolutional neural networks | |
| CN103873867B (en) | Free viewpoint video depth map distortion prediction method and free viewpoint video depth map coding method | |
| CN104506871B (en) | A kind of 3D video fast encoding methods based on HEVC | |
| CN102790895A (en) | Multi-viewpoint video encoding and viewpoint synthesis predication method based on least square | |
| CN113853796A (en) | Methods, apparatuses and computer program products for volumetric video encoding and decoding | |
| CN112634127B (en) | Unsupervised stereo image redirection method | |
| CN112927348B (en) | High-resolution human body three-dimensional reconstruction method based on multi-viewpoint RGBD camera | |
| CN108259917A (en) | 3 D video decoding method and system based on depth time domain down-sampling | |
| CN103379350B (en) | Virtual viewpoint image post-processing method | |
| CN107770511A (en) | A kind of decoding method of multi-view point video, device and relevant device | |
| CN103813149B (en) | A kind of image of coding/decoding system and video reconstruction method | |
| CN103139554A (en) | Method and device for optimizing three-dimensional video frequency distortion | |
| CN104350748B (en) | Use the View synthesis of low resolution depth map |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| C41 | Transfer of patent application or patent right or utility model | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20160614 Address after: 518000 new energy building, Nanhai Road, Shenzhen, Guangdong, Nanshan District A838 Patentee after: Meng Qi media (Shenzhen) Co. Ltd. Address before: 310027 Hangzhou, Zhejiang Province, Zhejiang Road, No. 38 Patentee before: Zhejiang University |
|
| C41 | Transfer of patent application or patent right or utility model | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20160831 Address after: 518000, 101, 2, Fengyun technology building, Fifth Industrial Zone, North Ring Road, Shenzhen, Guangdong, Nanshan District Patentee after: World wide technology (Shenzhen) Limited Address before: 518000 new energy building, Nanhai Road, Shenzhen, Guangdong, Nanshan District A838 Patentee before: Meng Qi media (Shenzhen) Co. Ltd. |
|
| TR01 | Transfer of patent right |
Effective date of registration: 20180903 Address after: 518000 B unit 101, Fengyun mansion 5, Xili street, Nanshan District, Shenzhen, Guangdong. Patentee after: Wan D display technology (Shenzhen) Co., Ltd. Address before: 518000 2 of Fengyun tower, Fifth Industrial Zone, Nanshan District North Ring Road, Shenzhen, Guangdong, 101 Patentee before: World wide technology (Shenzhen) Limited |
|
| TR01 | Transfer of patent right |