Specific implementation mode
To make the above purposes, features and advantages of the invention more obvious and understandable, below in conjunction with the accompanying drawings to the present invention
Specific implementation mode be described in detail.
Detail is elaborated in the following description in order to fully understand the present invention.But the present invention can with it is a variety of not
Other manner described here is same as to implement, those skilled in the art can do class without violating the connotation of the present invention
Like popularization.Therefore the present invention is not limited by following public specific implementation mode.
Just as described in the background section, in the prior art, mobile phone can only play out 3D videos, and most of at present
Video source or image source are 2D, when using existing, the mobile phone with single camera makes 3D videos in addition, effect
It is bad and have certain limitation.
Inventor has found that extraction image depth information regards caused by depending on video camera translational motion itself
Difference, but if video camera itself has rotary motion, it will cause occur global translation or rotary motion in video scene
Vector field, however due to the converting system of the current 2D videos applied to DTV to 3D videos and 2D videos can not be removed
Global motion in image then easily leads to transformed 3D videos so if applying on mobile terminal (such as mobile phone)
Poor display effect, and when there is the region in the region, smooth featureless blocked in video image, obtained based on above system
The effect of depth image and bad.
In addition, when carrying out the making of 3D videos using the camera of mobile phone, since mobile phone camera bore generally compares
It is small, therefore, the pattern distortion obtained by mobile phone camera than more serious, and above system be also not directed to the image of distortion into
Row correction.
Therefore, it is flat can not to directly apply to mobile phone for the converting system applied to the 2D videos of DTV to 3D videos
Platform.Thus inventor proposes, is first pre-processed to 2D videos, to remove the distortion of the 2D video images caused by camera
With rotated due to camera caused by 2D video images global motion, then to by the screening in pretreated 2D video images
The motion vector of the pixel of gear region and smooth region (being difficult to detect the region of feature) is detected, and removes motion vector
Cavity in incredible pixel, then 2D video images to eliminating the incredible pixel of motion vector is filled,
To obtain the relatively good depth image of effect.
Technical solution for a better understanding of the present invention first explains the related terms occurred in the present invention:
Match point:Pixel in the (i-1)-th frame image corresponding with the pixel in the i-th frame image.
Match block:Image block in the (i-1)-th frame image corresponding with the image block in the i-th frame image.
Motion vector:The relative displacement of the corresponding match point of pixel in i-th frame image.
Positive matching value:The corresponding positive movement vector of the matched point of pixel in i-th frame image.
Negative relational matching value:The corresponding counter motion vector of the matched point of pixel in i-th frame image.
Fig. 3 is referred to, Fig. 3 is the flow diagram of the conversion method of the video image of the embodiment of the present invention;Such as Fig. 3 institutes
Show, the conversion method of the video image includes:
Step S11:Pretreatment is carried out to the current frame image of two-dimensional video and obtains present frame pretreatment image, the pre- place
Reason includes:Remove the global motion of current frame image.
Step S12:Extract the depth image of the present frame pretreatment image.
Step S13:The left-eye image of current frame image is obtained based on the present frame pretreatment image and its depth image
And eye image.
Step S11 is executed, in the present embodiment, the 2D videos can be that user is obtained by network or other modes
The 2D videos that 2D video code flows obtain after video decodes can also be the 2D videos that user is shot by the camera of mobile phone.
The 2D videos are pre-processed, specifically, exactly each frame image in 2D videos is pre-processed.The pretreatment
Including:The global motion of current frame image is removed, the global motion refers to the video motion caused by the rotation by camera,
The background of whole image and target are all moving at this time.
It should be noted that for the 2D videos obtained through the above way, if the 2D video images are gone
In addition to distortion, then the global motion for removing 2D video images is only needed both to be needed if the 2D video images do not remove distortion
The distortion for removing the 2D video images is also required to remove the global motion of the 2D video images, specifically, in removal institute
State the distortion that the current frame image is removed before the global motion of current frame image.
In the present embodiment, does not remove distortion with current frame image and illustrated accordingly.Remove the distortion of current frame image
Lens calibration method, anti-distortion model method etc. may be used, the global motion for removing current frame image is primarily referred to as removal due to taking the photograph
The global motion as caused by head rotation, may be used optical flow method etc., using which kind of method depending on actual demand.
Specifically, the 2D video images of distortion are carried out using barrel-shaped (barrel) lens distortion model in the present embodiment
Correction, refers to Fig. 4, Fig. 4 is barrel lens distortion models, wherein figure (a) indicates that the image not distorted, figure (b) indicate abnormal
The image of change, for pixel in image, corresponding distortion formula is:
Wherein:ruRefer to position of the pixel apart from center of distortion in non-fault image, rdRefer in fault image, as
Position of the vegetarian refreshments apart from center of distortion, k is barrel distortion coefficient, depending on the optical characteristics of camera lens.And the purpose of distortion correction
It is then:Determine that each pixel corresponds to the position in figure (b) in figure (a), i.e.,:
rd=F (k, ru) (2)
Wherein F is the inverse operation of formula (1), is calculated for convenience in practical calculating process, it will usually by above-mentioned formula (2)
It is converted, obtains following formula:
After corresponding position of the position of each pixel in figure (a) in figure (b) is determined by above-mentioned formula, it will scheme
(b) pixel value of the pixel value of corresponding position as the pixel in figure (a) corresponding with the position in, you can reconstruct is not abnormal
The image of change.
After the distortion for eliminating current frame image by the above method, the global motion of current frame image is then removed,
Specifically, it can carry out in the following way:
Global motion vector field and the global rotational motion vector field for first obtaining current frame image, are then based on the institute of acquisition
The global motion vector field and global rotational motion vector field for stating current frame image carry out global translation and complete to current frame image
Office's rotation.
Those skilled in the art know, are sweared to the global motion vector field and global rotary motion for obtaining current frame image
Field is measured, then needs first to obtain the global motion vector of each pixel in current frame image and global rotational motion vector, currently
The global motion vector of all pixels point constitutes global motion vector field, the global rotation fortune of all pixels point in frame image
Dynamic vector constitutes global rotational motion vector field.And the global motion vector field of the current frame image based on acquisition and
Global rotational motion vector field carries out global translation and global rotation, and the first global fortune based on pixel to current frame image
Dynamic vector and global rotational motion vector to carry out global translation to pixel and the overall situation rotates, then by all pixels
Point carries out global translation and global rotation and then realizes the global translation to current frame image and global rotation.Below to obtaining picture
The global motion vector of vegetarian refreshments and global rotational motion vector are described in detail.
In the present embodiment, global motion vector can be divided into horizontal global motion vector and vertical global vector, right
For the mobile phone with attitude transducer (gyroscope), the level of pixel is global in the current frame image of camera shooting
The angle, θ that motion vector can relative to horizontal direction be rotated with mobile phone camerahIt weighs, vertical global vector can be with
The angle, θ rotated relative to vertical direction with mobile phone cameravIt weighs, the angle, θh、θvIt detects to obtain by gyroscope.
The global rotational motion vector of pixel can be with mobile phone camera along the angle, θ of itself optical axis rotationoTo weigh.
If mobile phone does not have attitude transducer, can be obtained by the method for image procossing the global motion vector and
Global rotational motion vector obtains the global motion vector of pixel especially by optical flow method in the present embodiment and global rotates
Motion vector.For optical flow method, brightness shape constancy is primarily based on it is assumed that i.e.:
IxU (x, y)+IyV (x, y)+It=0
Wherein:IxIt is pixel brightness to the derivative of x, IyIt is pixel brightness to the derivative of y, ItIt is pixel brightness to t
Derivative, u (x, y) be the corresponding motion vector of pixel (x, y) the directions x component, v (x, y) be pixel (x, y) correspondence
Motion vector the directions y component.
It is then based on light stream smoothness assumption, i.e., the pixel of each neighborhood moves in an identical manner in image, then
The corresponding motion vector of pixel (x, y) the component u (x, y) in the directions x and the component v (x, y) in the directions y pass through following formula
It obtains as follows:
Wherein:Ix1It is first pixel brightness to the derivative of x, IxnIt is nth pixel point brightness to the derivative of x;Iy1For
First pixel brightness is to the derivative of y, IynIt is nth pixel point brightness to the derivative of y;It1For first pixel brightness
To the derivative of t, ItnIt is nth pixel point brightness to the derivative of t, the pixel sum of 1≤n≤current frame image.
In the present embodiment, all pixels point in current frame image can not be calculated to accelerate calculating speed,
But according to certain spacing, such as:A pixel is taken every 10 pixels in the horizontal direction and the vertical direction, then
Substitute into formula (3) come obtain the corresponding motion vector of pixel (x, y) the directions x component u (x, y) and pixel (x, y)
Component v (x, y) of the corresponding motion vector in the directions y.
Component u (x, y) of the corresponding motion vector of pixel in the directions x and point in the directions y are obtained through the above way
After measuring v (x, y), average to all u (x, y), and using the average value as horizontal global motion vector θh, to all
V (x, y) averages, and using the average value as vertical global vector thetav。
Still it uses optical flow method to obtain the motion vector of pixel, i.e., the fortune of pixel is still obtained using formula (3)
Dynamic vector, at this time n=1.The motion vector of the pixel of acquisition is subtracted into above-mentioned horizontal global motion vector and obtains uoIt is (each
Pixel has corresponded to a uo), the motion vector of the pixel of acquisition is subtracted into above-mentioned vertical global vector and obtains vo
(each pixel has corresponded to a vo), and then pass through uoAnd voTo estimate the global rotational motion vector θ of pixelo。
Further, in order to the global rotational motion vector θ of quick obtaining pixelo, using simple searching method,
Specifically, be exactly at 0 ° arrive α degree range, every rotational motion vector of β degree interval calculation the component in the directions x and
The component in the directions y, i.e.,:
ui=ru(1-cosθi)
vi=rusinθi (4)
Wherein:uiIt is the corresponding rotational motion vector of pixel (x, y) in the component in the directions x, viIt is right for pixel (x, y)
The rotational motion vector answered is in the component in the directions y, ruFor the rotational motion vector of pixel,θi=β degree, 2 β
Degree, 3 β degree ..., α degree.Calculate separately u1With uoMean square error, u2With uoMean square error, u3With uoMean square error ...,With uoMean square error, and will be with uoMean square error minimum uminIt takes out;Calculate separately v1With voMean square error, v2With
voMean square error, v3With voMean square error ...,With voMean square error, and will be with voMean square error minimum vmin
It takes out;With umin(the corresponding angle of the radian) and vmin(the corresponding angle of the radian) is θiRange, still use above-mentioned public affairs
Formula (4) calculates a rotational motion vector in the component in the directions x and component in the directions y every γ degree, and obtains at this time
umin(the corresponding angle of the radian) and vmin(the corresponding angle of the radian), as the u finally obtainedmin(the corresponding angle of the radian
Degree) and vminWhen the difference of (the corresponding angle of the radian) is less than 0.1 °, by umin(the corresponding angle of the radian) or vmin(the radian
Corresponding angle) or umin(the corresponding angle of the radian) and vminThe average value of (the corresponding angle of the radian) is rotated as global
Motion vector θo.If u at this timemin(the corresponding angle of the radian) and vminThe difference of (the corresponding angle of the radian) is more than 0.1 °, then
Continue to calculate a rotational motion vector in the component in the directions x and component in the directions y every σ degree, until this acquisition
umin(the corresponding angle of the radian) and vminThe difference of (the corresponding angle of the radian) is less than 0.1 °.
With α=90 °, β=10 ° are described in detail the above method for γ=1 °.Primary rotation is first calculated every 10 °
Dynamic vector is transported in the component in the directions x and component in the directions y, that is, calculates separately u1With uoMean square error, u2With uoIt is equal
Square error, u3With uoMean square error ..., u9With uoMean square error, and will be with uoMean square error minimum uminIt takes out;Point
V is not calculated1With voMean square error, v2With voMean square error, v3With voMean square error ..., v9With voMean square error, and
It will be with voMean square error minimum vminIt takes out;By uminCorresponding angle, θuminAnd vminCorresponding angle, θvminAs formula
(4) θ iniRange, this sentences θvmin> θumin, θumin=1 °, θvminFor=20 °, due to γ=1 °, therefore every 1 degree of calculating
Rotational motion vector is in the component in the directions x and component in the directions y, then θ at this timei=1 °, 2 °, 3 ° ..., 19 °,
20 °, calculate separately u at this time1With uoMean square error, u2With uoMean square error, u3With uoMean square error ..., u20With uo's
Mean square error, and will be with uoMean square error minimum uminIt takes out;Calculate separately v at this time1With voMean square error, v2With vo's
Mean square error, v3With voMean square error ..., v20With voMean square error, and will be with voMean square error minimum vminIt takes
Go out;The u that will be obtained at this timeminCorresponding angle, θuminAnd vminCorresponding angle, θvminAs θiRange, if at this time obtaining
θuminAnd θvminDifference be less than 0.1 °, then by θuminOr θvminOrAs global rotational motion vector θo.It is no
Then, continue to repeat the above process every certain number of degrees, until the θ finally obtaineduminAnd θvminDifference be less than 0.1 °.
It should be noted that depending on the value of α, β, γ, σ ... are by concrete condition in the above process, generally for can be fast
Speed obtains the global rotational motion vector θ of pixelo, α generally takes 90 °, and the number of degrees of β, γ, σ ... are then between 1 °~10 °.
So far, the horizontal global motion vector θ of pixel is obtained by above-mentioned optical flow methodh, vertical global arrow
Measure θvWith global rotational motion vector θo。
The horizontal global motion vector θ of pixel based on acquisitionhWith vertical global vector thetavAnd pixel
Global rotational motion vector θoGlobal translation and global rotation are carried out to the pixel, picture is removed in particular by following formula
The global motion of vegetarian refreshments,
xd=xu+kxθh+rr(1-coSθo)
yd=yu+kyθv+rrsinθo
Wherein:(xu, yu) refer to the coordinate, (x for removing the pixel after global motiond, yd) refer to not removing global motion
Pixel coordinate, rrRefer to pixel at a distance from rotation center, kxRefer to that mobile phone often turns over 1 degree, image is in the horizontal direction
Translate how many pixel, kyRefer to that mobile phone often turns over 1 degree, image is in vertical direction translation how many pixel (kxAnd kyValue and camera shooting
Head physical characteristic is related, can be measured by the method for experiment).
Pixel (the x of global motion will not be removedd, yd) pixel value as removal global motion pixel (xu, yu)
Pixel value, you can with obtain removal global motion pixel.The overall situation is removed to all pixels point in current frame image
Movement, then eliminate the global motion of current frame image, and after removing global motion to current frame image, extraction present frame is located in advance
Manage the depth image of image.
The depth image (step S12) of present frame pretreatment image is extracted in the present embodiment in order to better understand, first letter
Singlely the principle that depth image is extracted from video image is introduced.Fig. 5 is referred to, Fig. 5 is the original of depth image extraction
Reason figure, as shown in figure 5, point O indicates that any point in image scene, video camera use the national forest park in Xiaokeng simplified, depending on
The optical center position of the corresponding video camera of two field pictures of frequency is respectively A, B, and f indicates that the focal length of video camera, Z indicate point O away from camera shooting
The distance of machine, then the picture point of point O different pixel a and b have been corresponded in two field pictures.If pixel a is current frame image
The picture point of midpoint O, pixel b are the picture point of previous frame image midpoint O, then pixel b is the match point of pixel a, namely works as
The match point of pixel a is pixel b in prior image frame.For pixel a and b, X1It indicates in pixel a and image
The distance of the heart, X2Indicate that at a distance from picture centre, if Z is far longer than f, (X can be obtained by geometrical relationship by pixel b1-X2)
Value be proportional to the 1/Z reciprocal of distance Zs of the point O away from video camera.Therefore, (X can be passed through1-X2) value be multiplied by a proportionality coefficient
To represent the depth information of O points.
Therefore the problem of extracting the depth image of current frame image be converted into extraction current frame image in each pixel with
In reference frame image (previous frame image) the problem of the motion vector of Corresponding matching point.And it can also be known from video camera by above-mentioned
The absolute value of the corresponding motion vector of pixel of close object is big, the corresponding movement arrow of pixel of the object remote from video camera
The absolute value of amount is small, if the information of the motion vector acquired is more accurate, then obtained depth information is also more accurate, Jin Ergen
The quality of the 3 D video gone out according to deep image rendering is also better.
Fig. 6 is referred to, Fig. 6 is the flow signal of the depth image of the extraction present frame pretreatment image of the embodiment of the present invention
Figure (corresponds to the step S12 in Fig. 3), as shown in fig. 6, the depth image of extraction present frame pretreatment image includes:
S121:Remove the incredible pixel of motion vector in the present frame pretreatment image.
S122:Obtain the color area segmentation information of the present frame pretreatment image.
S123:At least based on the color area segmentation information to eliminating the current of the incredible pixel of motion vector
Cavity in frame pretreatment image is filled, and obtains the depth image of present frame pretreatment image.
Step S121 is executed, the incredible pixel of motion vector in the present frame pretreatment image is removed in the present embodiment
It puts and includes:Including:
Present frame pretreatment image is matched with previous frame image, obtain present frame pretreatment image pixel with
The corresponding positive matching value of its match point and negative relational matching value;
The current frame image is divided into image block with predetermined size;
It removes the error of positive matching value and negative relational matching value described in the present frame pretreatment image and is more than first
When the pixel of limit value and the smoothness of described image block are more than the second threshold value, described image pixel in the block.
Present frame pretreatment image with previous frame image match in the present embodiment and obtains present frame pretreatment image
The corresponding positive matching value of the matched point of pixel and negative relational matching value block matching method, optical flow method and HRM may be used
Which kind of method is method etc. specifically use, determined by the computational complexity of this method.HRM methods are used in the present embodiment, for
For HRM methods, although its computational complexity is relatively high, the precision using the motion vector of HRM methods acquisition is higher, therefore
The depth information of acquisition is also more accurate, and the effect of the 3 D video rendered is also better.
Specifically, the first step:Estimation is carried out to the image block in present frame pretreatment image.
If present frame pretreatment image is first frame image (current frame image is first frame image) in the present embodiment, adopt
Estimation (the ruler of usual image block is carried out to the image block of the present frame pretreatment image with block matching motion estimation method
It is very little between 4*4~8*8 pixels), and using the estimation as initial motion vectors.Block matching motion estimation method is existing
Method for estimating, therefore not reinflated specific detailed description herein.
If the present frame pretreatment image is not first frame image (current frame image is not first frame image), will work as
(match block refers to and present frame pretreatment figure the corresponding motion vector of matched piece of image block of previous frame pretreatment image
The image block in the corresponding previous frame image of image block as in), the left image block of the image block of present frame pretreatment image
Matched piece of the upper images block of matched piece of corresponding motion vector, the image block of present frame pretreatment image is corresponding
Motion vector is as candidate motion vector.In the present embodiment, the size of described image block is preferably 4*4 pixels, described current
Left image block, the upper images block of the image block of frame pretreatment image are it is to be understood that with by a secondary 160*160 pixels
For image divides image block according to 4*4 pixel sizes, then 40*40 image block is shared, there are one sit for each image block correspondence
Mark, if wherein the coordinate of some image block is (2,10), then the coordinate of the image block on its corresponding left side is (2,9), top figure
As the coordinate of block is (1,10).
The matching error of three above-mentioned candidate motion vectors is calculated separately, the Candidate Motion arrow of matching error minimum is selected
Amount is used as initial motion vectors.The matching error is obtained by following formula:
Wherein:D is matching error, and M is the number of image block pixel in the horizontal direction, and N is image block in Vertical Square
To pixel number, Fc(x, y) is that coordinate is the pixel value of the pixel of (x, y), F in present frame pretreatment imager(x+
dx, y+dy) it is that coordinate is (x+d in previous frame imagex, y+dy) pixel pixel value, dxFor present frame pretreatment image
The corresponding motion vector of the matched point of pixel is in the component in the directions x, dyFor the pixel of present frame pretreatment image and its
Component of the corresponding motion vector of match point in the directions y.Depending on the value of M and N is by actual demand in the present embodiment.
Furthermore, it is necessary to explanation, for present frame pretreatment image is not first frame image, if present frame is located in advance
The image block of reason image is located at the boundary of present frame pretreatment image, then still needs to obtain by block matching motion estimation method
Initial motion vectors.
Second step:It is (each to the corresponding motion vector of each pixel inside the image block of present frame pretreatment image
The corresponding motion vector of pixel refers to the corresponding movement arrow of each matched point of pixel in present frame pretreatment image
Amount, the match point of the pixel of present frame pretreatment image refers to corresponding with the pixel in present frame pretreatment image previous
Pixel in frame image) initial motion vectors are assigned a value of, motion vector refinement value is obtained using pixel precision matching process.Tool
Body, it is obtained by following formula:
D (x, y)=di-|fc(x, y)-fr(x+dx, y+dy)|[ux, uy]T
Wherein:
D (x, y) is the motion vector refinement value of the pixel (x, y) of present frame pretreatment image, diIt is sweared for initial motion
Amount, fc(x, y) is that coordinate is the pixel value of the pixel of (x, y), f in present frame pretreatment imager(x+dx, y+dy) it is previous
Coordinate is (x+d in frame imagex, y+dy) pixel pixel value, Θ is Grads threshold, related to the smoothness of image.
Third walks:Obtain the corresponding motion vector of each pixel inside the image block of present frame pretreatment image, tool
Body, initial motion vectors motion vector refinement value corresponding with each pixel is added, you can to obtain each pixel
Corresponding motion vector.
Choose the matching error of the corresponding motion vector of each pixel inside the image block of present frame pretreatment image
In matching error corresponding with three candidate motion vectors obtained in the first step, the motion vector corresponding to minimum match error
The final motion vector of image block as present frame pretreatment image.Specifically, each pixel corresponding motion vector
Matching error is still calculated (M=0 at this time, N=0) using formula (5).
The final motion vector of the image block of the present frame pretreatment image of above-mentioned acquisition is present frame pretreatment image
The corresponding positive movement vector of the matched point of image pixel in the block namely present frame pretreatment image image block in
The corresponding positive matching value of the matched point of pixel, each image block in present frame pretreatment image is all made of
The method stated obtains the final motion vector of described image block, and then can obtain each pixel in present frame pretreatment image
The corresponding positive matching value of the matched point of point.And it is corresponding anti-for the matched point of the pixel of present frame pretreatment image
Acquisition to motion vector is similar with the acquisition of positive movement vector, unlike by the first step in above-mentioned HRM methods to the
Present frame pretreatment image in three steps replaces with previous frame image, and previous frame image is replaced with present frame pretreatment image
.
So far, the matched point of pixel that present frame pretreatment image is obtained by above-mentioned HRM methods is corresponding
Positive matching value and negative relational matching value.
Detect whether the error between above-mentioned positive matching value and negative relational matching value is more than the first threshold value, if more than then
The corresponding motion vector of the matched point of the pixel is insincere, it is on the contrary then to be credible.First thresholding by actual test and
It is fixed, can be the length of 3~10 pixels, the value of the first threshold value described in the present embodiment is the length of 5 pixels.
Also judged in the image block by calculating the smoothness of image block in present frame pretreatment image in the present embodiment
Including the corresponding motion vector of the matched point of pixel it is credible or insincere.Specifically, the present frame is located in advance
Reason image is divided into image block with predetermined size, and the predetermined size makes a reservation for depending on actual demand described in the present embodiment
Size is between 4*4~8*8 pixels, it is therefore preferable to 4*4 pixels.To each figure in the present frame pretreatment image after segmentation
As the smoothness of block is detected, whether the smoothness for detecting described image block is more than the second threshold value.
The smoothness of image block can be by obtaining the pixel of all pixels point that the image block is included in the present embodiment
The mean square deviation of value is weighed, i.e., the smoothness of the image block be it includes all pixels point pixel value mean square deviation, if institute
The smoothness for stating image block is more than second threshold value, the then corresponding movement of the matched point of pixel that the image block includes
Vector is insincere, it is on the contrary then to be credible, can be 10~100 depending on second thresholding is by actual test, institute in the present embodiment
The value for stating the second threshold value is 40.
By above-mentioned judgement, the corresponding movement arrow of the matched point of pixel in present frame pretreatment image is obtained
Amount is incredible pixel, then needs the incredible pixel of motion vector being removed, that is, it is pre- to remove the present frame
Handle pixel and described image block that the error of positive matching value and negative relational matching value in image is more than the first threshold value
When smoothness is more than the second threshold value, described image pixel in the block.
After the incredible pixel removal of motion vector in present frame pretreatment image, lead to present frame pretreatment figure
There are many cavities as in, therefore, it is necessary to be filled up to it.Using the color to present frame pretreatment image in the present embodiment
Color region is split and (image-region is divided into different color lumps according to the difference of color), obtains the present frame and locates in advance
The color area segmentation information for managing image, on the one hand retains present frame pretreatment figure using the segmentation information of the color area
On the other hand object edge as in also utilizes the color area being located in the color area segmentation information got around cavity
Depth information determine the depth information in cavity that needs fill, and then the cavity in present frame pretreatment image is filled out
It fills, to obtain the depth image of present frame pretreatment image.
It should be noted that if the present frame pretreatment image is after eliminating the incredible pixel of motion vector,
There is the cavity of large area in the present frame pretreatment image, then in addition to the color using the present frame pretreatment image obtained
Outside region segmentation information, it is also necessary to utilize the depth information of the depth image of at least previous frame image of the current frame image of caching
Come determine the cavity pixel depth information, and then to eliminating the present frame after the incredible pixel of motion vector
Cavity in pretreatment image is filled to obtain the depth image of present frame pretreatment image.Pass through caching in the present embodiment
The depth information of depth image of preceding 5 frame the large area cavity in present frame pretreatment image is filled, and depth
The depth information of image is obtained by following formula:
Wherein:D (x, y) is the depth information of pixel (x, y), and u (x, y) is the corresponding motion vector of pixel (x, y)
Component in the directions x, v (x, y) are component of the corresponding motion vector of pixel (x, y) in the directions y.It is pre-processed in conjunction with present frame
The depth information of the depth image of at least previous frame image of the color area segmentation information of image and the current frame image of caching
Cavity in present frame pretreatment image is filled as the prior art, therefore not reinflated specific detailed description herein.
Further, since between the image of different frame, the distance of cam movement may change, therefore, overall next
Say the ratio of the motion vector length of pixel in every frame image be also can it is changed (the i.e. same object camera fortune
Its depth does not change in dynamic process, but is caused corresponding with the object due to the difference of cam movement distance
The length of the motion vector of pixel changes), therefore, it needs through normalized method come the corresponding depth map of each frame of unification
The ratio of picture is normalized the corresponding depth image of each frame image.
And the depth information of the depth image of the picture frame using caching fills out present frame pretreatment image
For the region filled, due to the ratio of the ratio and the depth information of present frame pretreatment image of the depth information of the picture frame of caching
Example is different, therefore, is normalized with greater need for the depth image to present frame pretreatment image.Specifically, by following
The depth image of present frame pretreatment image is normalized in formula:
Wherein:DrFor the depth information of the pixel of normalized present frame pretreatment image, D is present frame pretreatment figure
The depth information of the pixel of picture, DminFor the minimum value of the pixel depth information of present frame pretreatment image, DmaxIt is current
The maximum value of the pixel depth information of frame pretreatment image.
After the depth image to the present frame pretreatment image is normalized, in the present embodiment also to normalization after
The depth image of present frame pretreatment image be filtered, primarily to pre-processing figure to the present frame after normalization
The depth image of picture carries out denoising.Smothing filtering, medium filtering and bilateral (Bilateral) filtering may be used in the present embodiment
In a kind of depth image to the present frame pretreatment image after the normalization be filtered.
After the depth image for obtaining the present frame pretreatment image with exact depth information through the above steps, hold
Row step S13 obtains the left-eye image and right eye of current frame image based on the present frame pretreatment image and its depth image
Image.Specifically,
Using the present frame pretreatment image as eye image or left-eye image;
The depth image of the present frame pretreatment image is mapped, to obtain mapping image;
Empty filling is carried out to the mapping image, obtains corresponding another eye pattern picture.
In the present embodiment, since the depth image to the present frame pretreatment image is also normalized and has been filtered,
Therefore the depth image of normalization and filtered present frame pretreatment image is mapped first, according to the one of Physiologic Studies
A little conclusions, the information that human brain more depends on right eye to obtain in synthetic stereo image are (next for the people for getting used to the right hand
Say), therefore the better viewing effect for helping to improve stereogram of eye image quality, therefore preferably will be described in the present embodiment
The depth image of present frame pretreatment image is as eye image, by the normalization and filtered present frame pretreatment image
Depth image be mapped as left-eye image, specifically, by following formula realize to it is described normalization and filtered present frame
The depth image of pretreatment image is mapped:
xl=xc+k/Z
Wherein, xcFor the abscissa of pixel in the depth image of present frame pretreatment image;xlFor by mapping acquisition
The abscissa of pixel in left-eye image;K is proportionality coefficient, represents the corresponding pixel distance of unit depth difference, size is by hand
The difference of the physical characteristic for the camera that machine uses and it is different, determined in particular by the method for actual measurement, Z be video camera and object
The distance between point.
It should be noted that in the present embodiment, since the depth image to present frame pretreatment image has also carried out normalizing
Change and filters, therefore above-mentioned xcIt should be the cross of pixel in the depth image of normalization and filtered present frame pretreatment image
Coordinate.
However, the mapping image (left-eye image) obtained by above-mentioned mapping in still will appear some cavity and again
Folded, wherein hollow sectors image information caused by due to blocking lacks to be formed, therefore there is still a need for occurring in mapping image
Cavity is filled, and in the present embodiment, carrying out cavity filling to the mapping image includes:The mapping image is carried out linear
Filtering is with the pixel value in filling cavity region, and the part of the pixel overlapping for occurring in mapping image, then is believed with depth
The pixel value of small pixel is ceased as final pixel value.
In addition, in order to reduce the generation in cavity, it can be to normalizing the depth with filtered present frame pretreatment image
Degree image is pre-processed before being mapped, and specifically, is located in advance using to normalization and filtered present frame in the present embodiment
The depth image of reason image is mapped again after first carrying out low-pass filtering, normalization and filtered present frame pretreatment image
Depth image is after low-pass filtering, the edge-smoothing of the depth image of acquisition, and then can reduce the generation in cavity, improves most
The quality of the left-eye image rendered eventually.
Using the present frame pretreatment image as eye image in the present embodiment, and normalization will be worked as with filtered
The depth image of previous frame pretreatment image is mapped again after being pre-processed, the image after the filling of cavity is as left-eye image;
And in other embodiments can also using the present frame pretreatment image as left-eye image, and will be to normalization and filtering after
Present frame pretreatment image depth image pre-processed after mapped again, cavity filling after image as right eye figure
Picture.
After the left-eye image and eye image that obtain current frame image by the conversion method of above-mentioned video image, it will obtain
Left-eye image and eye image input the 3D video display systems obtained can show relief 3D videos.
In addition, by the conversion method of above-mentioned video image, to the left-eye image of each frame image of the 2D videos of acquisition
Video coding is carried out with eye image, then can obtain 3 D video code stream.To a left side for the current frame image of the 2D videos of acquisition
Eye pattern picture and eye image carry out compression of images, then can obtain static single frames 3D rendering.
Corresponding to the conversion method of above-mentioned video image, the embodiment of the present invention also provides a kind of converting means of video image
It sets, refers to Fig. 7, Fig. 7 is the structural schematic diagram of the conversion equipment of the video image of the embodiment of the present invention, as shown in fig. 7, described
The conversion equipment of video image includes:
Pretreatment unit A10 carries out pretreatment suitable for the current frame image to two-dimensional video and obtains present frame pretreatment figure
Picture, the pretreatment include:Remove the global motion of current frame image.
Depth image extraction unit A11 is connected with the pretreatment unit A10, is suitable for extracting the present frame pretreatment
The depth image of image.
Deep image rendering unit A12 is connected with the depth image extraction unit A11, is suitable for being based on the present frame
Pretreatment image and its depth image obtain the left-eye image and eye image of current frame image.
The pretreatment unit A10 includes:
Vector field acquiring unit (not shown) is suitable for obtaining the global motion vector field of current frame image and global rotation
Turn motion vector field.
Global motion removal unit (not shown), be suitable for global motion vector field based on the current frame image and
Global rotational motion vector field carries out global translation and global rotation to current frame image.
In the present embodiment, the pretreatment unit A10 further includes:
Distort removal unit (not shown), is suitable for before the global motion for removing the current frame image described in removal
The distortion of current frame image.The distortion removal unit is suitable for the distortion using barrel distortion model removal current frame image.
The depth image extraction unit A11 includes:
Removal unit 110 is connected with the pretreatment unit A10, is transported suitable for removing in the present frame pretreatment image
The incredible pixel of dynamic vector.
Segmentation information acquiring unit 111 is connected with the pretreatment unit A10, is suitable for obtaining the present frame pretreatment
The color area segmentation information of image.
First empty fills unit 112, is connected with the removal unit 110 and segmentation information acquiring unit 111, fits respectively
In at least based on the color area segmentation information to eliminating the present frame pretreatment figure of the incredible pixel of motion vector
Cavity as in is filled, and obtains the depth image of present frame pretreatment image.
Wherein, the removal unit 110 includes:
First buffer unit (not shown) is suitable for storing the previous frame image of current frame image.
Matching unit (not shown) is obtained suitable for being matched with previous frame image to present frame pretreatment image
The corresponding positive matching value of the matched point of pixel and negative relational matching value of present frame pretreatment image.
Image segmentation unit (not shown), suitable for the present frame pretreatment image is divided into figure with predetermined size
As block.
First detection unit (not shown) is adapted to detect for the positive matching of the pixel in present frame pretreatment image
Whether the error of value and negative relational matching value is more than the first threshold value.
Second detection unit (not shown) is adapted to detect for the smoothness of the image block of described image cutting unit segmentation
Whether the second threshold value is more than.
First removal unit (not shown) is suitable for removing positive matching value described in the present frame pretreatment image
The smoothness of the pixel and described image block that are more than the first threshold value with the error of negative relational matching value is more than the second threshold value
When, described image pixel in the block.
In the present embodiment, the matching unit with previous frame image match to present frame pretreatment image uses block
One kind in method of completing the square, optical flow method and HRM methods.
In the present embodiment, the depth image extraction unit A11 further includes:
Second buffer unit (not shown) is suitable for caching the depth map of at least previous frame image of current frame image
Picture.
Described first empty fills unit 112, is further adapted in conjunction with the color area segmentation information and the second buffer unit
The depth information of the depth image of at least previous frame image of the current frame image of caching determines the depth of the pixel in the cavity
Information is spent, to be filled to the cavity.
The deep image rendering unit A12 includes:
Output unit 120 is connected with the pretreatment unit A10, is suitable for using the present frame pretreatment image as the right side
Eye pattern picture or left-eye image output.
Map unit 121 is connected with the described first empty fills unit 112, is suitable for the present frame pretreatment image
Depth image mapped, to obtain mapping image.
Second empty fills unit 122, is connected with the map unit 121, is suitable for carrying out cavity to the mapping image
Filling obtains corresponding another eye pattern picture.
In the present embodiment, the map unit 121 is suitable for carrying out the depth image of the present frame pretreatment image low
It is mapped again after pass filter.
The second empty fills unit 122 includes:
Second filter unit (not shown) is suitable for carrying out linear filtering to the mapping image.
In the present embodiment, the conversion equipment of the video image further includes:Video encoding unit (not shown), is suitable for
Left-eye image and eye image to each frame image of the two-dimensional video carry out Video coding, to obtain 3 D video code stream.
Image compression unit (not shown) is suitable for carrying out image to the left-eye image and eye image of the present frame
Compression, to obtain 3-D view.
Fig. 8 is referred to, Fig. 8 is the structural schematic diagram of the conversion equipment of the video image of another embodiment of the present invention, Fig. 8
In, the empty fills unit 112 of pretreatment unit B10, removal unit 110, segmentation information acquiring unit 111, first, output unit
120, the second empty fills unit 122 is similar in Fig. 7, the difference is that in the present embodiment, depth image extraction unit
B11 further includes other than including removal unit 110, segmentation information acquiring unit 111 and the first empty fills unit 112:
Normalization unit 113 is connected with the described first empty fills unit 112, is suitable for pre-processing the present frame and scheme
The depth image of picture is normalized.
First filter unit 114, is connected with the normalization unit 113, is suitable for pre-processing the present frame after normalization
The depth image of image is filtered.
Deep image rendering unit B 12 is worked as suitable for being based on the present frame pretreatment image and normalization with filtered
The depth image of previous frame pretreatment image obtains the left-eye image and eye image of current frame image.Therefore in Fig. 8, map unit
121 are connected with first filter unit 114, are suitable for the depth by normalization and filtered present frame pretreatment image
Image is mapped, to obtain mapping image.
First filter unit 114 is by one kind in smothing filtering, medium filtering and Bilateral filtering to described
The depth image of present frame pretreatment image after normalization is filtered.
The process that two-dimensional video is converted to 3 D video by the conversion equipment of the video image of the embodiment of the present invention can join
The conversion method for being admitted to the video image stated carries out, and details are not described herein.
In conclusion technical scheme of the present invention at least has the advantages that:
Pretreatment is first carried out by the current frame image to the two-dimensional video and obtains present frame pretreatment image, is then obtained
Take the depth image of the present frame pretreatment image, the depth image that is finally based on the present frame pretreatment image maps out
Left-eye image or eye image, and using the present frame pretreatment image as another eye pattern picture, it will on mobile phone realizing
While two-dimensional video is converted to 3 D video, but also user is making 3D videos by the mobile phone with single camera
In the process can be along any direction cell phone, and multi-pass operation need not be carried out and be obtained with the preferable 3D videos of effect,
It provide the user great convenience.
The distortion of the current frame image of two-dimensional video is eliminated by pretreated mode and global motion obtains currently
The pretreatment image of frame image is removed the incredible pixel of motion vector in present frame pretreatment image, and is based on
The color area segmentation information of the present frame pretreatment image is to eliminating the present frame of the incredible pixel of motion vector
Cavity in pretreatment image is filled, and then obtains depth image, relative to what is directly obtained by current frame image
Its accuracy is high for depth image, therefore also improves the quality of the 3D videos finally obtained.
Further, after the incredible pixel of motion vector in eliminating present frame pretreatment image, using slow
The depth information of the depth image of at least previous frame image for the current frame image deposited and the color area of the current frame image
Segmentation information is filled the large area cavity in present frame pretreatment image, further increases the present frame of acquisition
The accuracy of the depth image of pretreatment image.
It is normalized and is filtered by the depth image to present frame pretreatment image, further improve
The accuracy of the depth image of present frame pretreatment image, and then improve the quality of 3D videos.
Carrying out mapping to the depth image of the present frame pretreatment image includes:To the present frame pretreatment image
Depth image is mapped again after carrying out low-pass filtering so that the edge of the depth image of the present frame pretreatment image is flat
It is sliding, reduce the generation in cavity when mapping the depth image of the present frame pretreatment image, improves final acquisition
The left eye of current frame image or the quality of eye image, and then also improve the quality of 3D videos.
Although the invention has been described by way of example and in terms of the preferred embodiments, but it is not for limiting the present invention, any this field
Technical staff without departing from the spirit and scope of the present invention, may be by the methods and technical content of the disclosure above to this hair
Bright technical solution makes possible variation and modification, therefore, every content without departing from technical solution of the present invention, and according to the present invention
Technical spirit to any simple modifications, equivalents, and modifications made by above example, belong to technical solution of the present invention
Protection domain.