CN101198061A - Solid video stream encoding method based on sight point image mapping - Google Patents

Abstract

The invention relates to a stereo video stream coding method on the basis of viewpoint image mapping and belongs to the video coding technical field, wherein, a mapping image is obtained at first through mapping image pixel information of a first viewpoint to a second viewpoint, and then stereo video stream coding of the mapping image and a second viewpoint image is performed. The invention overcomes the defects of low coding efficiency of the stereo video coding method on the basis of parallax estimation and compensation, and large residual values and lowered compressibility after parallax estimation of a forecasting algorithm is combined, etc.The invention has the advantages of capability of making the best of correspondence which is formed between binocular viewpoint image data through mutual mapping of a three-dimensional space and improvement of coding performances of stereo video streams.

Description

Three-dimensional video-frequency stream encoding method based on the visual point image mapping

Technical field

The invention belongs to technical field of video coding, relate to the coding method of stereo video streaming, relate in particular to a kind of based on the visual point image mapping method for encoding stereo video of (also claiming the three-dimensional video-frequency coupling).

Background technology

The notion of parallax compensation is proposed since Lukacs, use parallax information set up three-dimensional video-frequency between the correlation of similar area, method for encoding stereo video based on disparity estimation and compensation grows up thereupon, and becomes a kind of classical technology of stereo scopic video coding.Be that the method for encoding stereo video that unit carries out disparity estimation and compensation is used in many viewpoints class of Moving Picture Experts Group-2 with the piece.In this method, encoder is at first used algorithm for single-pass video encoding compression algorithm left side passage video.For right passage video, the frame of each frame left side passage video sequence correspondence is predicted based on the optical parallax field of estimating, and optical parallax field and prediction residual image is encoded.Disparity estimation and compensation process all adopt block-based method; That is to say that for each image block in the right view, they seek a piece the most similar in left view.One preferably method be which provides less predicated error according to, between the motion compensated prediction between the different frame of the parallax compensation between the different views of identical time prediction and identical view, switch.The temporal scalability sexual norm of its available Moving Picture Experts Group-2 is realized, with the temporal scalability of MPEG-2, sequence is divided into the collection of two sub-sampling frames.Basic layer (such as, form by even frame) encode with the one-way movement compensation prediction, wherein reference frame is the even frame from preceding face code.The enhancement layer of being made up of remaining strange frame is encoded with the bi directional motion compensation prediction, and reference frame is from the strange frame of preceding face code, and another is from close even frame in the basic layer.Left view is encoded as basic layer, and right view is encoded as enhancement layer.For left view, image is encoded with I, B or P pattern.For B and P image, be used for predictive reference frames and only come from left view.For right view, image is encoded with P or B pattern, and in the P pattern, image carries out predictive coding with frame corresponding in the left view as the reference frame.In the B frame pattern, reference frame is from a frame of front in the right view, and another reference frame is a frame corresponding in the left view.The used motion vector of latter event is actually difference vector, and forecasting process is the parallax compensation prediction.The advantage of this method is to use the method and the technology of conventional motion compensation prediction in many algorithm for single-pass video encoding, but the efficient of its coding is not high.

In conjunction with the associated prediction algorithm of Forecasting Methodologies such as motion compensation, further improved performance based on the method for encoding stereo video of disparity estimation and compensation.Disparity estimation is similar with estimation, the both is the correlation of seeking between two width of cloth images, different is, estimation is to go up two adjacent width of cloth images at the time to carry out, because the correlation that the time goes up between the two adjacent width of cloth images is very strong, in estimation, have only a spot of pixel to be moved usually, so the residual values after the motion compensation is all less, even near null value.Yet in disparity estimation, two width of cloth correspondence image of left and right sides video channel are synchronous in time, and they are taken by two different at one time visual fields of camera and obtain, so the residual values after the disparity estimation is all bigger, thereby compression performance is descended.

Summary of the invention

The object of the present invention is to provide a kind ofly to have overcome based on the efficient of the method for encoding stereo video of disparity estimation and compensation coding lowly, and the residual values after the disparity estimation of associated prediction algorithm is big, the degradation drawback based on the visual point image mapping three-dimensional video-frequency stream encoding method of (also claiming the three-dimensional video-frequency coupling) under the compression performance.

The present invention is on the basis of conventional three-dimensional video-frequency stream encoding method, before encoding, the binocular vision dot image of stereoscopic video stream increases a mapping process, utilize the characteristics of binocular tri-dimensional video stream, the first visual point image Pixel Information is mapped to second viewpoint, obtain map image, then the map image and second visual point image are encoded with conventional three-dimensional video-frequency volume stream code method.

Three-dimensional video-frequency stream encoding method based on the visual point image mapping of the present invention is made up of the following step:

1. the stereo video streaming first visual point image Pixel Information is mapped to second viewpoint, obtains map image;

2. the map image and second visual point image are carried out the three-dimensional video-frequency stream encryption.

Above-described the stereo video streaming first visual point image Pixel Information is mapped to second viewpoint, obtains map image and realize by the following step:

A. establish p ₁(u ₁, v ₁) flow the image coordinate of the point in first visual point image, p for binocular tri-dimensional video ₂(u ₂, v ₂) be p ₁Be mapped to the image coordinate of the point in second visual point image, P (X _w, Y _w, Z _w) be the two corresponding spatial point (same point in the world coordinate system), according to the pin-hole imaging model, the some P in the world coordinate system and its be the subpoint p in the binocular vision dot image respectively ₁And p ₂The relational expression of representing with homogeneous coordinates is:

$s_1\left[\begin{array}{c}{u}_1\\ v_1\\ 1\end{array}\right]=M_1\left[\begin{array}{c}{X}_w\\ Y_w\\ Z_w\\ 1\end{array}\right]---\left(1\right)$

With

$s_2\left[\begin{array}{c}{u}_2\\ v_2\\ 1\end{array}\right]=M_2\left[\begin{array}{c}{X}_w\\ Y_w\\ Z_w\\ 1\end{array}\right]---\left(2\right)$

In formula (1) and (2), s ₁And s ₂Be respectively two image coordinate of stereo video streaming and the camera coordinate system scale factor when transforming mutually, M ₁And M ₂Be respectively the projection matrix of two views, projection matrix can be determined by the inner parameter and the external parameter of video camera.

B. with 3 * 4 projection matrix M ₁And M ₂3 * 3 parts on the left side are remembered respectively and are made M in the matrix ₁₁And M ₂₁, 3 * 1 parts on the right are remembered respectively and are made m ₁And m ₂If will $\left[\begin{array}{c}{X}_w\\ Y_w\\ Z_w\\ 1\end{array}\right]$ Note is done ${\left({\left[\begin{array}{c}{X}_w\\ Y_w\\ Z_w\end{array}\right]}^{T}1\right)}^T$ Then formula (1) and (2) can be written as respectively:

$s_1\left[\begin{array}{c}{u}_1\\ v_1\\ 1\end{array}\right]=M_{11}\left[\begin{array}{c}{X}_w\\ Y_w\\ Z_w\end{array}\right]+m_1---\left(3\right)$

With

$s_2\left[\begin{array}{c}{u}_2\\ v_2\\ 1\end{array}\right]=M_{21}\left[\begin{array}{c}{X}_w\\ Y_w\\ Z_w\end{array}\right]+m_2---\left(4\right)$

With in formula (3) and (4) $\left[\begin{array}{c}{X}_w\\ Y_w\\ Z_w\end{array}\right]$ Cancellation gets:

$s_2\left[\begin{array}{c}{u}_2\\ v_2\\ 1\end{array}\right]-s_1{M}_{21}{M}_{11}^{-1}\left[\begin{array}{c}{u}_1\\ v_1\\ 1\end{array}\right]=m_2-M_{21}{M}_{11}^{-1}{m}_1---\left(5\right)$

That is:

$\left[\begin{array}{c}{u}_2\\ v_2\\ 1\end{array}\right]=\frac{1}{s_2}(m_2-M_{21}{M}_{11}^{-1}{m}_1)+\frac{s_1}{s_2}{M}_{21}{M}_{11}^{-1}\left[\begin{array}{c}{u}_1\\ v_1\\ 1\end{array}\right]---\left(6\right)$

C. according to the characteristics of binocular tri-dimensional video stream, establish s ₁=s ₂=s, thus, formula (6) can be reduced to:

$\left[\begin{array}{c}{u}_2\\ v_2\\ 1\end{array}\right]=\frac{1}{s}(m_2-M_{21}{M}_{11}^{-1}{m}_1)+M_{21}{M}_{11}^{-1}\left[\begin{array}{c}{u}_1\\ v_1\\ 1\end{array}\right]---\left(7\right)$

In the formula (7), both sides are three-dimensional vectors, have promptly comprised three equatioies, behind the cancellation s wherein, just can obtain u ₂And v ₂

The present invention since the characteristics of at first utilizing three-dimensional video-frequency with the image mapped of stereo video streaming first viewpoint to second viewpoint, form map image, then its image with second viewpoint is carried out the three-dimensional video-frequency stream encryption together, thereby can utilize the corresponding relation that shines upon formation between the binocular vision dot image data by three dimensions mutually more fully, improve the coding efficiency of stereo video streaming.

Embodiment

Core content of the present invention is binocular tri-dimensional video to be flowed first visual point image be mapped to second viewpoint, obtains map image, and the image with second viewpoint carries out the three-dimensional video-frequency stream encryption together then.

Concrete grammar is: at first binocular tri-dimensional video is flowed the first visual point image Pixel Information and be mapped to second viewpoint, obtain map image, then this map image image of second viewpoint is together carried out the three-dimensional video-frequency stream encryption together.

The described stereo video streaming first visual point image Pixel Information is mapped to second viewpoint, and the concrete grammar that obtains map image step (unit) is:

If p ₁(u ₁, v ₁) flow the image coordinate of the point in first view, p for binocular tri-dimensional video ₂(u ₂, v ₂) be p ₁Be mapped to the image coordinate of the point in second view, P (X _w, Y _w, Z _w) be the two corresponding spatial point (same point in the world coordinate system), according to the pin-hole imaging model, and the characteristics of utilizing binocular tri-dimensional video to flow, can determine the subpoint p of some P in the binocular vision dot image in the world coordinate system roughly ₁And p ₂Relation, promptly be mapped to second viewpoint by the first visual point image Pixel Information, obtain map image.

Concrete implementation step is:

1) establishes p ₁(u ₁, v ₁) flow the image coordinate of the point in first view, p for binocular tri-dimensional video ₂(u ₂, v ₂) be p ₁Be mapped to the image coordinate of the point in second view, P (X _w, Y _w, Z _w) be the two corresponding spatial point (same point in the world coordinate system), according to the pin-hole imaging model, the some P in the world coordinate system and its be the subpoint p in the binocular vision dot image respectively ₁And p ₂The pass of representing with homogeneous coordinates is:

$s_1\left[\begin{array}{c}{u}_1\\ v_1\\ 1\end{array}\right]=M_1\left[\begin{array}{c}{X}_w\\ Y_w\\ Z_w\\ 1\end{array}\right]---\left(1\right)$

With

$s_2\left[\begin{array}{c}{u}_2\\ v_2\\ 1\end{array}\right]=M_2\left[\begin{array}{c}{X}_w\\ Y_w\\ Z_w\\ 1\end{array}\right]---\left(2\right)$

Wherein, s ₁And s ₂Be respectively two image coordinate of stereo video streaming and the camera coordinate system scale factor when transforming mutually, M ₁And M ₂Be respectively the projection matrix of two visual point images.Projection matrix can be determined by the inner parameter and the external parameter of video camera.

2) with 3 * 4 projection matrix M ₁And M ₂3 * 3 parts on the left side are remembered respectively and are made M in the matrix ₁₁And M ₂₁, 3 * 1 parts on the right are remembered respectively and are made m ₁And m ₂If will $\left[\begin{array}{c}{X}_w\\ Y_w\\ Z_w\\ 1\end{array}\right]$ Note is done ${\left({\left[\begin{array}{c}{X}_w\\ Y_w\\ Z_w\end{array}\right]}^{T}1\right)}^T$ Then formula (1) and (2) can be written as respectively:

$s_1\left[\begin{array}{c}{u}_1\\ v_1\\ 1\end{array}\right]=M_{11}\left[\begin{array}{c}{X}_w\\ Y_w\\ Z_w\end{array}\right]+m_1---\left(3\right)$

With

$s_2\left[\begin{array}{c}{u}_2\\ v_2\\ 1\end{array}\right]=M_{21}\left[\begin{array}{c}{X}_w\\ Y_w\\ Z_w\end{array}\right]+m_2---\left(4\right)$

With in formula (3) and (4) $\left[\begin{array}{c}{X}_w\\ Y_w\\ Z_w\end{array}\right]$ Cancellation gets:

$s_2\left[\begin{array}{c}{u}_2\\ v_2\\ 1\end{array}\right]-s_1{M}_{21}{M}_{11}^{-1}\left[\begin{array}{c}{u}_1\\ v_1\\ 1\end{array}\right]=m_2-M_{21}{M}_{11}^{-1}{m}_1---\left(5\right)$

That is:

$\left[\begin{array}{c}{u}_2\\ v_2\\ 1\end{array}\right]=\frac{1}{s_2}(m_2-M_{21}{M}_{11}^{-1}{m}_1)+\frac{s_1}{s_2}{M}_{21}{M}_{11}^{-1}\left[\begin{array}{c}{u}_1\\ v_1\\ 1\end{array}\right]---\left(6\right)$

3) according to the characteristics of binocular tri-dimensional video stream, can establish s ₁=s ₂=s, thus, formula (6) can be reduced to:

$\left[\begin{array}{c}{u}_2\\ v_2\\ 1\end{array}\right]=\frac{1}{s}(m_2-M_{21}{M}_{11}^{-1}{m}_1)+M_{21}{M}_{11}^{-1}\left[\begin{array}{c}{u}_1\\ v_1\\ 1\end{array}\right]---\left(7\right)$

In the formula (7), both sides are three-dimensional vectors, have promptly comprised three equatioies, behind the cancellation s wherein, just can obtain u ₂And v ₂

Claims (2)

1. three-dimensional video-frequency stream encoding method based on visual point image mapping is characterized in that it is made up of the following step:

A. the stereo video streaming first visual point image Pixel Information is mapped to second viewpoint, obtains map image;

B. the map image and second visual point image are carried out the three-dimensional video-frequency stream encryption.

2. three-dimensional video-frequency stream encoding method according to claim 1 is characterized in that described the stereo video streaming first visual point image Pixel Information being mapped to second viewpoint, obtains map image and is realized by the following step:

A. establish p ₁(u ₁, v ₁) flow the image coordinate of the point in first visual point image, p for binocular tri-dimensional video ₂(u ₂, v ₂) be p ₁Be mapped to the image coordinate of the point in second visual point image, P (X _w, Y _w, Z _w) be the two corresponding spatial point (same point in the world coordinate system), according to the pin-hole imaging model, the some P in the world coordinate system and its be the subpoint p in the binocular vision dot image respectively ₁And p ₂The relational expression of representing with homogeneous coordinates is:

$s_1\left[\begin{array}{c}{u}_1\\ v_1\\ 1\end{array}\right]=M_1\left[\begin{array}{c}{X}_w\\ Y_w\\ Z_w\\ 1\end{array}\right]---\left(1\right)$

With

$s_2\left[\begin{array}{c}{u}_2\\ v_2\\ 1\end{array}\right]=M_2\left[\begin{array}{c}{X}_w\\ Y_w\\ Z_w\\ 1\end{array}\right]---\left(2\right)$

In formula (1) and (2), s ₁And s ₂Be respectively two image coordinate of stereo video streaming and the camera coordinate system scale factor when transforming mutually, M ₁And M ₂Be respectively the projection matrix of two views, projection matrix can be determined by the inner parameter and the external parameter of video camera.

B. with 3 * 4 projection matrix M ₁And M ₂3 * 3 parts on the left side are remembered respectively and are made M in the matrix ₁₁And M ₂₁, 3 * 1 parts on the right are remembered respectively and are made m ₁And m ₂If will $\left[\begin{array}{c}{X}_w\\ Y_w\\ Z_w\\ 1\end{array}\right]$ Note is done ${\left({\left[\begin{array}{c}{X}_w\\ Y_w\\ Z_w\end{array}\right]}^{T}1\right)}^T$ Then formula (1) and (2) can be written as respectively:

$s_1\left[\begin{array}{c}{u}_1\\ v_1\\ 1\end{array}\right]=M_{11}\left[\begin{array}{c}{X}_w\\ Y_w\\ Z_w\end{array}\right]+m_1---\left(3\right)$

$s_2\left[\begin{array}{c}{u}_2\\ v_2\\ 1\end{array}\right]=M_{21}\left[\begin{array}{c}{X}_w\\ Y_w\\ Z_w\end{array}\right]+m_2---\left(4\right)$

With in formula (3) and (4) $\left[\begin{array}{c}{X}_w\\ Y_w\\ Z_w\end{array}\right]$ Cancellation gets:

$s_2\left[\begin{array}{c}{u}_2\\ v_2\\ 1\end{array}\right]-s_1{M}_{21}{M}_{11}^{-1}\left[\begin{array}{c}{u}_1\\ v_1\\ 1\end{array}\right]=m_2-M_{21}{M}_{11}^{-1}{m}_1---\left(5\right)$

That is:

$\left[\begin{array}{c}{u}_2\\ v_2\\ 1\end{array}\right]=\frac{1}{s_2}(m_2-M_{21}{M}_{11}^{-1}{m}_1)+\frac{s_1}{s_2}{M}_{21}{M}_{11}^{-1}\left[\begin{array}{c}{u}_1\\ v_1\\ 1\end{array}\right]---\left(6\right)$

C. according to the characteristics of binocular tri-dimensional video stream, establish s ₁=s ₂=s, thus, formula (6) can be reduced to:

$\left[\begin{array}{c}{u}_2\\ v_2\\ 1\end{array}\right]=\frac{1}{s}(m_2-M_{21}{M}_{11}^{-1}{m}_1)+M_{21}{M}_{11}^{-1}\left[\begin{array}{c}{u}_1\\ v_1\\ 1\end{array}\right]---\left(7\right)$

In the formula (7), both sides are three-dimensional vectors, have promptly comprised three equatioies, behind the cancellation s wherein, just can obtain u ₂And v ₂