CN102333230A - Method for improving quality of synthetized virtual views in three-dimensional video system - Google Patents

Abstract

The invention discloses a method for improving the quality of synthetized virtual views in a three-dimensional video system. In the method, uncompressed video and deeply compressed video and deeply synthetized virtual views are respectively adopted at server sides of the three-dimensional video system, and a wiener filter coefficient is obtained through computation and is sent to a user terminal of the three-dimensional video system; at the user terminal of the three-dimensional video system, wiener filtering is carried out on the synthetized virtual views by the wiener filter coefficient; and further the quality of the synthetized virtual views is improved. According to the invention, the influence of compression distortion of video information and deep information in the three-dimensional video system on the quality of the synthetized virtual views is considered, so the quality of the synthetized virtual views is improved, the frame of the three-dimensional video system is perfected, and the method can be applied to the application of three-position videos.

Description

Synthetic virtual view method for quality in a kind of raising 3 D video system

Technical field

The present invention relates to synthetic virtual view method for quality in a kind of raising 3 D video system, belong to the virtual view synthesis technical field in the 3 D stereo video coding standard.

Background technology

The 3 D stereo video is meant that as main video applications in future the user can enjoy real 3 D stereo video content through 3 D stereo video display device.The correlation technique of 3 D video, such as, the technology such as demonstration of the collection of 3 D stereo video, 3 D stereo video coding, 3 D stereo video are paid close attention to widely.In order to promote the standardization of 3 D stereo video technology, 2002, (Motion Picture Experts Group MPEG) proposed any viewpoint TV (Free View Television, notion FTV) in Motion Picture Experts Group.It can provide vividly real, interactively 3 D stereo audiovisual system.The user can watch the 3 D stereo video of this angle from different angles, makes the user have and incorporates the sense of reality in the video scene.FTV can be widely used in fields such as broadcast communication, amusement, education, medical treatment and video monitoring.In order to make the user can watch 3 D stereo video at any angle, FTV system service end uses the video camera array of having demarcated to obtain the video on certain viewpoint.And, utilize corrected video information to generate the virtual view of virtual view through the virtual view synthetic technology to the video correction on the different points of view.MPEG suggestion is at present specifically used based on the degree of depth-image (Depth-Image Based Rendering, virtual view synthetic technology DIBR).Depth information is generally represented through depth map.The main process that virtual view synthesizes is following:

1). confirm to want the relative position of virtual view in video camera array.

2). confirm to be used for the texture video of synthetic virtual view.

3). confirm step 2) the corresponding depth map of texture video.

4). according to step 2) with 3) and in texture video and depth map, adopt the DIBR technology, synthetic virtual view.

The standardization effort of FTV is divided into two stages to carry out.Phase I is 2006 to 2008 the expansion scheme-MVC H.264/AVC (Multi-View Video Coding) that formulated by JVT (Joint Video Team, joint video code sets).MVC can encode to many viewpoints texture video.But to finally realize the function of FTV system, also must encode depth information.The standardization formulation work of FTV has at present got into second stage, i.e. 3DVC (Three Dimensional Video Coding).3DVC mainly pays close attention to the expression and the coding of depth information, and the combined coding of texture video and depth information.Among the 3DVC, depth information is represented through depth map.

The leading indicator of weighing the 3DVC performance is the quality of synthetic virtual view, and the encoder bit rate of texture video, depth map.The quality of virtual view:

Usually (Peak Signal-to-Noise Ratio PSNR) weighs the quality of video to adopt Y-PSNR.The computing formula of PSNR is shown below,

$\mathrm{PSNR}=10\×\log \left(\frac{{255}^2}{\mathrm{MSE}}\right)$

MSE representes the mean square error between original view and the synthetic virtual view in the following formula, is used for weighing the distortion of virtual view, and the coding distortion of the coding distortion of texture video, depth map.

In practical application, the view of virtual view is non-existent, does not also promptly have original view.But,, at first adopt the existing texture video of un-encoded and the corresponding synthetic virtual view V of depth map thereof therefore for weighing the performance of 3DVC because 3DVC mainly pays close attention to coding efficiency _Orig, the depth map that adopts the texture video of the reconstruction after process is encoded and the back of encoding to rebuild then synthesizes virtual view V _Rec, at last through calculating V _RecWith V _OrigBetween MSE, and then obtain PSNR, to weigh the performance of 3DVC.

As shown in Figure 1, in the 3D video system, the virtual view that the user watches is that video, the depth information through decoding and rebuilding is synthetic through the DIBR technology.As shown in Figure 2, the pixel m in the left side camera can project on the object point M in the 3D world coordinate system through following formula,

(X，Y，Z) ^T＝RgA ^-1(x，y，1) ^TgZ+t，

In the following formula, (X .Y, Z) ^TBe the coordinate of M, Z is the depth value of object point M, (x, y, 1) ^TBe the homogeneous coordinates of pixel m, use usually

Expression, A and R are the inside and outside parameter matrixs of video camera, t=(t _x, t _y, t _z) ^TBe the coordinate of camera lens center in the 3D world coordinate system.Object point M in the 3D world coordinate system can project on the pixel m ' in the camera plane of right side through following formula,

sg(x′，y′，1) ^T＝A′gR′ ^-1g[(X，Y，Z) ^T-t′]，

Wherein, (x ', y ', 1) ^TBe the homogeneous coordinates of m ', use usually

Expression, s is a scalar factor, A ', R ', t ' they are respectively the coordinate of optical center in the 3D world coordinate system of inner parameter matrix, external parameter matrix and the right side video camera of right side video camera.3D camera acquisition system for calibration accuracy; A ' is identical with A; R ' is identical with R; Only there is the difference on the horizontal coordinate between t ' and the t; Thus, the relation between and

can be represented with following formula

That is,

s·(x′，y′，1) ^T＝Z·(x，y，1) ^T+A·R ^-1·(t _x-t _x′，t _y-t _y′，t _z-t _z′) ^T.

Thus, the pixel m in the left side camera just can be corresponding with the right side video camera in pixel m ', the basic principle of DIBR technology that Here it is.

In addition, for fear of block the cavitation that phenomenon produces in virtual view, the synthetic virtual view of the video content of two video cameras was as shown in Figure 3 about existing virtual view synthetic technology often adopted.Among Fig. 3, ω _LAnd ω _RBe two weight coefficients, satisfy ω _R+ ω _L=1.

Existing virtual view synthetic technology has been considered and has been blocked linearity, improved the quality of synthetic virtual view.But the 3D video system by shown in Figure 1 can find out that the coding distortion of video, the degree of depth also has very big influence to the quality of synthetic virtual view.Existing virtual view composition algorithm is not considered the influence of the coding distortion of video, the degree of depth to synthetic virtual view quality.

Summary of the invention

Consider not that to what existing virtual view synthetic method existed coding distortion is to synthesizing the problem of virtual view quality influence; The present invention is according to the influence to synthetic virtual view quality of the compression artefacts of video, the degree of depth, proposes synthetic virtual view method for quality in a kind of raising 3 D video system.

Synthetic virtual view method for quality is in the 3 D video system in the raising 3 D video of the present invention system, adopts Weiner filter that synthetic virtual view is carried out filtering, improves the quality of synthetic virtual view, specifically may further comprise the steps:

(1) adopt depth map (Depth) and texture video (Texture) to synthesize virtual view V without overcompression _Orig

(2) at the coding side of 3 D video system depth map and texture video are encoded, and obtain the depth map Depth_rec of local decode reconstruction and the texture video Texture_rec of reconstruction;

(3) according to Depth_rec and the synthetic virtual view V of Texture_rec _Rec

(4) with V _OrigAs primary signal, V _RecAs the signal behind the primary signal interpolation noise, and calculate V _RecThe auto-correlation function matrix and the V of signal _OrigAnd V _RecBetween the cross-correlation function matrix, obtain wiener filter coefficients through finding the solution Wei Na-Hough Equation for Calculating;

(5) coefficient with Weiner filter is sent to the terminal of 3 D video system, the terminal that depth map after will encoding simultaneously and texture video send to the 3 D video system through encoding;

(6), at first depth map and texture video are decoded, and adopt based on the synthetic virtual view V of the virtual view synthetic technology of the degree of depth-image at the terminal of 3 D video system _{Rec_terminal}, and wiener filter coefficients decoded;

(7) utilize wiener filter coefficients that decoding obtains to V _{Rec_terminal}Carry out filtering, obtain final virtual view.

The present invention has considered the influence of the compression artefacts of video, the degree of depth to synthetic virtual view quality, adopts Weiner filter to improve the quality of the synthetic virtual view of 3 D video system terminal.

Description of drawings

Fig. 1 is the system block diagram of 3 D video;

Fig. 2 is a DIBR technology sketch map;

Fig. 3 is existing virtual view synthetic technology sketch map;

Fig. 4 adopts 3 D video system block diagram of the present invention.

Fig. 5 is the experimental result sketch map to 3 D video sequence B ookarrival.

Fig. 6 is the experimental result sketch map to 3 D video sequence dancer.

Fig. 7 is the experimental result sketch map to 3 D video sequence Lovebird1.

Embodiment

As shown in Figure 4, synthetic virtual view method for quality specifically comprises the steps: in the raising 3 D video of the present invention system

(1) adopt depth map (Depth) and texture video (Texture) to synthesize virtual view (V without overcompression _Orig);

(2) at the coding side of 3 D video system depth map and texture video are encoded, and obtain the depth map Depth_rec of local decode reconstruction and the texture video Texture_rec of reconstruction;

(3) according to Depth_rec and Texture_rec at synthetic virtual view V _Rec

(4) with V _OrigAs primary signal, V _RecAs the signal behind the primary signal interpolation noise, and calculate V _RecThe auto-correlation function matrix and the V of signal _OrigAnd V _RecBetween the cross-correlation function matrix, obtain wiener filter coefficients through finding the solution following Wei Na-Hough Equation for Calculating,

Wherein,

Be V _RecThe auto-correlation function matrix,

Be V _OrigAnd V _RecBetween the cross-correlation function matrix, C is the wiener filter coefficients matrix;

(5) coefficient with Weiner filter is sent to the terminal of 3 D video system, the terminal that depth map after will encoding simultaneously and texture video send to the 3 D video system through encoding;

(6), at first depth map and texture video are decoded, and adopt based on the synthetic virtual view V of the virtual view synthetic technology of the degree of depth-image at the terminal of 3 D video system _{Rec_terminal}, and wiener filter coefficients decoded;

(7) utilize wiener filter coefficients that decoding obtains to V _{Rec_terminal}Carry out filtering, obtain final virtual view.

Effect of the present invention can further specify through experiment.

Experiment test under the condition of different quantization parameters coding, the quality of the synthetic virtual view of a plurality of 3 D video sequences.Fig. 5 has compared in the 3 D video system, adopts the present invention and does not adopt rate distortion curve of the present invention.Wherein Fig. 5 is the experimental result to 3 D video sequence B ookarrival, and Fig. 6 is the experimental result to 3 D video sequence dancer, and Fig. 7 is the experimental result to 3 D video sequence Lovebird1.Visible by Fig. 5, Fig. 6 and Fig. 7, behind employing the present invention, under the identical condition of 3 D video system coding code check, the objective quality of synthetic virtual view is higher, explains that the present invention has improved the code efficiency of depth map.As far as 3 D video sequence B ookarrival, the objective quality of synthetic virtual view on average increases 0.531dB; As far as 3 D video sequence dancer, the objective quality of synthetic virtual view on average increases 0.217dB; As far as 3 D video sequence Lovebird1, the objective quality of synthetic virtual view on average increases 0.473dB.

Claims (1)

1. one kind is improved synthetic virtual view method for quality in the 3 D video system, it is characterized in that:

In the 3 D video system, adopt Weiner filter that synthetic virtual view is carried out filtering, improve the quality of synthetic virtual view, specifically may further comprise the steps:

(1) adopt depth map and texture video to synthesize virtual view V without overcompression _Orig

(2) at the coding side of 3 D video system depth map and texture video are encoded, and obtain the depth map Depth_rec of local decode reconstruction and the texture video Texture_rec of reconstruction;

(3) according to Depth_rec and the synthetic virtual view V of Texture_rec _Rec

(4) with V _OrigAs primary signal, V _RecAs the signal behind the primary signal interpolation noise, and calculate V _RecThe auto-correlation function matrix and the V of signal _OrigAnd V _RecBetween the cross-correlation function matrix, obtain wiener filter coefficients through finding the solution Wei Na-Hough Equation for Calculating;

(5) coefficient with Weiner filter is sent to the terminal of 3 D video system, the terminal that depth map after will encoding simultaneously and texture video send to the 3 D video system through encoding;

(6), at first depth map and texture video are decoded, and adopt based on the synthetic virtual view V of the virtual view synthetic technology of the degree of depth-image at the terminal of 3 D video system _{Rec_terminal}, and wiener filter coefficients decoded;

(7) utilize wiener filter coefficients that decoding obtains to V _{Rec_terminal}Carry out filtering, obtain final virtual view.

Images