CN1269359C - Video error blanketing method based on motion vector extrapolation and motion vector search - Google Patents

Abstract

The present invention discloses a video error masking method based on motion vector extrapolation and motion vector search. When a transmission error occurs, correct motion vector information stored at the time of video decoding is extrapolated according to the motion tendency, the motion vector of a damaged image block (8*8 pixels) is estimated by the overlapping area of an extrapolated macro blocks and a damaged area, and image error masking is carried out; when correct information exists on the periphery of the damaged area, the correct information can be combined with frame matching and optimization of motion vector search, and an optimum motion vector estimated value is obtained so that more accurate masking can be carried out. The method can effectively mask and restore video image information which is loss and damaged, and even when the entire frame of images is lost or damaged, the image information can still be restored with satisfaction. The method of the present invention can remarkably improve the reliability of video transmission and improve the quality of the video image, and the present invention is particularly suitable for PSTN or wireless channels with poor transmission performance.

Description

Spread to the video error coverage method of motion vector sought outward based on motion vector

Affiliated technical field

The invention belongs to digital video communications field, relate in particular to the error concealment and the recovery technology of the digital video transmission under low rates such as Public Switched Telephone Network, wireless mobile communications and the adverse circumstances.

Background technology

Current, video telephone research causing extensive concern both domestic and external under Public Switched Telephone Network and the mobile communication environment, yet, because at present digital video compress technique that phone adopts and standard all are as the core technology of compressing with motion compensated predictive coding (Motion Compensation Prediction) and variable-length encoding (VariableLength Coding), when in the environment that very easily takes place to disturb, transmitting with the data flow after this class technique compresses, the meeting that influences that information dropout produces is promptly spread on time and space, thereby cause video quality obviously to descend, even produce the visual effect of extreme difference.At present existing many error-control techniques are used for resisting the video transmission interference effect, and wherein, error concealment (Error concealment) technology is a kind of method of improving video effect that extensively adopts in decoder.

From available research achievements, the error concealment technology can be divided into the space and cover with time domain and cover two kinds.

When (Intra) encodes in the video image affected area is frame, available spatial domain (spatial) information or frequency domain (frequency) information around it is carried out interpolation and is covered, but this class technology is easy to generate significantly " bulk " phenomenon, is difficult for recovering the details of image.

When affected area is interframe (Inter) when coding, then mainly adopt time domain (temporal) to cover, promptly according to around correct motion vector information estimate and recover the image information that mistake is lost.Time domain is covered processing procedure and is divided into " motion vector (Motion Vector is called for short MV) estimation " and " compensation substitutes " two stages, and its successful key is how correctly to find out the motion vector of damaged image piece.

At present, the estimation of impaired motion vector mainly is correlation properties of utilizing the space.For example, when motion vector damages or loses, motion vector can be made as zero, carry out time domain alternative (Temporal Replacement is called for short TR) simply, it is for static relatively zone, as background etc., effect is fine, but to the regional effect extreme difference of rapid movement.The motion vector of losing also available around it the mean value (Average is called for short AV) or intermediate value (Median) of adjacent piece motion vector estimate, or with about it, about, the interpolation of direction such as diagonal angle substitutes.These methods are fine in level and smooth moving region effect, but in non-level and smooth moving region, and as the motion of different directions, object boundary etc., it is relatively poor then to cover effect.

Border coupling (Side Matching) technology is a kind of method of motion vector estimation preferably, and it mates the distortion factor (Side Match Distortion) minimum with the border is criterion, selects only motion vector from one group of candidate motion vector.This continuity with image affected area border judge the major advantage of motion vector be motion vector estimation based on distortion measurement, shortcoming is in the moving region with special texture and the object boundary place, causes easily and covers failure.

Above-mentioned these methods mainly are to utilize the damaged macro block (Macroblock is called for short MB) and the motion vector of adjacent macroblocks around it to have this feature of certain correlation, and are comparatively effective when motion vector is correct around the damaged macro block.If but the impaired or whole two field picture of macro block of multirow is lost continuously, then said method can lose efficacy, and can only take simple time domain concealing method usually.In IP-based Internet Video Applications or mobile communication environment, data loss rate is all very high usually, and in this case, a plurality of macro blocks damage even a two field picture is impaired all might take place.Because the restriction of real-time video transmission, these impaired images can not adopt the method that retransfers to recover again.

The objective of the invention is to propose a kind of video error coverage method of spreading to motion vector sought based on motion vector outward, it can carry out the error concealment of video image under the impaired or whole two field picture loss situation of macro block of continuous multirow, recover damaged image, overcome the deficiencies in the prior art, improve the picture quality of digital video phone transmission.

Summary of the invention

The present invention solves its technical problem, and the technical scheme that is adopted is:

A kind of video error coverage method of spreading to motion vector sought based on motion vector outward, steps such as error detection occurs, the motion vector by the receiving terminal video decode extrapolated, time domain is covered are formed, be characterized in: before the error detection occurs, all motion vectors of present frame under the stored record; The false retrieval of being on duty is surveyed when finding transmission error, the image affected area is divided into the damaged block of 8 * 8 pixels; And then carry out motion vector and extrapolate; Concrete grammar is:

If encoder is to encode with the macro block of 16 * 16 pixels, then, obtain the outside forecast macro block according to its movement tendency with the macro block motion vector extrapolation of all 16 * 16 pixels in the reference frame; Judge overlapping area between outside forecast macro block and the damaged block, the motion vector of the extrapolation macro block of overlapping area maximum is made as the motion vector of damaged block, thereby estimates the motion vector of damaged block; If encoder is to encode with the piece of 8 * 8 pixels, then the motion vector of piece is extrapolated, obtain the outside forecast piece, its motion vector is made as the motion vector of damaged block, thereby estimates the motion vector of damaged block;

According to the motion vector that above-mentioned steps estimates, use the image block of the image block replace damaged of the relevant position in the reference frame.

The invention has the beneficial effects as follows: based on the piece concealing method of time domain extrapolation, under the impaired or whole two field picture loss situation of macro block of continuous multirow, utilize the effective information of reference frame, blanketing a question when having solved the entire image LOF preferably, can improve the reliability of digital video image transmission significantly, improve the display effect of image; And the present invention and operating such H.26X, applied widely.

Above-mentioned video error coverage method of spreading to motion vector sought based on motion vector outward, when impaired or when losing image-region and on adjacent space, having the information that correctly receives, can combine with the frame coupling, optimize the motion vector that prediction estimates, its concrete practice is:

With motion vector, damaged block correspondence position and adjacent on every side proper exercise vector thereof in reference frame of correct piece around the damaged block, and the motion vector of the damaged block that estimates, form candidate motion vector set and close; From candidate motion vector set closes, select a motion vector successively, in reference frame, obtain the image block of relevant position, 1 to 2 pixel is outwards expanded on border around the image block, form frame; And damaged image piece frame and formed image block frame mated, a best motion vector of frame coupling is the motion vector after optimizing.

Above-mentioned video error coverage method of spreading to motion vector sought based on motion vector outward, also can carry out optimal selection again by search to the motion vector after optimizing, promptly, in reference frame, determine the center of compensation block, as starting point according to the motion vector after optimizing; Principle is searched for or is searched for fast in utilization entirely, and all the adjacent image point points in the certain limit around it are searched for, and finally establishes optimum motion vector according to the frame matching principle.

Utilization motion vector sought and frame have overcome the incorrect alternative shortcoming of generation on object boundary to a certain extent than method of completing the square, further improve the display effect of image.

Description of drawings:

Below in conjunction with the drawings and specific embodiments the present invention is described in further detail.

Fig. 1 is the FB(flow block) of the embodiment of the invention

Fig. 2 is extrapolation motion vector and the interpolative prediction macro block example of embodiment.

Fig. 3 is the overlapping region of maximum of embodiment and the piece motion vector schematic diagram of estimation.

Fig. 4 is the candidate motion vector set schematic diagram of embodiment.

Fig. 5 is the block-based frame coupling schematic diagram of embodiment.

Fig. 6 is that video test sequence " Foreman " objective performance of various concealing methods under whole frame of 115 LOFs and different packet loss rate situation of embodiment compares.

Fig. 7 is that video test sequence " Foreman " the 115th frame of embodiment is all lost, carry out error concealment after, be diffused into the subjective picture quality of the 118th frame.

Fig. 8 is under the packet loss of video test sequence " Foreman " 15% of embodiment, carries out the image effect of the 86th frame behind the error concealment.

Embodiment:

When the digital video telephone system realizes decoding, need at first to consider that the various information that may occur disturb and the packet loss problem, increase a large amount of unusual controls, as, whether the timestamp of judgment data bag or sequence number be correct, or whether the motion vector of judging each coding base unit has exceeded possible scope, whether the whether overshoot scope of compress coding data that receives, the entry address of tabling look-up when judging decoding be whether illegal, decoded data are normal, whether the number in each data block greater than normal or the like.In case the generation transmission is disturbed, then whole packet all can be dropped in based on the transmission of bag; In transmission, after decoder determines that mistake takes place, can not accurately determine the physical location that mistake takes place usually based on data flow, can only in data flow, search for forward, searching is different from the heavy synchronization character of normal video data, in case find, decoder just obtains again synchronously.Between these two synchronization characters, the discarded part divided data information of then having to.No matter which kind of situation all must adopt the error concealment technology to remedy the data of losing.

After finding loss of data,, then carry out spatial domain or frequency domain and cover for in-frame encoding picture;

For inter coded images; owing to have very strong correlation between the continuous video frame image; the object of which movement that is showed in the previous frame can continue to move along former direction in subsequent frame usually, only just can make an exception when moving object appearance, disappearance or video scene switch.Therefore, when image-region even whole two field picture were lost, the present invention utilized this rule to estimate exactly and recovers motion vector, and then carries out time domain and cover recovery, improved picture quality.

Fig. 1 shows embodiment of the invention treatment step:

During video decode (before the error detection occurs), all motion vectors of present frame under the stored record; When error detection occurs is found transmission error, carry out the motion vector extrapolation, obtain the outside forecast macro block according to its movement tendency, the motion vector of the extrapolation macro block of area maximum overlapping between outside forecast macro block and the damaged block is made as the motion vector of damaged block, as the motion vector that estimates; When image block is correct, carry out frame coupling and motion vector sought on every side; Carrying out time domain according to optimal motion vector then covers.

Specifically, the present embodiment detailed process is as follows:

Before the error detection occurs, stored record present frame motion vector (only needing the motion vector of record one two field picture); When finding transmission error, in order more accurately to compensate and to cover, the image affected area is divided into the damaged block of 8 * 8 pixels, with piece as the elementary cell of covering; And then carry out the motion vector extrapolation, as shown in Figure 2.

When encoder is to encode with the piece of 16 * 16 pixels, then, obtain the outside forecast macro block according to its movement tendency with the macro block motion vector extrapolation of all 16 * 16 pixels in the reference frame; Judge overlapping area between outside forecast macro block and the damaged block, the motion vector of the extrapolation macro block of overlapping area maximum is made as the motion vector of damaged block, thereby estimates the motion vector of damaged block, as shown in Figure 3.For instance, suppose that MB moves to the n-1 frame with invariable point-to-point speed from the n-2 frame 29 within a short period of time, be extrapolated to the n frame again, in the n frame, set up the outside forecast macro block thus.Occupy the area of damaged block according to the outside forecast macro block, promptly add up the number of pixel in the overlapping region, the motion vector of damaged block just can be determined.If damaged block and all outside forecast macro block are not overlapping, then the motion vector of this piece is made as the motion vector that equals its left side piece.If the piece of image left margin or Intra-coded blocks, then its motion vector is set to 0.

When encoder is to encode with the piece of 8 * 8 pixels, then the motion vector of piece is extrapolated, obtain the outside forecast piece, its motion vector is made as the motion vector of damaged block, thereby estimates the motion vector of damaged block;

According to the motion vector that above step estimates, use the image block of the image block replace damaged of the relevant position in the reference frame.

When impaired or when losing image-region and on adjacent space, having the information (being that the impaired part of image is not an entire image) that correctly receives, can combine with the frame coupling, optimize the motion vector that prediction estimates, make full use of the information in correct zone on every side, further improve the correctness of covering piece, concrete steps comprise:

A) with motion vector, damaged block correspondence position and adjacent on every side proper exercise vector thereof in reference frame of correct piece around the damaged block, and the motion vector of the damaged block that estimates, form candidate motion vector set and close;

Generally, the data about damaged block are all unreliable, for this reason, and when selecting candidate MV, by following Scheme Choice: for present frame: select the motion vector of upper and lower 6 pieces of damaged block, add the motion vector that this piece estimates; For reference frame: the correspondence position and the MV of 9 pieces on every side thereof; Amount to 16 in the candidate motion vector set, as shown in Figure 4.

B) from candidate motion vector set closes, select a motion vector successively, in reference frame, obtain the image block of relevant position, 1 to 2 pixel is outwards expanded on border around the image block, form frame, as shown in Figure 5.

The border matching principle that more existing research documents propose is to judge around the damaged macro block adjacent image point and cover error minimum between the alternative motion compensation macroblock boundaries pixel (being that continuity is best), and its shortcoming is the continuity that can not guarantee object edge preferably.For this reason, the inventive method is mated block-based border, has adopted the border matching principle of revising.Judge no longer promptly whether the damaged block border is continuous, but 1-2 pixel width (being called the frame coupling) will be enlarged around the motion compensation block in the reference frame, judge in these frames and the present frame whether mate most around the damaged image piece (the average variance minimum is called for short MSE).Simultaneously, with due regard to around the correct degree of adjacent block, give bigger weights for the MSE of correct adjacent block.

C) damaged image piece frame and formed image block frame are mated, a best motion vector of frame coupling is the motion vector after optimizing.Consider the needs of real-time operation, only consider the frame of three directions usually, the coupling distortion factor is calculated as formula (1)

D _T＝w _UD _U+w _LD _L+w _DD _D (1)

Be frame distortion factor D _TFrame distortion factor weighting sum between edge pixel upper and lower for impaired image block, that three directions in the left side are adjacent and the motion compensation block.If adjacent block is correct the reception, then weight w=1 if be to extrapolate to cover to obtain, then weighed and is planted w=1/4.Three adjacent frames (1 pixel wide) distortion factor defines as formula (2):

$D_L=\underset{i=0}{\overset{7}{\mathrm{\Σ}}}{[{\hat{f}}_n(p-1,q+i)-f_n(p-1,q+i)]}^2$

$D_U=\underset{i=0}{\overset{7}{\mathrm{\Σ}}}{[{\hat{f}}_n(p+i,q-1)-f_n(p+i,q-1)]}^2$

$D_D=\underset{i=0}{\overset{7}{\mathrm{\Σ}}}{[{\hat{f}}_n(p+i,q+8)-f_n(p+i,q+8)]}^2---\left(2\right)$

Here, ${\hat{f}}_n(p,q)=f_{n-1}(p+d^x\left(B\right),q+d^y\left(B\right)),$ Promptly equal the pixel f in the n frame _n(p, q) motion-compensated values in the n-1 frame.D (B) is the candidate motion vector of 8 * 8 damaged image piece.According to these motion vectors, obtain motion compensation block, carry out frame coupling again, with frame mate most as the motion vector after optimizing.

The present invention can also carry out optimal selection by motion vector sought again to the motion vector after optimizing, and promptly according to the motion vector after optimizing, determines the center of compensation block in reference frame, as starting point; Principle is searched for or is searched for fast in utilization entirely, and all the adjacent image point points in the certain limit around it are searched for, and finally establishes optimum motion vector according to the frame matching principle.

At last,, from reference frame, find motion compensation block, substitute current impaired piece, finish error concealment according to optimal motion vector.

Below in conjunction with a concrete legend the inventive method is done more concrete explanation:

QCIF (Quarter Common Intermediate Format) video test sequence with 172 * 144 sizes, compress with ITU-T video compression standard test model TMN8 encoder H.263+, adopt default mode, motion vector of each MB, the frame frequency of coding was 30 frame/seconds, with reference to frame frequency was 30 frame/seconds, and default quantization step is 13, and having only first frame is the I frame.

For the ease of the video transmission under the Internet environment, used GOB (Group of Block, 9 MB of 1 row are as 1 GOB) (interleaved) encapsulation strategy interweaves, odd number, the even number GOB of one frame video flowing are distinguished corresponding two UDP message bags, and encapsulate RFC2429 RTP (Real-time Transport Protocol) agreement again at the packet skin.Under strenuous exercise's situation, on average about nearly 500 bytes of each IP packet.This method has higher efficiency of transmission and robustness (Robustness), under the situation of losing a bag, still might reach the basis data of GOB up and down, and the GOB that loses is covered recovery.

Only concealing method of the present invention is described at this with losing of 1 piece in the macro block.

Suppose correct reception of the 114th frame and decoding, the odd number GOB of the 115th frame loses, and investigates the recovery of the 3rd the 3# piece among the 30#MB among the GOB now and covers situation (Fig. 3).Be (0,0) if the pixel of image the 1st row the 1st row is sat, then the coordinate in the 30#MB upper left corner is (96,32), and the coordinate of 3# piece wherein is (96,40), and the upper left corner coordinate of 29#MB is (80,32).

The 1st step: all macro blocks of the 114th frame are extrapolated according to its motion vector, (in fact since H.263+ to motion vector be limited in ± 16 pixel coverages in, only need to 30# and on every side 8 macro blocks extrapolate), 29#MB for example, its motion vector that correctly receives is (5,3), according to movement tendency, then the motion prediction macro block upper left corner of this macro block extrapolation is (85,35).

The 2nd step: according to the coordinate and the size (8 * 8 pixel) of the 3# piece among the 30#MB, can calculate the overlapping region is 5 * 8=40 pixel.Through calculating relatively, in this piece, this shared overlapping area maximum in outside forecast zone is then with the motion vector of its motion vector (5,3) as this damaged block.

The 3rd step: with the upper left corner in 114 frames is that (96-5=91,8 * 8 region units 30-3=37) substitute corrupted or lost view data.

The 4th step: because odd number GOB is impaired, the macro block about 30#MB is all lost, but its macro block up and down is correct.Therefore, the weights of the frame matching degree on top, the left side around the 3# piece are got 1/4, and following weights are 1.At the 115th frame, choose this 3, the motion vector of following 3 pieces and motion vector (5,3) that recovers above piece, get again in 114 frames and the corresponding locus of this piece and the motion vector of 8 pieces on every side thereof.These 16 motion vectors successively as the motion vector of damaged block, are found the frame of corresponding blocks in 114 frames, by formula calculate (1) (2), find a motion vector of distortion factor minimum at last with 3 frames of 3# piece among the 115 frame 30#MB;

The 5th step: to this motion vector is the center, again according to counterclockwise from inside to outside point by point search around the center ± all picture elements of 4, find out the motion vector of frame coupling distortion factor minimum, as the motion vector of optimum, and cover the view data of damage with data block corresponding in 114 frames.

All pieces of losing are all handled according to above-mentioned steps, can be covered all image informations of makeing mistakes.If whole two field picture is lost (odd even GOB all loses continuously) fully, then, therefore, also just needn't carry out the 4th, 5 step frame couplings again and calculate owing to do not have information trusty around losing piece.

Be objective evaluation this method, adopt reconstructed image Y-PSNR PSNR as evaluating.The Y-PSNR PSNR of single-frame images by formula (3) calculates.The PSNR of a video test sequence by formula (4) calculates.

$\mathrm{PSNR}={10\mathrm{<figure-callout\; id="10"\; label="log"\; filenames="C0311737700122.png"\; state="\{\{state\}\}">log</figure-callout>}}_{10}\frac{{255}^2}{\frac{1}{K}\underset{x,y}{\mathrm{\&Sigma;}}{({\mathrm{\&psi;}}_1(x,y)-{\mathrm{\&psi;}}_2(x,y))}^2}---\left(3\right)$

$\mathrm{PSNR}={10\mathrm{<figure-callout\; id="10"\; label="log"\; filenames="C0311737700122.png"\; state="\{\{state\}\}">log</figure-callout>}}_{10}\frac{{255}^2}{\frac{1}{K\&CenterDot;N}\underset{n}{\mathrm{\&Sigma;}}\underset{x,y}{\mathrm{\&Sigma;}}{({\mathrm{\&psi;}}_1(x,y,n)-{\mathrm{\&psi;}}_2(x,y,n))}^2}---\left(4\right)$

Here, ψ ₁Be raw video image pixel value, ψ ₂Be the back reconstructed image pixel value that decompresses, K is the summation of a frame pixel, and N is the total frame number of video sequence.

Be the contrast experiment, present embodiment has also been realized with the AV method of 3 the motion vector mean values in impaired MB top as the estimating motion vector.And this special circumstances of all entirely true recovery of all motion vectors behind the hypothesis missing image, this is actually the upper limit of time domain concealing method.If whole two field picture is lost, owing to there is not the adjacent MB in space, then AV method passive movement vector is 0 time domain alternative method, promptly substitutes with former frame.For the repeated experiments of different packet loss at random 30 times, remove two and two minimum Y-PSNRs (PSNR) the highest, calculate the mean value of all the other 24 PSNR.

Fig. 6 has provided the experimental result of the inventive method to strenuous exercise's cycle tests " Foreman ", and Fig. 7 has provided the 115th frame and all lost the video subjective effect that carries out the 118th two field picture behind the error concealment, and Fig. 8 is the video effect of the 86th frame under 15% packet loss.

Method of the present invention has solved the impaired and serious diffusion problem brought of high bit-error situation figure below picture frame preferably and has had more eager to do well in everything video data and cover and recovery capability.Computation complexity is low, and the real-time video transmission that is particularly suitable under the serious interference situation is used.

Claims (3)

1, a kind ofly spread to the video error coverage method of motion vector sought outward, step such as cover by error detection occurs, motion vector extrapolation, the time domain of receiving terminal video decode and form, it is characterized in that based on motion vector:

(1) before the error detection occurs, all motion vectors of present frame under the stored record; When error detection occurs is found transmission error, the image affected area is divided into the damaged block of 8 * 8 pixels; And then carry out motion vector and extrapolate; Concrete grammar is:

If a encoder is to encode with the macro block of 16 * 16 pixels, then, obtain the outside forecast macro block according to its movement tendency with the macro block motion vector extrapolation of all 16 * 16 pixels in the reference frame; Judge overlapping area between outside forecast macro block and the damaged block, the motion vector of the extrapolation macro block of overlapping area maximum is made as the motion vector of damaged block, thereby estimates the motion vector of damaged block;

If the b encoder is to encode with the piece of 8 * 8 pixels, then the motion vector of piece is extrapolated, obtain the outside forecast piece, its motion vector is made as the motion vector of damaged block, thereby estimates the motion vector of damaged block;

(2) motion vector that estimates according to step (1), the image block of the image block replace damaged of the relevant position in the usefulness reference frame.

2, a kind of video error coverage method of spreading to motion vector sought based on motion vector outward as claimed in claim 1, it is characterized in that: when there is the information that correctly receives in the damaged image zone on adjacent space, can combine with the frame coupling, optimize the motion vector that prediction estimates:

A, with motion vector, damaged block correspondence position and adjacent on every side proper exercise vector thereof in reference frame of correct piece around the damaged block, and the motion vector of the damaged block that estimates is formed candidate motion vector set and is closed;

B, from candidate motion vector set closes, select a motion vector successively, in reference frame, obtain the image block of relevant position,, form frame outside 1 to 2 pixel of expansion in border around the image block;

C, damaged image piece frame and the formed image block frame of step b are mated, a best motion vector of frame coupling is the motion vector after optimizing.

3, a kind of video error coverage method of spreading to motion vector sought based on motion vector outward as claimed in claim 2, it is characterized in that: the motion vector after optimizing is carried out optimal selection again by search, promptly according to the motion vector after optimizing, in reference frame, determine the center of compensation block, as starting point; Principle is searched for or is searched for fast in utilization entirely, and all the adjacent image point points in the certain limit around it are searched for, and finally establishes optimum motion vector according to the frame matching principle.

Images