CN105025201A - Space-time continuum video background repair method - Google Patents

Abstract

The invention discloses a space-time continuum video background repair method. Firstly, a playground of a cavity part in a video is estimated through a coarse-to-fine playground transmission method, so that the whole playground is natural and coherent; then, the playground is used as a guide, the cavity part is filled through a markov random field based optimization process, preferentially, known pixel in adjacent frames is arranged in a missing part; and finally, on the basis of an illumination migration idea, the invention provides an illumination adjustment method. According to the method, illumination conditions outside the cavity are effectively transmitted into the cavity, so that the illumination discontinuous phenomenon in the repaired cavity is eliminated. Compared with the existing video repair method, the method can better repair the seriously discontinuous video background, less limits the motion of a camera, and guarantees consistency of the resultant illumination under the condition of obvious illumination change.

Description

A kind of video background restorative procedure of space and time continuous

Technical field

The invention belongs to technical field of image processing, particularly relate to a kind of video background restorative procedure of space and time continuous.

Background technology

Universal along with video capture equipment, the video data in network and terminal significantly increases.But be subject to the impact of objective environment, there is the object of redundancy in a lot of video, have impact on the expression of theme and the complete of background.Remove unnecessary object and repair respective background region, that often can improve video views and admires experience.Such as, sometimes there is unnecessary passerby in the film of domestic consumer's shooting, recovery technique can be adopted to remove.In addition, background recovery technique can expand the application of original video, and such as, in three-dimensional streetscape map project, unnecessary pedestrian and vehicle in the street video utilizing background reparation can remove photographic car shooting, thus reconstruct street architecture thing better.

It is difficult for carrying out background reparation to the shot by camera video of compound movement.First due to the rotary motion of camera, between different frame there is projective transformation in background, and repair process must make up the difference on this visual angle.Secondly, some regional depths change in background is seriously discontinuous, causes simple method to be difficult to its motion of accurate description.Finally, under outdoor shooting environmental, when video camera rotates, often there is illumination in the reparation result of video discontinuous, need to regulate.For the data division lacked in video, have already been proposed multiple restorative procedure.But wherein quite a few method be only applicable to video camera static or with the situation of scene parallel motion.Although the video camera that remaining method can be used for compound movement shot the video, mostly utilize single should the relation, be only applicable to scene and form simple situation.In order to practical requirement, current applied widely in the urgent need to one, the video background restorative procedure of Be very effective.

Summary of the invention

The present invention, in order to solve above-mentioned technical problem, proposes the video background restorative procedure that sports ground guides.Technical scheme of the present invention is: a kind of video background restorative procedure of space and time continuous, is characterized in that, comprise the steps:

Step 1, is specified in input video V the foreground object Ω needing to remove by user, when the foreground object Ω removed is from other foreground objects Ω _×during rear side motion, user-interactive is also needed to mark Ω _×;

Step 2, makes Φ=V-Ω-Ω _×, Φ represents in video V the known portions that can be used for repairing lack part, calculates the positive and negative bidirectional-movement field of Φ;

Step 3, builds the space-time pyramid of sports ground, on the bottom, by minimizing following energy equation, calculates the initial value of lack part bidirectional-movement field:

$\underset{Q\⋐\mathrm{\Ω}\∩\∂\mathrm{\Ω}}{\Σ}\underset{P\⋐\mathrm{\Φ}}{\min }D(P,Q)$

Wherein, P and Q represents the pixel cube of time-space domain, and Q is targeting cube, comprises at least one unknown pixel, is positioned at Ω and border near zone thereof in, P is source cube, not containing unknown pixel; D (P, Q) represents the distance in sports ground between pixel cube;

Step 4, propagates into upper strata by the initial value of the lack part bidirectional-movement field of the space-time pyramid bottom, from the second layer, by minimizing following energy equation, calculates the fine values of lack part bidirectional-movement field:

$\mathrm{\β}\frac{1}{N_{\mathrm{\Ω}}}\underset{P\⋐\mathrm{\Ω}}{\Σ}\underset{Q\⋐{\mathrm{\Φ}}^{\′}}{\min }D(P,Q)+(1-\mathrm{\β})\frac{1}{N_{{\mathrm{\Φ}}^{\′}}}\underset{Q\⋐{\mathrm{\Φ}}^{\′}}{\Σ}\underset{P\⋐\mathrm{\Ω}}{\min }D(Q,P)$

Φ ' represents the known region around empty Ω, N _{Φ '}and N _Ωrepresent the cubical sum of pixel in Φ ' and Ω region, β is the weight controling parameters of two, front and back, and D (P, Q) and D (Q, P) are the distance between sports ground pixel cube;

Step 5, under the guiding of step 2 to the 4 complete bidirectional-movement fields calculated, utilizes one by one and closes on frame reparation present frame, obtain some local solutions of present frame;

Step 6, for present frame f _tinterior arbitrary missing pixel p, selects candidate point { p from the local solution set that step 5 obtains ¹, p ²..., p ⁿ, then carry out the optimization based on markov random file, obtain global solution, this process is minimized by following energy equation:

Σ _pE _p(L(p))+αΣ _p,qE _p,q(L(p),L(q))

Wherein function L (p) and L (q) are the mappings of the candidate point sequence number that pixel p and q finally select to it, and q is the 4 neighborhood points of p, and α is the controling parameters of two weights in front and back, E _p(L (p)) is data item, weighs and utilizes p ^{l (p)}the energy cost of repairing pixel p.E _p,q(L (p), L (q)) is continuous items, weighs and utilizes p ^{l (p)}and p ^{l (q)}repair neighbor p, the energy cost of q;

Step 7, the repair process of repeated execution of steps 5, step 6 secondary;

Step 8, from step 7 repair after video, select a key frame every ￠ frame; The illumination of adjustment key frame restoring area;

Step 9, propagates into non-key frame by key frame illumination adjustment result, completes the Illumination adjusting of whole video.

Preferably, also comprise the steps: in described step 3

Step 3.1, builds the space-time pyramid of original video sports ground;

Step 3.2, repair first from the bottom, one by one for targeting cube Q mates arest neighbors P, when known pixels number ratio is more than 60% in target pixel block, it is mated, otherwise skip in this circulation, only consider difference in cube between known pixels during coupling and sue for peace; Distance in sports ground between pixel cube adopts following tolerance:

$D(P,Q)=\mathrm{\Σ}(1-\frac{m_k\·m_{\hat{k}}}{\left|m_k\right|\left|m_{\hat{k}}\right|})$

K and represent all pixels of carrying out contrasting in P and Q, m _kwith represent the homogeneous coordinates of the light stream of correspondence position;

Step 3.3, copies to the position of Q by the P that coupling obtains, but only retains the light stream value corresponding with unknown position in targeting cube, in circulation next time, is considered as known by this position light stream;

Step 3.4, is cycled to repeat step 3.2 and 3.3, until whole pixel is filled light stream in cavity;

Step 3.5, repairs first after terminating, bottom sports ground is divided into overlapped pixel cube, distance between complete computation source cube and targeting cube, after coupling by source cube complete copy to targeting cube, for the position of overlap, its final light stream is determined by following formula:

$m_k=\frac{{\mathrm{\Σ}}_i{\mathrm{\ω}}_i{m}_k^i}{{\mathrm{\Σ}}_i{\mathrm{\ω}}_i}$

M in formula _krepresent the final light stream at pixel k place, represent i-th light stream of source cube at pixel k place copied.Weights ω _i=s _iλ _i, weigh the similitude between target pixel block and source pixel block, D (Q ⁱ, P ⁱ) be block of pixels Q ⁱwith P ⁱbetween distance, σ _dfor all D (Q ⁱ, P ⁱ) intermediate value, λ _iweigh target pixel block Q ⁱapart from the how far at empty edge, first calculate the distance L (Q at edge, each target pixel block center position cavity ⁱ), then calculate corresponding weights σ _lfor all L (Q ⁱ) intermediate value;

Said process successive ignition, when front and back, two times result difference is less than threshold value 15, or stops when iterations reaches preset value 20, obtains the initial value of lack part bidirectional-movement field.

Preferably, also comprise the steps: in described step 4

Step 4.1, propagates into adjacent last layer by the result of the bottom, forms initial value;

Step 4.2, is divided into overlapped pixel cube by sports ground in current layer;

Step 4.3, to two, the front and back alternating iteration optimization of formula in step 4, optimizing process adopts the method in step 3.5, when optimizing last item, searches for each targeting cube arest neighbors source cube calculate distance D (Q, P) between the two; When optimizing latter one, search for its arest neighbors targeting cube for each source cube and calculate distance D (P, Q);

Step 4.4, if there is last layer, then repairs result by current layer motion field and propagates into last layer sports ground, obtain the fine values of lack part bidirectional-movement field.

Preferably, also comprise the steps: in described step 5

Step 5.1, by present frame f _tbe decomposed into the two-dimensional block overlapped each other, the block of pixels containing missing pixel is called target pixel block, for each target pixel block T, according to forward light stream, find it at f _t+ithe block of pixels S of middle correspondence;

Step 5.2, copies to the position of T by the color value of S, obtain a local solution f _t ^t+i, in local solution, the final color value of pixel p is all weighted averages overlapping the color value of this point, weights ω _keach source pixel block meets Gaussian function, and parameter σ=10000 in this Gaussian function;

Step 5.3, according to the corresponding relation represented by reverse light stream, according to method described in step 5.1-5.2, uses f _t+irepair f _t, obtain another local solution

Step 5.4, for f _tall closes on frame f _t ^t+ii ∈ [-r, r], repeat step 5.1 to 5.3, obtain altogether 4r local solution, r is the radius closing on frame interval; 4r obtains two local solutions by every closing on frame, forms local solution set

Preferably, also comprise the steps: in described step 6

Step 6.1, if p is f _tinterior a certain missing pixel is that p selects n candidate point { p ¹, p ²..., p ⁿ, wherein, n≤4r; Each candidate point derives from interior a certain local solution, and identical with p coordinate figure; During selection, according to symmetric order from the near to the remote, from f _t-1and f _t+1start, to f _t-iand f _t+iset direction, wherein i > 0;

Step 6.2, for each pixel p provides initial value I (p), I (p), for all local solutions are in the weighted average of p place color value, first expands to the mask in cavity, selects one piece of reference zone EV (f near present frame cavity _t), to EV (f _t) in all pixels, find it closing on the corresponding points in frame according to light stream corresponding relation respectively, both calculating color data error, finally sues for peace; The weight that each corresponding points are shared in I (p) computing formula is defined as follows:

${\widetilde{\mathrm{\ω}}}_i=\exp (-\underset{p\∈EV\left(f_t\right)}{\Σ}|\left|f_t\right(p)-f_{t+i}\left(p^{\′}\right)\left|{|}_2\right)$

Wherein, p' represents that p is closing on the corresponding points in frame, EV (f _t) represent the reference zone that expansive working obtains, thus, in present frame, initial value I (p) of arbitrary missing pixel p is:

$I\left(p\right)=\frac{{\mathrm{\Σ}}_{i\∈\[-r,r\]}{\widetilde{\mathrm{\ω}}}_i{f}_{t+i}\left(p^{\′}\right)}{{\mathrm{\Σ}}_{i\∈\[-r,r\]}{\widetilde{\mathrm{\ω}}}_i}$

In formula, r is the radius closing on frame interval, for the weight that each corresponding points are shared in I (p), p' is that p is closing on the corresponding points in frame.

Definition of data item in Markov random field model is selected candidate point p by step 6.3 ^{l (p)}and the color distortion between p point initial value I (p):

E _p(L(p))＝||p ^L(p)-I(p)|| ₂

Coherent item in model is used for punishing that neighbor pixel derives from the color and gradient disparities that different local solution brings:

E _p,q(L(p),L(q))＝(||p ^L(p)-p ^L(q)|| ₂+||q ^L(p)-q ^L(q)|| ₂)+λ(||▽p ^L(p)-▽p ^L(q)|| ₂+||▽q ^L(p)-▽q ^L(q)|| ₂)

Wherein ▽ p and ▽ q represents the Grad at pixel place;

Substitute into the general equation in step 6 by these two, and adopt graph-cut figure segmentation method to solve, obtain this and repair result;

Preferably, also comprise the steps: in described step 8

Step 8.1, from the video after background reparation, extracts a frame as key frame every 5 frames, and extracts the textural characteristics on key frame;

Step 8.2, utilizes mean-shift mean shift algorithm to carry out cluster to original key frame, is decomposed into some fragments, mask corresponding for each fragment is carried out at least 3 times and expands, thus generate the overlapping region of at least 3 pixels between which;

Step 8.3, to each the fragment T comprising missing pixel, finds and mates with the immediate fragment S of its distance on texture from Φ;

Step 8.4, is moved to the illumination of S on T by adaptive illumination migration algorithm.In overlapping region, adopt the method transition light photograph of cross fade, any two fragment T ₁and T ₂overlapping region in the final illumination of pixel be:

$\mathrm{\ρ}\×{\mathrm{lum}}_{T_1}+(1-\mathrm{\ρ})\×{\mathrm{lum}}_{T_2}$

In formula with represent fragment T respectively ₁and T ₂at the illumination value at this some place, the value of parameter ρ is from T ₁to T ₂0 is reduced to linearly by 1.

Preferably, also comprise the steps: in described step 9

Step 9.1, if key frame f _tin, fragment T _tcomprise missing pixel, S _t∈ Φ is closest to T on texture _tfragment, according to light stream correspondence, at f _t+1in, S _tmove to S _t+1position, T _tmove to T _t+1position, by S _t+1illumination move to T _t+1on, if S _t+1move to outside frame, or containing the unknown pixel more than 20%, be then T _t+1at f _t+1inside again mate a fragment that texture is the most close, and the illumination on new arest neighbors is moved to T _t+1on, according to two-way light stream, by f _tillumination adjusting result propagate in the non-key frame of both sides;

Step 9.2, if f _t1and f _t2continuous print two key frames, according to video playback order, f _t1be positioned at f _t2above, f _t1and f _t2middle non-key frame is from f _t1and f _t2obtain two and propagate result, its final illumination is the linear weighted function mean value of these two results, and the result that nearer key frame is propagated occupies higher proportion.

Preferably, in described step 4: Φ ', by obtaining the expansion of empty mask, stops expanding when the number of pixels in expansion gained region arrives the twice of missing pixel number in cavity; If hole region missing pixel quantity exceedes 1/4 of sum of all pixels in frame, then stop when expansion area number of pixels is equal with missing pixel number expanding further; β is the weight controling parameters of two, front and back, and this reference value is 0.75.

Preferably, in described step 4, α is the controling parameters of two weights in front and back, provides reference value 6 herein; Described step 8, from step 7 repair after video, select a key frame every ￠ frame; ￠ is 5.

Preferably, in described step 7, the repair process of repeated execution of steps 5, step 6 it is secondary, be at least 2; The value of r and n is reduced at every turn; During last execution, mandatory requirement r=1, n=2.

The invention has the beneficial effects as follows: a kind of video background restorative procedure of space and time continuous, compared with existing video repairing method, the present invention can repair the serious discontinuous video background of the degree of depth better, less to the restriction of camera motion, and in the obvious situation of illumination variation, ensure the illumination consistency of result.

Accompanying drawing explanation

Fig. 1 is flow chart of the present invention.

Embodiment

Below in conjunction with embodiment and accompanying drawing, the present invention is described in further detail, but embodiments of the present invention are not limited thereto.

With reference to Fig. 1, flow chart of the present invention, a kind of video background restorative procedure of space and time continuous, comprises the steps:

Step 1, is specified in input video V the foreground object Ω needing to remove by user, when the foreground object Ω removed is from other foreground objects Ω _×during rear side motion, user-interactive is also needed to mark Ω _×;

Step 2, makes Φ=V-Ω-Ω _×, Φ represents in video V the known portions that can be used for repairing lack part, calculates the positive and negative bidirectional-movement field of Φ;

Step 3, builds the space-time pyramid of sports ground, on the bottom, by minimizing following energy equation, calculates the initial value of lack part bidirectional-movement field:

$\underset{Q\⋐\mathrm{\Ω}\∩\∂\mathrm{\Ω}}{\Σ}\underset{P\⋐\mathrm{\Φ}}{\min }D(P,Q)$

Wherein, P and Q represents the pixel cube of time-space domain, and Q is targeting cube, comprises at least one unknown pixel, is positioned at Ω and border near zone thereof in, P is source cube, not containing unknown pixel; D (P, Q) represents the distance in sports ground between pixel cube;

Also comprise the steps: in described step 3

Step 3.1, builds the space-time pyramid of original video sports ground;

Step 3.2, repair first from the bottom, one by one for targeting cube Q mates arest neighbors P, because empty inside is not containing any data, therefore only when in target pixel block, known pixels number ratio is more than 60%, it is mated, otherwise skip in this circulation, only consider difference in cube between known pixels during coupling and sue for peace; Distance in sports ground between pixel cube adopts following tolerance:

$D(P,Q)=\mathrm{\Σ}(1-\frac{m_k\·m_{\hat{k}}}{\left|m_k\right|\left|m_{\hat{k}}\right|})$

K and represent all pixels of carrying out contrasting in P and Q, m _kwith represent the homogeneous coordinates of the light stream of correspondence position;

Step 3.3, copies to the position of Q by the P that coupling obtains, but only retains the light stream value corresponding with unknown position in targeting cube, in circulation next time, is considered as known by this position light stream;

Step 3.4, is cycled to repeat step 3.2 and 3.3, until whole pixel is filled light stream in cavity;

Step 3.5, repair after terminating first, bottom sports ground is divided into overlapped pixel cube, because now targeting cube is complete, therefore can distance between complete computation source cube and targeting cube, after coupling by source cube complete copy to targeting cube, for the position of overlap, its final light stream is determined by following formula:

$m_k=\frac{{\mathrm{\Σ}}_i{\mathrm{\ω}}_i{m}_k^i}{{\mathrm{\Σ}}_i{\mathrm{\ω}}_i}$

M in formula _krepresent the final light stream at pixel k place, represent i-th light stream of source cube at pixel k place copied.Weights ω _i=s _iλ _i, weigh the similitude between target pixel block and source pixel block, D (Q ⁱ, P ⁱ) be block of pixels Q ⁱwith P ⁱbetween distance, σ _dfor all D (Q ⁱ, P ⁱ) intermediate value, λ _iweigh target pixel block Q ⁱapart from the how far at empty edge, first calculate the distance L (Q at edge, each target pixel block center position cavity ⁱ), then calculate corresponding weights σ _lfor all L (Q ⁱ) intermediate value;

Said process successive ignition, when front and back, two times result difference is less than threshold value 15, or stops when iterations reaches preset value 20, obtains the initial value of lack part bidirectional-movement field.

Step 4, propagates into upper strata by the initial value of the lack part bidirectional-movement field of the space-time pyramid bottom, from the second layer, by minimizing following energy equation, calculates the fine values of lack part bidirectional-movement field:

$\mathrm{\β}\frac{1}{N_{\mathrm{\Ω}}}\underset{P\⋐\mathrm{\Ω}}{\Σ}\underset{Q\⋐{\mathrm{\Φ}}^{\′}}{\min }D(P,Q)+(1-\mathrm{\β})\frac{1}{N_{{\mathrm{\Φ}}^{\′}}}\underset{Q\⋐{\mathrm{\Φ}}^{\′}}{\Σ}\underset{P\⋐\mathrm{\Ω}}{\min }D(Q,P)$

Φ ' represents the known region around empty Ω, by obtaining the expansion of empty mask, stops expanding when the number of pixels in expansion gained region arrives the twice of missing pixel number in cavity; If hole region missing pixel quantity exceedes 1/4 of sum of all pixels in frame, then stop when expansion area number of pixels is equal with missing pixel number expanding further; N _{Φ '}and N _Ωrepresent the cubical sum of pixel in Φ ' and Ω region, β is the weight controling parameters of two, front and back, and concrete value can be determined according to corresponding video content, provide reference value 0.75 herein, D (P, Q) and D (Q, P) are the distance between sports ground pixel cube;

Also comprise the steps: in described step 4

Step 4.1, propagates into adjacent last layer by the result of the bottom, forms initial value;

Step 4.2, is divided into overlapped pixel cube by sports ground in current layer;

Step 4.3, to two, the front and back alternating iteration optimization of formula in step 4, optimizing process adopts the method in step 3.5, when optimizing last item, searches for each targeting cube arest neighbors source cube calculate distance D (Q, P) between the two; When optimizing latter one, search for its arest neighbors targeting cube for each source cube and calculate distance D (P, Q);

Step 4.4, if there is last layer, then repairs result by current layer motion field and propagates into last layer sports ground, obtain the fine values of lack part bidirectional-movement field.

Step 5, under the guiding of the complete bidirectional-movement field that step 2-4 calculates, utilizes one by one and closes on frame reparation present frame, obtain some local solutions of present frame;

Also comprise the steps: in described step 5

Step 5.1, by present frame f _tbe decomposed into the two-dimensional block overlapped each other, the block of pixels containing missing pixel is called target pixel block, for each target pixel block T, according to forward light stream, find it at f _t+ithe block of pixels S of middle correspondence;

Step 5.2, copies to the position of T by the color value of S, obtain a local solution f _t ^t+i, in local solution, the final color value of pixel p is all weighted averages overlapping the color value of this point, weights ω _keach source pixel block meets Gaussian function, and parameter σ=10000 in this Gaussian function;

Step 5.3, according to the corresponding relation represented by reverse light stream, according to method described in step 5.1-5.2, uses f _t+irepair f _t, obtain another local solution

Step 5.4, for f _tall closes on frame f _t ^t+i(i ∈ [-r, r]) repeats step 5.1 to 5.3, and obtain altogether 4r local solution, r is the radius closing on frame interval; 4r obtains two local solutions by every closing on frame, forms local solution set

Step 6, for present frame f _tinterior arbitrary missing pixel p, selects candidate point { p from the local solution set that step 5 obtains ¹, p ²..., p ⁿ, then carry out the optimization based on markov random file, obtain global solution, this process is minimized by following energy equation:

Σ _pE _p(L(p))+αΣ _p,qE _p,q(L(p),L(q))

Wherein function L (p) and L (q) are the mappings of the candidate point sequence number that pixel p and q finally select to it, and q is the 4 neighborhood points of p, and α is the controling parameters of two weights in front and back, provides reference value 6 herein; E _p(L (p)) is data item, weighs and utilizes p ^{l (p)}the energy cost of repairing pixel p.E _p,q(L (p), L (q)) is continuous items, weighs and utilizes p ^{l (p)}and p ^{l (q)}repair neighbor p, the energy cost of q;

Also comprise the steps: in described step 6

Step 6.1, if p is f _tinterior a certain missing pixel is that p selects n (n≤4r) individual candidate point { p ¹, p ²..., p ⁿ, each candidate point derives from interior a certain local solution, and identical with p coordinate figure; During selection, according to symmetric order from the near to the remote, from f _t-1and f _t+1start, to f _t-iand f _t+ithe set direction of (i > 0);

Step 6.2, for each pixel p provides initial value I (p), I (p), for all local solutions are in the weighted average of p place color value, first expands to the mask in cavity, selects one piece of reference zone EV (f near present frame cavity _t), to EV (f _t) in all pixels, find it closing on the corresponding points in frame according to light stream corresponding relation respectively, both calculating color data error, finally sues for peace; The weight that each corresponding points are shared in I (p) computing formula is defined as follows:

${\widetilde{\mathrm{\ω}}}_i=\exp (-\underset{p\∈EV\left(f_t\right)}{\Σ}|\left|f_t\right(p)-f_{t+i}\left(p^{\′}\right)\left|{|}_2\right)$

Wherein, p' represents that p is closing on the corresponding points in frame, EV (f _t) represent the reference zone that expansive working obtains, thus, in present frame, initial value I (p) of arbitrary missing pixel p is:

$I\left(p\right)=\frac{{\mathrm{\Σ}}_{i\∈\[-r,r\]}{\widetilde{\mathrm{\ω}}}_i{f}_{t+i}\left(p^{\′}\right)}{{\mathrm{\Σ}}_{i\∈\[-r,r\]}{\widetilde{\mathrm{\ω}}}_i}$

In formula, r is the radius closing on frame interval, for the weight that each corresponding points are shared in I (p), p' is that p is closing on the corresponding points in frame.

Definition of data item in Markov random field model is selected candidate point p by step 6.3 ^{l (p)}and the color distortion between p point initial value I (p):

E _p(L(p))＝||p ^L(p)-I(p)|| ₂

Coherent item in model is used for punishing that neighbor pixel derives from the color and gradient disparities that different local solution brings:

E _p,q(L(p),L(q))＝(||p ^L(p)-p ^L(q)|| ₂+||q ^L(p)-q ^L(q)|| ₂)+λ(||▽p ^L(p)-▽p ^L(q)|| ₂+||▽q ^L(p)-▽q ^L(q)|| ₂)

Wherein ▽ p and ▽ q represents the Grad at pixel place;

Substitute into the general equation in step 6 by these two, and adopt graph-cut figure segmentation method to solve, obtain this and repair result;

Step 7, the repair process of repeated execution of steps 5, step 6 2-3 time, reduces the value of r and n at every turn; During last execution, mandatory requirement r=1, n=2;

Step 8, from step 7 repair after video, select a key frame every five frames; The illumination of adjustment key frame restoring area;

Also comprise the steps: in described step 8

Step 8.1, from the video after background reparation, extracts a frame as key frame every 5 frames, and extracts the textural characteristics on key frame;

Step 8.2, utilizes mean-shift mean shift algorithm to carry out cluster to original key frame, is decomposed into some fragments.The quantity of final cluster can be set to 2,3 or 4 according to video content.Mask corresponding for each fragment is carried out 3 to 4 times to expand, thus generate the overlapping region of 3 to 4 pixels between which;

Step 8.3, to each the fragment T comprising missing pixel, finds and mates with the immediate fragment S of its distance on texture from Φ;

Step 8.4, is moved to the illumination of S on T by adaptive illumination migration algorithm.In overlapping region, adopt the method transition light photograph of cross fade, any two fragment T ₁and T ₂overlapping region in the final illumination of pixel be:

$\mathrm{\ρ}\×{\mathrm{lum}}_{T_1}+(1-\mathrm{\ρ})\×{\mathrm{lum}}_{T_2}$

In formula with represent fragment T respectively ₁and T ₂at the illumination value at this some place, the value of parameter ρ is from T ₁to T ₂0 is reduced to linearly by 1.

Step 9, propagates into non-key frame by key frame illumination adjustment result, completes the Illumination adjusting of whole video.Also comprise the steps: in described step 9

Step 9.1, if key frame f _tin, fragment T _tcomprise missing pixel, S _t∈ Φ is closest to T on texture _tfragment, according to light stream correspondence, at f _t+1in, S _tmove to S _t+1position, T _tmove to T _t+1position, by S _t+1illumination move to T _t+1on, if S _t+1move to outside frame, or containing the unknown pixel more than 20%, be then T _t+1at f _t+1inside again mate a fragment that texture is the most close, and the illumination on new arest neighbors is moved to T _t+1on, according to two-way light stream, by f _tillumination adjusting result propagate in the non-key frame of both sides;

Step 9.2, if f _t1and f _t2continuous print two key frames, according to video playback order, f _t1be positioned at f _t2above, f _t1and f _t2middle non-key frame is from f _t1and f _t2obtain two and propagate result, its final illumination is the linear weighted function mean value of these two results, and the result that nearer key frame is propagated occupies higher proportion.

Claims (10)

1. a video background restorative procedure for space and time continuous, is characterized in that, comprise the steps:

Step 1, is specified in input video V the foreground object Ω needing to remove by user, when the foreground object Ω removed is from other foreground objects Ω _×during rear side motion, user-interactive is also needed to mark Ω _×;

Step 2, makes Φ=V-Ω-Ω _×, Φ represents in video V the known portions that can be used for repairing lack part, calculates the positive and negative bidirectional-movement field of Φ;

Step 3, builds the space-time pyramid of sports ground, on the bottom, by minimizing following energy equation, calculates the initial value of lack part bidirectional-movement field:

$\underset{Q\⋐\mathrm{\Ω}\∩\∂\mathrm{\Ω}}{\Σ}\underset{P\⋐\mathrm{\Φ}}{\min }D(P,Q)$

Wherein, P and Q represents the pixel cube of time-space domain, and Q is targeting cube, comprises at least one unknown pixel, is positioned at Ω and border near zone thereof in, P is source cube, not containing unknown pixel; D (P, Q) represents the distance in sports ground between pixel cube;

Step 4, propagates into upper strata by the initial value of the lack part bidirectional-movement field of the space-time pyramid bottom, from the second layer, by minimizing following energy equation, calculates the fine values of lack part bidirectional-movement field:

$\mathrm{\β}\frac{1}{N_{\mathrm{\Ω}}}\underset{P\⋐\mathrm{\Ω}}{\Σ}\underset{Q\⋐{\mathrm{\Φ}}^{\′}}{\min }D(P,Q)+(1-\mathrm{\β})\frac{1}{N_{{\mathrm{\Φ}}^{\′}}}\underset{Q\⋐{\mathrm{\Φ}}^{\′}}{\Σ}\underset{P\⋐\mathrm{\Ω}}{\min }D(Q,P)$

Φ ' represents the known region around empty Ω, N _{Φ '}and N _Ωrepresent the cubical sum of pixel in Φ ' and Ω region, β is the weight controling parameters of two, front and back, and D (P, Q) and D (Q, P) are the distance between sports ground pixel cube;

Step 5, under the guiding of step 2 to the 4 complete bidirectional-movement fields calculated, utilizes one by one and closes on frame reparation present frame, obtain some local solutions of present frame;

Step 6, for present frame f _tinterior arbitrary missing pixel p, selects candidate point { p from the local solution set that step 5 obtains ¹, p ²..., p ⁿ, then carry out the optimization based on markov random file, obtain global solution, this process is minimized by following energy equation:

${\mathrm{\Σ}}_p{E}_p\left(L\right(p\left)\right)+{\mathrm{\α\Σ}}_{p,q}{E}_{p,q}\left(L\right(p),L(q\left)\right)$

Wherein function L (p) and L (q) are the mappings of the candidate point sequence number that pixel p and q finally select to it, and q is the 4 neighborhood points of p, and α is the controling parameters of two weights in front and back, E _p(L (p)) is data item, weighs and utilizes p ^{l (p)}the energy cost of repairing pixel p; E _p,q(L (p), L (q)) is continuous items, weighs and utilizes p ^{l (p)}and p ^{l (q)}repair neighbor p, the energy cost of q;

Step 7, the repair process of repeated execution of steps 5, step 6 secondary;

Step 8, from step 7 repair after video, select a key frame every ￠ frame; The illumination of adjustment key frame restoring area;

Step 9, propagates into non-key frame by key frame illumination adjustment result, completes the Illumination adjusting of whole video.

2. the video background restorative procedure of a kind of space and time continuous according to claim 1, is characterized in that, also comprise the steps: in described step 3

Step 3.1, builds the space-time pyramid of original video sports ground;

Step 3.2, repair first from the bottom, one by one for targeting cube Q mates arest neighbors P, when known pixels number ratio is more than 60% in target pixel block, it is mated, otherwise skip in this circulation, only consider difference in cube between known pixels during coupling and sue for peace; Distance in sports ground between pixel cube adopts following tolerance:

$D(P,Q)=\mathrm{\Σ}(1-\frac{m_k\·m_{\hat{k}}}{\left|m_k\right|\left|m_{\hat{k}}\right|})$

K and represent all pixels of carrying out contrasting in P and Q, m _kwith represent the homogeneous coordinates of the light stream of correspondence position;

Step 3.3, copies to the position of Q by the P that coupling obtains, but only retains the light stream value corresponding with unknown position in targeting cube, in circulation next time, is considered as known by this position light stream;

Step 3.4, is cycled to repeat step 3.2 and 3.3, until whole pixel is filled light stream in cavity;

Step 3.5, repairs first after terminating, bottom sports ground is divided into overlapped pixel cube, distance between complete computation source cube and targeting cube, after coupling by source cube complete copy to targeting cube, for the position of overlap, its final light stream is determined by following formula:

$m_k=\frac{{\mathrm{\Σ}}_i{\mathrm{\ω}}_i{m}_k^i}{{\mathrm{\Σ}}_i{\mathrm{\ω}}_i}$

M in formula _krepresent the final light stream at pixel k place, represent i-th light stream of source cube at pixel k place copied, weights ω _i=s _iλ _i, weigh the similitude between target pixel block and source pixel block, D (Q ⁱ, P ⁱ) be block of pixels Q ⁱwith P ⁱbetween distance, σ _dfor all D (Q ⁱ, P ⁱ) intermediate value, λ _iweigh target pixel block Q ⁱapart from the how far at empty edge, first calculate the distance L (Q at edge, each target pixel block center position cavity ⁱ), then calculate corresponding weights ${\mathrm{\λ}}_i=\exp (-L(Q^i)/2{\mathrm{\σ}}_l^2),$ σ _lfor all L (Q ⁱ) intermediate value;

Said process successive ignition, when front and back, two times result difference is less than threshold value 15, or stops when iterations reaches preset value 20, obtains the initial value of lack part bidirectional-movement field.

3. the video background restorative procedure of a kind of space and time continuous according to claim 1, is characterized in that, also comprise the steps: in described step 4

Step 4.1, propagates into adjacent last layer by the result of the bottom, forms initial value;

Step 4.2, is divided into overlapped pixel cube by sports ground in current layer;

Step 4.3, to two, the front and back alternating iteration optimization of formula in step 4, optimizing process adopts the method in step 3.5, when optimizing last item, searches for each targeting cube arest neighbors source cube calculate distance D (Q, P) between the two; When optimizing latter one, search for its arest neighbors targeting cube for each source cube and calculate distance D (P, Q);

Step 4.4, if there is last layer, then repairs result by current layer motion field and propagates into last layer sports ground, obtain the fine values of lack part bidirectional-movement field.

4. the video background restorative procedure of a kind of space and time continuous according to claim 1, is characterized in that, also comprise the steps: in described step 5

Step 5.1, by present frame f _tbe decomposed into the two-dimensional block overlapped each other, the block of pixels containing missing pixel is called target pixel block, for each target pixel block T, according to forward light stream, find it at f _t+ithe block of pixels S of middle correspondence;

Step 5.2, copies to the position of T by the color value of S, obtain a local solution in local solution, the final color value of pixel p is all weighted averages overlapping the color value of this point, weights ω _keach source pixel block meets Gaussian function, and parameter σ=10000 in this Gaussian function;

Step 5.3, according to the corresponding relation represented by reverse light stream, according to method described in step 5.1-5.2, uses f _t+irepair f _t, obtain another local solution

Step 5.4, for f _tall closes on frame i ∈ [-r, r], repeat step 5.1 to 5.3, obtain altogether 4r local solution, r is the radius closing on frame interval; 4r obtains two local solutions by every closing on frame, forms local solution set

5. the video background restorative procedure of a kind of space and time continuous according to claim 1, is characterized in that, also comprise the steps: in described step 6

Step 6.1, if p is f _tinterior a certain missing pixel is that p selects n candidate point { p ¹, p ²..., p ⁿ, wherein, n≤4r; Each candidate point derives from interior a certain local solution, and identical with p coordinate figure; During selection, according to symmetric order from the near to the remote, from f _t-1and f _t+1start, to f _t-iand f _t+iset direction, wherein i > 0;

Step 6.2, for each pixel p provides initial value I (p), I (p), for all local solutions are in the weighted average of p place color value, first expands to the mask in cavity, selects one piece of reference zone EV (f near present frame cavity _t), to EV (f _t) in all pixels, find it closing on the corresponding points in frame according to light stream corresponding relation respectively, both calculating color data error, finally sues for peace; The weight that each corresponding points are shared in I (p) computing formula is defined as follows:

${\widetilde{\mathrm{\ω}}}_i=\exp (-\underset{p\∈EV\left(f_t\right)}{\Σ}|\left|f_t\right(p)-f_{t+i}\left(p^{\′}\right)\left|{|}_2\right)$

Wherein, p' represents that p is closing on the corresponding points in frame, EV (f _t) represent the reference zone that expansive working obtains, thus, in present frame, initial value I (p) of arbitrary missing pixel p is:

$I\left(p\right)=\frac{{\mathrm{\Σ}}_{i\∈\[-r,r\]}{\widetilde{\mathrm{\ω}}}_i{f}_{t+i}\left(p^{\′}\right)}{{\mathrm{\Σ}}_{i\∈\[-r,r\]}{\widetilde{\mathrm{\ω}}}_i}$

In formula, r is the radius closing on frame interval, for the weight that each corresponding points are shared in I (p), p' is that p is closing on the corresponding points in frame;

Definition of data item in Markov random field model is selected candidate point p by step 6.3 ^{l (p)}and the color distortion between p point initial value I (p):

E _p(L(p))＝||p ^L(p)-I(p)|| ₂

Coherent item in model is used for punishing that neighbor pixel derives from the color and gradient disparities that different local solution brings:

$E_{p,q}\left(L\right(p),L(q\left)\right)=\left(\right||p^{L\left(p\right)}-p^{L\left(q\right)}|{|}_2+\left|\right|q^{L\left(p\right)}-q^{L\left(q\right)}\left|{|}_2\right)+\mathrm{\λ}\left(\right||\▿p^{L\left(p\right)}-\▿p^{L\left(q\right)}|{|}_2+\left|\right|\▿q^{L\left(p\right)}-\▿q^{L\left(q\right)}\left|{|}_2\right)$

Wherein with represent the Grad at pixel place;

Substitute into the general equation in step 6 by these two, and adopt graph-cut figure segmentation method to solve, obtain this and repair result.

6. the video background restorative procedure of a kind of space and time continuous according to claim 1, is characterized in that, also comprise the steps: in described step 8

Step 8.1, from the video after background reparation, extracts a frame as key frame every 5 frames, and extracts the textural characteristics on key frame;

Step 8.2, utilizes mean-shift mean shift algorithm to carry out cluster to original key frame, is decomposed into some fragments, mask corresponding for each fragment is carried out at least 3 times and expands, thus generate the overlapping region of at least 3 pixels between which;

Step 8.3, to each the fragment T comprising missing pixel, finds and mates with the immediate fragment S of its distance on texture from Φ;

Step 8.4, moves on T by adaptive illumination migration algorithm by the illumination of S, in overlapping region, adopts the method transition light photograph of cross fade, any two fragment T ₁and T ₂overlapping region in the final illumination of pixel be:

$\mathrm{\ρ}\×{\mathrm{lum}}_{T_1}+(1-\mathrm{\ρ})\×{\mathrm{lum}}_{T_2}$

In formula with represent fragment T respectively ₁and T ₂at the illumination value at this some place, the value of parameter ρ is from T ₁to T ₂0 is reduced to linearly by 1.

7. the video background restorative procedure of a kind of space and time continuous according to claim 1, is characterized in that, also comprise the steps: in described step 9

Step 9.1, if key frame f _tin, fragment T _tcomprise missing pixel, S _t∈ Φ is closest to T on texture _tfragment, according to light stream correspondence, at f _t+1in, S _tmove to S _t+1position, T _tmove to T _t+1position, by S _t+1illumination move to T _t+1on, if S _t+1move to outside frame, or containing the unknown pixel more than 20%, be then T _t+1at f _t+1inside again mate a fragment that texture is the most close, and the illumination on new arest neighbors is moved to T _t+1on, according to two-way light stream, by f _tillumination adjusting result propagate in the non-key frame of both sides;

Step 9.2, if f _t1and f _t2continuous print two key frames, according to video playback order, f _t1be positioned at f _t2above, f _t1and f _t2middle non-key frame is from f _t1and f _t2obtain two and propagate result, its final illumination is the linear weighted function mean value of these two results, and the result that nearer key frame is propagated occupies higher proportion.

8. the video background restorative procedure of a kind of space and time continuous according to claim 1, it is characterized in that, in described step 4: Φ ', by obtaining the expansion of empty mask, stops expanding when the number of pixels in expansion gained region arrives the twice of missing pixel number in cavity; If hole region missing pixel quantity exceedes 1/4 of sum of all pixels in frame, then stop when expansion area number of pixels is equal with missing pixel number expanding further; β is the weight controling parameters of two, front and back, and this reference value is 0.75.

9. the video background restorative procedure of a kind of space and time continuous according to claim 1, is characterized in that: in described step 4, and α is the controling parameters of two weights in front and back, provides reference value 6 herein; Described step 8, from step 7 repair after video, select a key frame every ￠ frame; ￠ is 5.

10. the video background restorative procedure of a kind of space and time continuous according to claim 1, is characterized in that: in described step 7, the repair process of repeated execution of steps 5, step 6 it is secondary, be at least 2; The value of r and n is reduced at every turn; During last execution, mandatory requirement r=1, n=2.