CN102609956A - Editing method for human motions in videos - Google Patents

Abstract

The invention discloses an editing method for human motions in videos, which supports users to edit human motions in videos so as to obtain videos containing new human motions. The method concretely comprises the following three steps (namely, human posture analysis, posture data editing, and image deformation): firstly, by using an immune evolution based hierarchical posture analysis method, through global posture optimization, posture sequence smoothing and local attitude optimization calculation, obtaining the posture data of a human motion inputted in a video; then, editing the calculated posture data of the human motion in a way of sketch interaction so as to generate new posture data of the human motion; and finally, deforming each frame of image inputted in the video by using a model-driven moving least square image deformation method so as to generate a video containing a new human motion. By using the method disclosed by the invention, a human motion in a video can be edited so as to obtain a video containing a new human motion, therefore, the method has important applications in the fields such as film post-production and advertisement design.

Description

Human motion edit methods in a kind of video

Technical field

The present invention relates to the human motion edit methods in a kind of video, belong to computer picture Video processing research field, the method that specifically a kind of user of support edits the human motion in the video.

Background technology

Picture editting's technology is widely used in real life, as adopting edit tools such as Photoshop image energy is realized such as multiple editing operations such as tone adjustment, structural adjustment, image repair, image denoising, anamorphoses.But at present existing edit tool such as the Effects of Adobe and the Shake of Apple etc. to video only can realize scratching basic operations such as figure and foreground extraction.Realize editor, remain the task that item has challenge like the editor of personal appearance in the video and motion etc. to video content.Yet video editing has important application in fields such as advertisement design, film post-production, and potential using value makes video editing become one of important topic of computer picture Video processing research field.

At present, the existing research of video editing mainly concentrates on three aspects such as basic exercise editor, motion special effect processing, special object editor.Like document 1:SCHOLZ, V., EL-ABED; S., SEIDEL, H.-P.; AND MAGNOR; M.A.Editing object behavior in video sequences.CGF, 2009.28 (6): 1632-1643 has realized the video editing method that some are basic, as removes and add the processing etc. of motion editing, non-rigid object deformation, keyframe interpolation and the camera motion of object, object; This generic operation can only be realized the basic edit of video then, can't satisfy such as the outward appearance of editor's human body and the requirement of attitude; Document 2:WANG, J., DRUCKER, S.M.; AGRAWALA, M., AND COHEN; M.F The cartoon animation filter.ACM TOG, 2006,25 (3): 1169-1173. has proposed a kind of animation filtering method; This method can make the human motion in the animation become and more exaggerate and innervation, but this method only is the outward appearance that changes human body in the image through deformation method, is difficult to realize the accurate control to attitude; Document 3:LEYVAND, T., COHEN-OR, D.; DROR, G., AND LISCHINSKI; D.Data-driven enhancement of facial attractiveness.ACM TOG, 2008,27 (3): 1-9 goes out a two-dimentional deformation function from facial image training focusing study; Can carry out the attractive force of deformation to the people's face in the input picture, but this method only can be handled facial image, can't be extended to such as among the human body editor with the increase expression.In addition, the research to human appearance editor in image and the video has also appearred in recent years, like document 4:ZHOU, S.; FU, H., LIU, L.; COHEN-OR, D., AND HAN, X.Parametric reshaping of human bodies in images.ACM TOG; 2010,29 (4): 1-10 introduces the parametrization three-dimensional (3 D) manikin, has realized the direct editor to human appearance through the appearance attribute of adjustment human body; Document 5:ARJUN Jain; Thorsten Thorm ¨ ahlen; Hans-Peter Seidel and Christian Theobalt.MovieReshape:Tracking and Reshaping of Humans in Videos.ACM TOG, 2010,29 (5) work with document 4 expand to video; Realized editor, but this method can not be edited the attitude of human body to human appearance in the video.

Summary is got up; The research of current video edit methods mainly concentrates on basic exercise editor, motion special effect processing, special object editor etc.; Also only limit to the editor of human appearance to the editor of human body, visible as yet to the editor's of human body attitude in the video and motion research.This mainly is because the video human motion editing is the task that item has challenge, comprises following reason: at first, the human skeleton between the maintenance frame of video and the consistance of outward appearance are the primary difficult problems of video human motion editing; Secondly, how effective user interface being provided is the important difficult problem that research faces.Yet the video human motion editing has important application in advertisement design and film making field, and human body attitude and motion editing technology can support the artist to adopt more freely way of art to go to describe plot through move edit.Important academic values and application prospect make video human attitude editor become research contents of the present invention.

Summary of the invention

Goal of the invention: technical matters to be solved by this invention is the deficiency to prior art, has proposed the human motion edit methods in a kind of video.

Technical scheme: the invention discloses the human motion edit methods in a kind of video, may further comprise the steps:

Step 1, human body attitude analysis: adopt stratification pose refinement method, calculate the human body attitude of each two field picture in the input video, obtain the attitude data of human motion based on immunoevolution;

Step 2, attitude data editor: adopt grass to paint the attitude data of the human motion that interactive mode edit step one obtains, generate the attitude data of new human motion;

Step 3, image deformation: as controlled condition, adopt the mobile least square image deformation method of model-driven that each two field picture in the input video is carried out deformation with the attitude data of human motion new in the step 2, generate the video that comprises new human motion.

Based on the stratification posture analysis method of immunoevolution, belong to production posture analysis method described in the step 1, manikin of its explicit definition is realized the human body attitude estimation through projection and the fitness between characteristics of image of optimizing manikin.Hierarchy optimization thought is the attitude data that calculates the human motion in the input video through overall pose refinement, level and smooth, the local pose refinement of attitude sequence.Owing to utilized the time sequence information of motion and, effectively improved the accuracy of posture analysis to the local optimum of attitude.May further comprise the steps:

Step 21, overall pose refinement: to each two field picture z of input video _t, adopt immunoevolution pose refinement method to calculate a candidate attitudes set that comprises N candidate attitudes

Wherein,

Be a candidate attitudes, i is the sequence number of candidate attitudes, i=1 ..., N, N are total number of candidate attitudes, t=1 ..., T, T are the totalframes of input video;

Step 22, the attitude sequence is level and smooth: adopt the attitude sequence smoothing method based on dynamic programming, gather from candidate attitudes

Candidate attitudes of middle selection is as image z _tEstimate attitude

Wherein, attitude is estimated in h (t) expression Gather in candidate attitudes

In sequence number, h (t) ∈ [1, N];

Step 23, local pose refinement: utilize and estimate attitude

Reorientation image z _tIn human body contour outline; Adopt immunoevolution pose refinement method to calculate the local attitude of human body, obtain image z _tThe attitude data y of human motion _t

The pose refinement of immunoevolution described in the step 21 method is a kind of optimization method of having gathered evolutionary mechanism and immunologic mechanism, through introducing attitude vaccine and immune operator, has improved the convergence and the local optimum ability of pose refinement, specifically may further comprise the steps:

Step 31, initialization: it is individual in the restriction range of vaccine V definition, to produce M attitude at random

Be designated as population A ₀Wherein, Be an attitude individuality, m is the individual sequence number of attitude, and M is that the individual total number of attitude is the scale of population in the population, and 0 has represented the algebraically of population, y _jBe each individual dimension of attitude, j=1 ..., J, J are the individual dimension of attitude; It is individual that said vaccine V has defined attitude

The span of each dimension, i.e. min (y _j)＜y _j＜max (y _j), min (y _j) and max (y _j) be respectively y _jThe minimum value of value and maximal value;

Step 32, the iteration optimization population A ₀, may further comprise the steps:

Step 321 is calculated k for population A _kIn the individual fitness of each attitude, if current population A _kIn comprise the optimum posture individuality, then out of service and output result, otherwise continue; The individual fitness of said attitude refers to the individual corresponding attitude model of attitude and the distance of characteristics of image, adopts profile and edge feature to calculate the individual fitness of attitude

Computing method are $E(y_t^{m,k},z_t)=\mathrm{Exp}(-\mathrm{\Σ}(w^s(y_t^{m,k},z_t)+w^e\left(y_t^{m,k}{z}_t\right))),$ Wherein,

Be that k is for population A _kIn an attitude individuality,

Be distance calculation formula based on profile, Be distance calculation formula based on the edge; Said optimum posture individuality refers to fitness

Attitude less than threshold value e is individual, threshold value e span 0.001～0.002;

Step 322, genetic operator: adopt Gauss's cross and variation operator in the genetic operator to k for population A _kIn each attitude individuality carry out cross and variation operation, population B in the middle of obtaining _kWherein

(y ' _t) ^{M, k}Operate the attitude individuality that obtains for cross and variation;

Step 323, vaccine inoculation: for middle population B _kIn each attitude individuality vaccine inoculation V all, obtain antibody population C _kWherein

(y " _t) ^{M, k}For vaccine inoculation gets the attitude individuality that V arrives; Said vaccine inoculation V be meant for attitude individual (y ' _t) ^{M, k}, (y ' _t) ^{M, k}∈ B _kIf,

Then

If ${\left(y_t^{\′}\right)}_j^{m,k}>\mathrm{Max}\left(y_j\right),$ Then ${\left(y_t^{\′}\right)}_j^{m,k}=\mathrm{Max}\left(y_j\right);$

Step 324, Immune Selection: antagonist population C _kCarry out immune detection and select, obtain interim population D with annealing _kSaid immune detection is meant antagonist population C _kIn each attitude individual (y " _t) ^{M, k}If, the attitude individuality (y " _t) ^{M, k}Fitness improve, promptly E ((y " _t) ^{M, k}, z _t)-E ((y ' _t) ^{M, k}, z _t)＞0, then with individuality (y " _t) ^{M, k}Add interim population D _kSaid annealing selects to be meant antagonist population C _kIn each attitude individual (y " _t) ^{M, k}If, the attitude individuality (y " _t) ^{M, k}Fitness reduce, promptly E ((y " _t) ^{M, k}, z _t)-E ((y ' _t) ^{M, k}, z _t)＜0, then with probability P ((y " _t) ^{M, k}) with attitude individual (y " _t) ^{M, k}Add interim population D _k, probability $P\left({\left(y_t^{\′\′}\right)}^{m,k}\right)=e^{\frac{E({\left(y_t^{\′\′}\right)}^{m,k},z_t)}{T_k}}/\underset{i=1}{\overset{M}{\mathrm{\Σ}}}{e}^{\frac{E({\left(y_t^{\′\′}\right)}^{m,k},z_t)}{T_k}};$ Wherein, E ((y " _t) ^{M, k}, z _t) be the fitness function in the step 321, T _kBe temperature control sequence, T _k=ln (T ₀/ k+1), T ₀Be initial temperature, k is the algebraically of population;

Step 325, structure new population: adopt following steps to construct male parent population A of new generation _K+1: add up interim population D _kThe individual number M ' of middle attitude; To population A _kWith interim population D _kIn to add up to the attitude of M+M ' individual, attitude identical in the individual attitude individuality of deletion M+M ' is individual; Calculate the individual fitness of residue attitude, select preceding M the highest attitude individuality of fitness to constitute male parent population A of new generation _K+1

Step 326, as iterations k during less than threshold value C, k=k+1 returns step 321; Otherwise execution in step 33, threshold value C span 80～120;

Step 33, output: calculate k for population A _kIn the individual fitness of attitude, select that the highest top n attitude of fitness is individual to constitute the candidate attitudes set

The sequence of attitude described in the step 22 is level and smooth, estimates through the time sequence information constraint attitude of introducing motion, can improve the accuracy of posture analysis, specifically may further comprise the steps:

Step 41 is gathered from candidate attitudes

Candidate attitudes of middle selection

As image z _tEstimate attitude; For one of input video structure is estimated attitude sequence H=h (1) h (2) ... h (T), wherein, h (t) ∈ [1, N] presentation video z _tThe sequence number of estimating attitude;

Step 42, computed image feature constraint f _{H (t)}: computed image z _tCandidate attitudes set

In each candidate attitudes Fitness

Then the computing method of image feature constraints are: $f_{h\left(t\right)}=E(y_t^{h\left(t\right)},z_t)/\underset{i=1}{\overset{N}{\mathrm{\Σ}}}E(y_t^i,z_t);$

Step 43 is calculated space-time restriction O _{H (t) h (t+1)}: computed image z _tEstimate attitude

With image z _T+1Estimate attitude Between space-time restriction $O_{h\left(t\right)h(t+1)}=\mathrm{Exp}(-\mathrm{Dis}(y_t^{h\left(t\right)},y_{t+1}^{h(t+1)})/{\mathrm{\σ}}^2);$ Wherein, σ is a nuclear parameter,

Be attitude And attitude

Between distance, computing method are: $\mathrm{Dis}(y_t^{h\left(t\right)},y_{t+1}^{h(t+1)})=\underset{j=0}{\overset{J}{\mathrm{\Σ}}}{w}^j{\left|\right|\mathrm{Log}{\left({\left(y_j\right)}_t^{h\left(t\right)}\right)}^{-1}{\left(y_j\right)}_{t+1}^{h(t+1)}\left|\right|}^2,$ Y wherein _jBe each dimension of attitude, j=1 ..., J, w _jWeight for each dimension.

Step 44 is set up evaluation function: based on image feature constraints f _{H (t)}With space-time restriction O _{H (t) h (t+1)}, set up and estimate attitude sequence H=h (1) h (2) ... the evaluation function of h (T) Wherein α is the weight controlled variable, and the human body attitude pace of change in α value and the input video is inversely proportional to, and span is 0.5～2.

Step 45, the attitude sequence is found the solution: adopt dynamic programming method to find the solution attitude sequence H, calculate each two field picture z _tEstimate attitude

Local pose refinement described in the step 23 at first utilizes and estimates the human body contour outline in the attitude reorientation image, adopts immunoevolution pose refinement method to optimize the body local attitude then, improves the precision of posture analysis, may further comprise the steps:

Step 51, profile reorientation: utilize and estimate attitude

Generate corresponding attitude model, and with the image z of attitude model projection to correspondence _tThe plane of delineation; The view field of attitude model is carried out the corrosion and the expansive working of image, and the zone marker that the corrosion operation is obtained is a prospect, and the zone that expansive working is obtained is labeled as background with exterior portions; According to the prospect and the background of mark, utilization figure segmentation method calculates image z _tIn human body contour outline s ' _t

Step 52, initialization of population: utilize image z _tEstimate attitude Adopt Gauss's Forecasting Methodology to obtain the attitude individuality of M accord with normal distribution

And use

The initialization population A ₀

Step 53, local pose refinement: the human body contour outline s ' that utilizes step 51 to obtain _tCalculate the individual fitness of attitude, adopt immunoevolution pose refinement method to calculate the candidate attitudes set

Step 54, attitude generates: gather from candidate attitudes The middle individual y of the highest attitude of fitness that selects _tAs image z _tThe attitude data of human motion.

The editor of attitude data described in the step 2; The employing grass that refers to is painted the attitude data of the human motion that interactive mode edit step one obtains; Generate the attitude data of new human motion, the attitude data of new human motion will be as the constraint condition of image deformation in the step 3.Grass is painted the mutual custom that interactive mode meets the user, has good usability.Specifically may further comprise the steps: the attitude data y of the human motion that obtains for the step 1 posture analysis _t, adopt grass to paint selected wherein K the key poses y of interactive mode _k, k=1 ..., K, K＜T; To each key poses y _k, adopt grass to paint the terminal articulation point reposition that interactive mode is specified the human body key position; Adopt multiple priority inverse kinematics method to find the solution and obtain new human body attitude data y ' _kWith y ' _kAdopt the attitude data y of the human motion that the hypercomplex number interpolation method obtains the step 1 posture analysis for key frame _tCarry out interpolation, generate the attitude data y ' of new human motion _t

The mobile least square image deformation method of model-driven described in the step 3; The attitude data of the new human motion that obtains with editor that refers to is as deformation controlled condition; Adopt respectively based on line segment deformation and based on the mobile least square image deformation method of reference mark deformation each frame of input video is carried out deformation; To obtain comprising the video of new human motion, specifically may further comprise the steps:

Step 71, the attitude data y of the human motion that the step 1 posture analysis is obtained _tBe plotted to image z with the skeleton mode _tThe plane of delineation, obtain the line-segment sets b in the image _{T, l}The attitude data y ' of the human motion that step 2 attitude editor is obtained _tBe plotted to image z with the skeleton mode _tThe plane of delineation, obtain the line-segment sets b ' in the image _{T, l}Wherein, l=1 ... L, L are human body skeleton segment number;

Step 72 is with line-segment sets b _{T, l}As the initial position of control line segment, with line-segment sets b ' _{T, l}As the target location of control line segment, utilize mobile least square image deformation method, to image z based on line segment deformation _tCarry out deformation, obtain the video frame images z ' after the deformation _t

Step 73 adopts the video frame images z ' after the Gaussian Background modeling method is calculated deformation _tIn human body contour outline s " _tUtilize the attitude data y ' that edits the human motion that obtains _tGenerate corresponding attitude model, with the video frame images z ' of attitude model projection after the deformation _tThe plane of delineation on; Calculate the outline m of attitude model projection _t

Step 74 is to human body contour outline s " _tThe equidistance sampling obtains sampling point set

Outline m to model projection _tThe equidistance sampling obtains sampling point set

Wherein, j=1 ..., P, P are the number of sampled point;

Step 75 is with sampling point set

As the initial position at reference mark, with sampling point set

As the target location at reference mark, utilize mobile least square image deformation method, to video frame images z ' based on reference mark deformation _tCarry out deformation, obtain comprising each two field picture z of the video of new human motion " _t

Beneficial effect: the invention discloses the human motion edit methods in a kind of video, support that the user edits to obtain comprising the video of new human motion human motion in the video.The present invention has following characteristics: 1, support the user to adopt the grass of nature to paint interactive mode and easily the human motion in the video is edited, obtain comprising the video of new human motion; 2, the stratification of immunoevolution described in the present invention posture analysis method has good convergence and global optimum's property, can be automatically, the human body attitude in the analysis video accurately; 3, the mobile least square image deformation method of model-driven described in the present invention can effectively guarantee the authenticity of human body attitude and outward appearance behind the image deformation, makes that the visual effect of the new video that generates is more natural.This paper invention is having important application in fields such as film post-production, advertisement designs.

Description of drawings

Below in conjunction with accompanying drawing and embodiment the present invention is done specifying further, above-mentioned and/or otherwise advantage of the present invention will become apparent.

Fig. 1 is the bright main process flow diagrams of we.

Fig. 2 is an immunoevolution pose refinement method flow diagram of the present invention.

Fig. 3 a, Fig. 3 b, the human body attitude data result figure that Fig. 3 c adopts middle-levelization of the present invention posture analysis method to obtain for three pin images to input video.

The attitude data of the new human motion that the attitude data of the human motion that Fig. 4 a obtains for posture analysis of the present invention, Fig. 4 b obtain for the attitude data editor.

Embodiment:

The invention discloses a kind of cartographical sketching that adopts and carry out the scheme that attitude modeling and editor and motion generate, may further comprise the steps:

Step 1, human body attitude analysis: adopt stratification pose refinement method, calculate the human body attitude of each two field picture in the input video, obtain the attitude data of human motion based on immunoevolution;

Step 2, attitude data editor: adopt grass to paint the attitude data of the human motion that interactive mode edit step one obtains, generate the attitude data of new human motion;

Step 3, image deformation: as controlled condition, adopt the mobile least square image deformation method of model-driven that each two field picture in the input video is carried out deformation with the attitude data of human motion new in the step 2, generate the video that comprises new human motion.

More particularly; The invention discloses the human motion edit methods in a kind of video; Support that the user edits to obtain comprising the video of new human motion human motion in the video; The alms giver will relate to human body attitude analysis, attitude data editor, image deformation three big gordian techniquies in fact, and treatment scheme is as shown in Figure 1.It at first is the human body attitude analysis; The present invention adopts the stratification posture analysis method based on immunoevolution, calculates the attitude data of the human motion in the input video through overall pose refinement, attitude sequence layering processing policy level and smooth, local pose refinement; Next is the attitude data editor, and the present invention adopts grass to paint the attitude data of the human motion that interactive mode editor aforementioned calculation obtains, and generates the attitude data of new human motion; The 3rd is image deformation, and the present invention adopts the mobile least square image deformation method of model-driven that each two field picture in the input video is carried out deformation, generates the video that comprises new human motion.Introduce the main flow process of each embodiment part below respectively:

1, human body attitude analysis

This paper invents the human body attitude of employing based on each two field picture of stratification posture analysis method calculating input video of immunoevolution, specifically comprises global optimization, three steps of level and smooth, the local pose refinement of attitude sequence.

1.1 overall pose refinement

Overall situation pose refinement belongs to production posture analysis method, and manikin of its explicit definition is realized the human body attitude estimation through projection and the fitness between characteristics of image of optimizing manikin.That the present invention uses is SCAPE model (document 6:ANGUELOV, D., SRINIVASAN, P.; KOLLER, D., THRUN, S.; RODGERS, J., AND DAVIS, J.2005.SCAPE:Shape Completion and Animation of People.ACM TOG; 2005,24 (3): 408-416.), adopt profile and edge feature to calculate individual fitness (document 7:Leonid Sigal, Alexandru O.Balan; Michael J.Black, HUMANEVA:Synchronized Video and Motion Capture Dataset and Baseline Algorithm for Evaluation of Articulated Human Motion.Int J Comput Vis, 2010,87:4-27); Optimization method is immunoevolution pose refinement method (document 8: Jiao Licheng, Du Haifeng, a work such as Liu Fang; Immune optimization: calculate, learn and identification, Science Press, 2006.)。Each two field picture z to input video _t, adopt immunoevolution pose refinement method to calculate a candidate attitudes set that comprises N candidate attitudes

Immunoevolution pose refinement method flow diagram is as shown in Figure 2, specifically may further comprise the steps:

Step 31, initialization: it is individual in the restriction range of vaccine V definition, to produce M attitude at random

Be designated as population A ₀Wherein,

Be an attitude individuality, m is the individual sequence number of attitude, and M is the individual total number of attitude in the population, and 0 has represented the algebraically of population, y _jBe each individual dimension of attitude, j=1 ..., J, J are the individual dimension of attitude; It is individual that said vaccine V has defined attitude

The span of each dimension, i.e. min (y _j)＜y _j＜max (y _j), min (y _j) and max (y _j) be respectively y _jThe minimum value of value and maximal value; During the present invention realizes, the dimension J=69 that attitude is individual, the individual total number M=200 of attitude in the population, min (y _j) and max (y _j) obtain through the statistics of exercise data.

Step 32, the iteration optimization population A ₀, may further comprise the steps:;

Step 321 is calculated k for population A _kIn the individual fitness of each attitude, if current population A _kIn comprise the optimum posture individuality, then out of service and output result, otherwise continue; The individual fitness of said attitude refers to the individual corresponding attitude model of attitude and the distance of characteristics of image, adopts profile and edge feature to calculate the individual fitness of attitude

Computing method are $E(y_t^{m,k},z_t)=\mathrm{Exp}(-\mathrm{\Σ}(w^s(y_t^{m,k},z_t)+w^e(y_t^{m,k},z_t))),$ Wherein,

Be that k is for population A _kIn an attitude individuality,

Be the distance calculation formula based on profile, subscript s is the abbreviation of profile English word silhouette,

Be the distance calculation formula based on the edge, subscript e is the abbreviation of edge English word edge; Said optimum posture individuality refers to fitness

Attitude less than threshold value e is individual, threshold value e=0.0015;

Step 322; Genetic operator: adopt the Gauss's cross and variation operator (document: 9:Ge JK in the genetic operator; Qiu YH, Wu CM, et al.Summary of genetic algorithms research.Application Research of Computers; 2008,25 (010): 2911-2916) to k for population A _kIn each attitude individuality carry out cross and variation operation, population B in the middle of obtaining _kWherein

(y ' _t) ^{M, k}Operate the attitude individuality that obtains for cross and variation;

Step 323, vaccine inoculation: for middle population B _kIn each attitude individuality vaccine inoculation V all, obtain antibody population C _kWherein

(y " _t) ^{M, k}For vaccine inoculation gets the attitude individuality that V arrives; Said vaccine inoculation V be meant for attitude individual (y ' _t) ^{M, k}, (y ' _t) ^{M, k}∈ B _kIf,

Then

If ${\left(y_t^{\′}\right)}_j^{m,k}>\mathrm{Max}\left(y_j\right),$ Then ${\left(y_t^{\′}\right)}_j^{m,k}=\mathrm{Max}\left(y_j\right);$

Step 324, Immune Selection: antagonist population C _kCarry out immune detection and select, obtain interim population D with annealing _kSaid immune detection is meant antagonist population C _kIn each attitude individual (y " _t) ^{M, k}If, the attitude individuality (y " _t) ^{M, k}Fitness improve, promptly E ((y " _t) ^{M, k}, z _t)-E ((y ' _t) ^{M, k}, z _t)＞0, then with individuality (y " _t) ^{M, k}Add interim population D _kSaid annealing selects to be meant antagonist population C _kIn each attitude individual (y " _t) ^{M, k}If, the attitude individuality (y " _t) ^{M, k}Fitness reduce, promptly E ((y " _t) ^{M, k}, z _t)-E ((y ' _t) ^{M, k}, z _t)＜0, then with probability P ((y " _t) ^{M, k}) with attitude individual (y " _t) ^{M, k}Add interim population D _k, probability $P\left({\left(y_t^{\″}\right)}^{m,k}\right){=e}^{\frac{E({\left(y_t^{\′\′}\right)}^{m,k},z_t)}{T_k}}/\underset{i=1}{\overset{M}{\mathrm{\Σ}}}{e}^{\frac{E({\left(y_t^{\′\′}\right)}^{m,k},z_t)}{T_k}};$ Wherein, E ((y " _t) ^{M, k}, z _t) be the fitness function in the step 321, T _kBe temperature control sequence, T _k=ln (T ₀/ k+1), T ₀Be initial temperature, k is the algebraically of population; Initial temperature T during the present invention realizes ₀=1000.

Step 325, structure new population: adopt following steps to construct male parent population A of new generation _K+1: add up interim population D _kThe individual number M ' of middle attitude; To population A _kWith interim population D _kIn to add up to the attitude of M+M ' individual, attitude identical in the individual attitude individuality of deletion M+M ' is individual; Calculate the individual fitness of residue attitude, select preceding M the highest attitude individuality of fitness to constitute male parent population A of new generation _K+1

Step 326, as iterations k during less than threshold value C, k=k+1 returns step 321; Otherwise execution in step 33; During the present invention realizes, threshold value C=100;

Step 33, output: calculate k for population A _kIn the individual fitness of attitude, select that the highest top n attitude of fitness is individual to constitute the candidate attitudes set

During the present invention realizes, the scale N=20 of candidate attitudes set.

1.2 the attitude sequence is level and smooth

The level and smooth computation process of attitude sequence is to gather from candidate attitudes

Candidate attitudes of middle selection

As image z _tEstimate attitude, construct one and estimate attitude sequence H=h (1) h (2) ... h (T) also sets up its evaluation function, adopts dynamic programming method (document 10: Pan Jingui then; Gu Tiecheng; Li Chengfa etc. translate, introduction to algorithms, China Machine Press; 2006, the 191-212 pages or leaves) optimize this and estimate the attitude sequence to obtain the best attitude sequence of estimating.The attitude sequence has smoothly been utilized the time sequence information of motion, can improve the accuracy that human body attitude is analyzed, and specifically may further comprise the steps:

Step 41 is gathered from candidate attitudes

Candidate attitudes of middle selection

As image z _tEstimate attitude; For one of input video structure is estimated attitude sequence H=h (1) h (2) ... h (T), wherein, h (t) ∈ [1, N] presentation video z _tThe sequence number of estimating attitude;

Step 42, computed image feature constraint f _{H (t)}: computed image z _tCandidate attitudes set

In each candidate attitudes

Fitness

Then the computing method of image feature constraints are: $f_{h\left(t\right)}=E(y_t^{h\left(t\right)},z_t)/\underset{\text{I=1}}{\overset{N}{\mathrm{\Σ}}}\text{E}(y_t^i,z_t);$

Step 43 is calculated space-time restriction O _{H (t) h (t+1)}: computed image z _tEstimate attitude With image z _T+1Estimate attitude

Between space-time restriction $O_{h\left(t\right)h(t+1)}=\mathrm{Exp}(-\mathrm{Dis}(y_t^{h\left(t\right)},y_{t+1}^{h(t+1)})/{\mathrm{\σ}}^2);$ Wherein, σ is a nuclear parameter,

Be attitude And attitude

Between distance, computing method are: $\mathrm{Dis}(y_t^{h\left(t\right)},y_{t+1}^{h(t+1)})=\underset{j=0}{\overset{J}{\mathrm{\Σ}}}{w}_j{\left|\right|\mathrm{Log}{\left({\left(y_j\right)}_t^{h\left(t\right)}\right)}^{-1}{\left(y_j\right)}_{t+1}^{h(t+1)}\left|\right|}^2,$ Y wherein _jBe each dimension of attitude, j=1 ..., J, w _jWeight for each dimension; In the present invention realized, nuclear parameter σ=10 were provided with different weight w to different articulation points _j, wherein crucial articulation point comprises that it is 1 that root joint, knee joint, elbow joint, trans-articular, the corresponding dimension of a shoulder joint are established weight, it is 0 that weight is established in other joints;

Step 44 is set up evaluation function: based on image feature constraints f _{H (t)}With space-time restriction O _{H (t) h (t+1)}, set up and estimate attitude sequence H=h (1) h (2) ... the evaluation function of h (T)

Wherein α is the weight controlled variable, and value is relevant with human body attitude pace of change in the input video, for the video of human motion rapid speed; As run etc. and to get a=2; For the slower video of movement velocity, get a=0.5 as walk waiting, and other most of videos are got a=1 and are got final product.In the present invention realizes, get α=0.5;

Step 45, the attitude sequence is found the solution: adopt dynamic programming method to find the solution attitude sequence H, calculate each two field picture z _tEstimate attitude

1.3 local pose refinement

Local pose refinement basic thought is through improving the precision that characteristics of image calculates; Adopt immunoevolution pose refinement method to optimize the local attitude of human body; Its treatment scheme is: based on the attitude model projection marking video two field picture of estimating attitude; Employing figure segmentation method calculates accurate human body contour outline, utilizes immunoevolution pose refinement method to realize local optimum, specifically may further comprise the steps:

Step 51, profile reorientation: utilize and estimate attitude

Generate corresponding attitude model, and with the image z of attitude model projection to correspondence _tThe plane of delineation; The view field of attitude model is carried out the corrosion and the expansive working of image, and the zone marker that the corrosion operation is obtained is a prospect, and the zone that expansive working is obtained is labeled as background with exterior portions; According to the prospect and the background of mark, utilization figure segmentation method calculates image z _tIn human body contour outline s ' _t

Step 52, initialization of population: utilize image z _tEstimate attitude

Adopt Gauss's Forecasting Methodology to obtain the attitude individuality of M accord with normal distribution

And use

The initialization population A ₀

Step 53, local pose refinement: the human body contour outline s ' that utilizes step 51 to obtain _tCalculate the individual fitness of attitude, adopt immunoevolution pose refinement method to calculate the candidate attitudes set

Step 54, attitude generates: gather from candidate attitudes

The middle individual y of the highest attitude of fitness that selects _tAs image z _tThe attitude data of human motion.

Fig. 3 a, Fig. 3 b, the human body attitude data result figure that Fig. 3 c adopts middle-levelization of the present invention posture analysis method to obtain for three pin images to input video.

2, attitude data editor

The present invention paints interactive mode with grass and is applied among the attitude data editor, has improved user's operability.Attitude editor's flow process is at first by the key poses of the attitude data of specifying human motion; Adopt grass to paint the position of the human body target articulation point of interactive mode designated key attitude then; And then adopt the inverse kinematics method to find the solution new human body attitude; At last original attitude data is carried out the attitude data that interpolation obtains new human motion, specifically may further comprise the steps:

Step 61, the attitude data y of the human motion that obtains for the step 1 posture analysis _t, adopt grass to paint selected wherein K the key poses y of interactive mode _k, k=1 ..., K, K＜T; In the present invention, key poses number K generally gets.

Step 62 is to each key poses y _k, adopt grass to paint the terminal articulation point reposition that interactive mode is specified the human body key position;

Step 63; Adopt multiple priority inverse kinematics method (document: 11:Paolo Baerlocher; Ronan Boulic, An inverse kinematic architecture enforcing an arbitrary number of strict priority levels, The Visual Computer; 2004,20 (6): 402-417) find the solution and obtain new human body attitude data y ' _k

Step 64 is with y ' _kAdopt the attitude data y of the human motion that the hypercomplex number interpolation method obtains the step 1 posture analysis for key frame _tCarry out interpolation, generate the attitude data y ' of new human motion _t

3, image deformation

The thinking of image deformation be the attitude data of the new human motion that obtains with editor as deformation controlled condition, adopt successively based on line segment deformation and based on mobile least square image deformation method (document 12:M ¨ ULLER, the M. of reference mark deformation; HEIDELBERGER, B., TESCHNER; M., AND GROSS, M.Meshless deformations based on shape matching.ACM TOG; 2005; 24 (3): 471-478.) each two field picture to input video carries out deformation, to obtain comprising the video of new human motion, specifically may further comprise the steps:

Step 71, the attitude data y of the human motion that the step 1 posture analysis is obtained _tBe plotted to image z with the skeleton mode _tThe plane of delineation, obtain the line-segment sets b in the image _{T, l}The attitude data y ' of the human motion that step 2 attitude editor is obtained _tBe plotted to image z with the skeleton mode _tThe plane of delineation, obtain the line-segment sets b ' in the image _{T, l}Wherein, l=1 ... L, L are human body skeleton segment number; During the present invention realizes, the skeleton segment number L=25 of human body.

Step 72 is with line-segment sets b _{T, l}As the initial position of control line segment, with line-segment sets b ' _{T, l}As the target location of control line segment, utilize mobile least square image deformation method, to image z based on line segment deformation _tCarry out deformation, obtain the video frame images z ' after the deformation _t

Step 73; Adopt Gaussian Background modeling method (document 13:C.Stauffer and W.E.L. Grimson; Adaptive background mixture models for real-time tracking; Proc.IEEE CVPR, June 1999:246-252) calculates video frame images z ' after the deformation _tIn human body contour outline s " _tUtilize the attitude data y ' that edits the human motion that obtains _tGenerate corresponding attitude model, with the video frame images z ' of attitude model projection after the deformation _tThe plane of delineation on; Calculate the outline m of attitude model projection _t

Step 74 is to human body contour outline s " _tThe equidistance sampling obtains sampling point set Outline m to model projection _tThe equidistance sampling obtains sampling point set

Wherein, j=1 ..., P, P are the number of sampled point; During the present invention realizes, sampled point number P=150.

Step 75 is with sampling point set

As the initial position at reference mark, with sampling point set

As the target location at reference mark, utilize mobile least square image deformation method, to video frame images z ' based on reference mark deformation _tCarry out deformation, obtain comprising each two field picture z of the video of new human motion " _t

The invention provides the human motion edit methods in a kind of video; The method and the approach of concrete this technical scheme of realization are a lot, and the above only is a preferred implementation of the present invention, should be understood that; For those skilled in the art; Under the prerequisite that does not break away from the principle of the invention, can also make some improvement and retouching, these improvement and retouching also should be regarded as protection scope of the present invention.The all available prior art of each ingredient not clear and definite in the present embodiment realizes.

Claims (7)

1. the human motion edit methods in the video is characterized in that, may further comprise the steps:

Step 1, human body attitude analysis: adopt stratification pose refinement method, calculate the human body attitude of each two field picture in the input video, obtain the attitude data of human motion based on immunoevolution;

Step 2, attitude data editor: adopt grass to paint the attitude data of the human motion that interactive mode edit step one obtains, generate the attitude data of new human motion;

Step 3, image deformation: as controlled condition, adopt the mobile least square image deformation method of model-driven that each two field picture in the input video is carried out deformation with the attitude data of human motion new in the step 2, generate the video that comprises new human motion.

2. the human motion edit methods in a kind of video according to claim 1 is characterized in that step 1 may further comprise the steps:

Step 21, overall pose refinement: to each two field picture z of input video _t, adopt immunoevolution pose refinement method to calculate a candidate attitudes set that comprises N candidate attitudes

Wherein, Be a candidate attitudes, i is the sequence number of candidate attitudes, i=1 ..., N, N are total number of candidate attitudes, t=1 ..., T, T are the totalframes of input video;

Step 22, the attitude sequence is level and smooth: gather from candidate attitudes

Candidate attitudes of middle selection is as image z _tEstimate attitude

Wherein, attitude is estimated in h (t) expression

Gather in candidate attitudes

In sequence number, h (t) ∈ [1, N];

Step 23, local pose refinement: utilize and estimate attitude

Reorientation image z _tIn human body contour outline; Adopt immunoevolution pose refinement method to calculate the local attitude of human body, obtain image z _tThe attitude data y of human motion _t

3. the stratification posture analysis method based on immunoevolution according to claim 2 is characterized in that step 21 may further comprise the steps:

Step 31, initialization: it is individual in the restriction range of vaccine V definition, to produce M attitude at random

Be designated as population A ₀Wherein,

Be an attitude individuality, m is the individual sequence number of attitude, and M is the individual total number of attitude in the population, and 0 has represented the algebraically of population, y _jBe each individual dimension of attitude, j=1 ..., J, J are the individual dimension of attitude; It is individual that said vaccine V has defined attitude

The span of each dimension, i.e. min (y _j)＜y _j＜max (y _j), min (y _j) and max (y _j) be respectively y _jThe minimum value of value and maximal value;

Step 32, the iteration optimization population A ₀, may further comprise the steps:

Step 321 is calculated k for population A _kIn the individual fitness of each attitude, if current population A _kIn comprise the optimum posture individuality, then out of service and output result, otherwise continue; The individual fitness of said attitude refers to the individual corresponding attitude model of attitude and the distance of characteristics of image, is expressed as Computing method are $E(y_t^{m,k},z_t)=\mathrm{Exp}(-\mathrm{\Σ}(w^s(y_t^{m,k},z_t)+w^e(y_t^{m,k},z_t))),$ Wherein,

Be that k is for population A _kIn an attitude individuality,

Be distance calculation formula based on profile, Be distance calculation formula based on the edge; Said optimum posture individuality refers to fitness

Attitude less than threshold value e is individual, threshold value e span 0.001～0.002;

Step 322, genetic operator: to k for population A _kIn each attitude individuality carry out cross and variation operation, population B in the middle of obtaining _kWherein

(y ' _t) ^{M, k}Operate the attitude individuality that obtains for cross and variation;

Step 323, vaccine inoculation: for middle population B _kIn each attitude individuality vaccine inoculation V all, obtain antibody population C _kWherein

(y " _t) ^{M, k}For vaccine inoculation gets the attitude individuality that V arrives;

Said vaccine inoculation V be meant for attitude individual (y ' _t) ^{M, k}, (y ' _t) ^{M, k}∈ B _kIf,

Then ${\left(y_t^{\′}\right)}_j^{m,k}=\mathrm{Min}\left(y_j\right);$ If ${\left(y_t^{\′}\right)}_j^{m,k}>\mathrm{Max}\left(y_j\right),$ Then ${\left(y_t^{\′}\right)}_j^{m,k}=\mathrm{Max}\left(y_j\right);$

Step 324, Immune Selection: antagonist population C _kCarry out immune detection and select, obtain interim population D with annealing _kSaid immune detection is meant antagonist population C _kIn each attitude individual (y " _t) ^{M, k}If, the attitude individuality (y " _t) ^{M, k}Fitness improve, promptly E ((y " _t) ^{M, k}, z _t)-E ((y ' _t) ^{M, k}, z _t)＞0, then with individuality (y " _t) ^{M, k}Add interim population D _kSaid annealing selects to be meant antagonist population C _kIn each attitude individual (y " _t) ^{M, k}If, the attitude individuality (y " _t) ^{M, k}Fitness reduce, promptly E ((y " _t) ^{M, k}, z _t)-E ((y ' _t) ^{M, k}, z _t)＜0, then with probability P ((y " _t) ^{M, k}) with attitude individual (y " _t) ^{M, k}Add interim population D _k, probability $P\left({\left(y_t^{\″}\right)}^{m,k}\right)=e^{\frac{E({\left(y_t^{\′\′}\right)}^{m,k},z_t)}{T_k}}/\underset{i=1}{\overset{M}{\mathrm{\Σ}}}{e}^{\frac{E({\left(y_t^{\′\′}\right)}^{m,k},z_t)}{T_k}};$ Wherein, E ((y " _t) ^{M, k}, z _t) be the fitness function in the step 321, T _kBe temperature control sequence, T _k=ln (T ₀/ k+1), T ₀Be initial temperature, k is the algebraically of population;

Step 325, structure new population: adopt following steps to construct male parent population A of new generation _K+1: add up interim population D _kThe individual number M ' of middle attitude; To population A _kWith interim population D _kIn to add up to the attitude of M+M ' individual, attitude identical in the individual attitude individuality of deletion M+M ' is individual; Calculate the individual fitness of residue attitude, select preceding M the highest attitude individuality of fitness to constitute male parent population A of new generation _K+1

Step 326, as iterations k during less than threshold value C, k=k+1 returns step 321; Otherwise execution in step 33, threshold value C span 80～120;

Step 33, output: calculate k for population A _kIn the individual fitness of attitude, select that the highest top n attitude of fitness is individual to constitute the candidate attitudes set

4. the stratification posture analysis method based on immunoevolution according to claim 3 is characterized in that step 22 may further comprise the steps:

Step 41 is gathered from candidate attitudes Candidate attitudes of middle selection

As image z _tEstimate attitude; For one of input video structure is estimated attitude sequence H=h (1) h (2) ... h (T), wherein, h (t) ∈ [1, N] presentation video z _tThe sequence number of estimating attitude;

Step 42, computed image feature constraint f _{H (t)}: computed image z _tCandidate attitudes set

In each candidate attitudes

Fitness

Then the computing method of image feature constraints are: $f_{h\left(t\right)}=E(y_t^{h\left(t\right)},z_t)/\underset{i=1}{\overset{N}{\mathrm{\Σ}}}E(y_t^i,z_t);$

Step 43 is calculated space-time restriction O _{H (t) h (t+1)}: computed image z _tEstimate attitude

With image z _T+1Estimate attitude

Between space-time restriction $O_{h\left(t\right)h(t+1)}=\mathrm{Exp}(-\mathrm{Dis}(y_t^{h\left(t\right)},y_{t+1}^{h(t+1)})/{\mathrm{\σ}}^2);$

Wherein, σ is a nuclear parameter, span 1～10; Be attitude And attitude

Between distance, computing method are: $\mathrm{Dis}(y_t^{h\left(t\right)},y_{t+1}^{h(t+1)})=\underset{j=0}{\overset{J}{\mathrm{\Σ}}}{w}_j{\left|\right|\mathrm{Log}{\left({\left(y_j\right)}_t^{h\left(t\right)}\right)}^{-1}{\left(y_j\right)}_{t+1}^{h(t+1)}\left|\right|}^2,$ Y wherein _jBe each dimension of attitude, j=1 ..., J, w _jWeight for each dimension;

Step 44 is set up evaluation function: based on image feature constraints f _{H (t)}With space-time restriction O _{H (t) h (t+1)}, set up and estimate attitude sequence H=h (1) h (2) ... the evaluation function of h (T)

Wherein α is the weight controlled variable, α span 0.5～2;

Step 45, the attitude sequence is found the solution: find the solution attitude sequence H, calculate each two field picture z _tEstimate attitude

5. the stratification posture analysis method based on immunoevolution according to claim 4 is characterized in that step 23 may further comprise the steps:

Step 51, profile reorientation: utilize and estimate attitude

Generate corresponding attitude model, and with the image z of attitude model projection to correspondence _tThe plane of delineation; The view field of attitude model is carried out the corrosion and the expansive working of image, and the zone marker that the corrosion operation is obtained is a prospect, and the zone that expansive working is obtained is labeled as background with exterior portions; According to the prospect and the background of mark, computed image z _tIn human body contour outline s ' _t

Step 52, initialization of population: utilize image z _tEstimate attitude Adopt Gauss's Forecasting Methodology to obtain the attitude individuality of M accord with normal distribution

And use

The initialization population A ₀

Step 53, local pose refinement: the human body contour outline s ' that utilizes step 51 to obtain _tCalculate the individual fitness of attitude, adopt immunoevolution pose refinement method to calculate the candidate attitudes set

Step 54, attitude generates: gather from candidate attitudes The middle individual y of the highest attitude of fitness that selects _tAs image z _tThe attitude data of human motion.

6. the human motion edit methods in a kind of video according to claim 5 is characterized in that step 2 comprises: the attitude data y of the human motion that the step 1 posture analysis is obtained _k, adopt grass to paint selected wherein K the key poses y of interactive mode _k, k=1 ..., K, K＜T; To each key poses y _k, adopt grass to paint the terminal articulation point reposition that interactive mode is specified the human body key position; Find the solution and obtain new human body attitude data y ' _kWith y ' _kAdopt the attitude data y of the human motion that the hypercomplex number interpolation method obtains the step 1 posture analysis for key frame _tCarry out interpolation, generate the attitude data y ' of new human motion _t

7. the human motion edit methods in a kind of video according to claim 6 is characterized in that step 3 may further comprise the steps:

Step 71, the attitude data y of the human motion that the step 1 posture analysis is obtained _tBe plotted to image z with the skeleton mode _tThe plane of delineation, obtain the line-segment sets b in the image _{T, l}The attitude data y ' of the human motion that step 2 attitude editor is obtained _tBe plotted to image z with the skeleton mode _tThe plane of delineation, obtain the line-segment sets b ' in the image _{T, l}Wherein, l=1 ... L, L are human body skeleton segment number;

Step 72 is with line-segment sets b _{T, l}As the initial position of control line segment, with line-segment sets b ' _{T, l}As the target location of control line segment, utilize mobile least square image deformation method, to image z based on line segment deformation _tCarry out deformation, obtain the video frame images z ' after the deformation _t

Step 73, the video frame images z ' after the calculating deformation _tIn human body contour outline s " _tUtilize the attitude data y ' that edits the human motion that obtains _tGenerate corresponding attitude model, with the video frame images z ' of attitude model projection after the deformation _tThe plane of delineation on; Calculate the outline m of attitude model projection _t

Step 74 is to human body contour outline s " _tThe equidistance sampling obtains sampling point set

Outline m to model projection _tThe equidistance sampling obtains sampling point set

Wherein, j=1 ..., P, P are the number of sampled point;

Step 75 is with sampling point set As the initial position at reference mark, with sampling point set

As the target location at reference mark, utilize mobile least square image deformation method, to video frame images z ' based on reference mark deformation _tCarry out deformation, obtain comprising each two field picture z of the video of new human motion " _t

Images