CN103533237A - Method for extracting video key frame from video - Google Patents

Abstract

The invention relates to a method for extracting a video key frame from a video and belongs to the technical field of image processing. According to the method for extracting the video key frame from the video, an operator carries out video shooting on interested scenes by a device. In the shooting process, video frames, accelerated speed information, azimuth information and scale information of the video are recorded synchronously by the device. After the shooting is finished, weight of each frame of the video frame is calculated directly according to the accelerated speed information, the azimuth information and the scale information. Finally, an expected video key frame is extracted according to the weight and the expected key frame quantity. The method for extracting the video key frame from the video can be used for accurately extracting the video key frame from the video by using a smaller calculation amount.

Description

A kind of method of extracting key frame of video from video

Technical field

The present invention relates to a kind of method of extracting key frame of video from video, belong to technical field of image processing.

Background technology

Along with the increase of hand-hold type capture apparatus (as mobile phone, digital camera and hand-hold type video camera) quantity in recent years, the number of videos that personal user uses portable equipment to take also rolls up.Various video captures application on smart mobile phone, as Instagram etc., has also stimulated the propagation of these videos, and Instagram, in 24 hours of reaching the standard grade, just has 5,000,000 videos to be uploaded.

So many application causes to produce a large amount of videos every day.The places different from Word message are, video information cannot directly be retrieved, and therefore from a large amount of videos, finding Useful Information is the work extremely taking time and effort.The method of taking is at present mainly to rely on the content of manually checking video, and it is marked, and this is the unusual method of poor efficiency undoubtedly.

Therefore the extraction of key frame becomes very significant work.The Key-frame Extraction Algorithm existing at present mainly contains following several: 1) according to regular time interval extract key frame, 2) calculate the difference of adjacent a few frame in color (or gray scale), determine whether it is key frame.3) method based on motion analysis.

It is the most directly perceived according to Fixed Time Interval, extracting key frame, calculates the simplest way, but the shortcoming of this method also clearly, and it is equally distributed that key frame may not be certain.Therefore to be applicable to content simple for these class methods, the video that the time is shorter.

The method of calculating adjacent frame difference is more reasonable, but more difficult choosing of threshold value, and threshold value value is too small may cause selecting too many key frame, and value is excessive likely can cause leaking choosing.And this method is calculated upper more complicated.

Main method based on motion analysis is mainly to calculate the amount of exercise in camera lens by optical flow analysis, in the place of amount of exercise minimum, chooses key frame, and its shortcoming is also that amount of calculation is excessive.

Summary of the invention

The present invention seeks to propose a kind of method of extracting key frame of video from video, to solve the shortcoming of existing extraction method of key frame, in content, the behavior of camera zoom and the video of taking in conjunction with camera each time the acceleration information of engraving device and filming apparatus towards etc. information, intention according to video capture person, calculates key frame.

The method of extracting key frame of video from video that the present invention proposes, comprises the following steps:

(1) use video capture device, floor scene, obtains video, sets total T frame frame of video in video, and records each and take the convergent-divergent yardstick information of filming apparatus camera constantly;

(2) use the frequency identical with capture video, record respectively each and take moment filming apparatus linear acceleration information along x, y, z axle in rectangular coordinate system;

(3) use the frequency identical with capture video, with aspect sensor, record each shooting and constantly take and be equipped in the azimuth information in above-mentioned rectangular coordinate system;

(4) according to the azimuth information of above-mentioned record, linear acceleration information and yardstick information, from video, extract key frame, comprise the following steps:

(4-1) characteristic information of the device while taking k frame frame of video in extraction video, comprising: k frame frame of video is taken filming apparatus azimuth information o constantly _k=[o _x,k, o _y,k, o _z,k] ^t, o wherein _x,krepresent the k frame frame of video shooting rolling angle of filming apparatus constantly, install the angle of minor face and horizontal plane, o _y,krepresent the k frame frame of video shooting luffing angle of filming apparatus constantly, install the angle of long limit and horizontal plane, o _z,krepresent the k frame frame of video shooting angle of vacillating now to the left, now to the right of filming apparatus constantly, the direction that immediate shipment top set end points to and the angle of direct north; K frame frame of video is taken the acceleration information a of filming apparatus constantly _k=[a _x,k, a _y,k, a _z,k] ^t, a wherein _x,k, a _y,k, a _z,kfor device is respectively at the x of rectangular coordinate system, y, the acceleration on z axle, yardstick information s _kbe illustrated in the convergent-divergent yardstick of camera while taking k frame;

(4-2) adopt discrete cosine transform, video obtained above is carried out to feature information extraction, obtain the frame of video characteristic information f of k frame frame of video in video _k;

(4-3) repeating step (4-1) and step (4-2), obtain the filming apparatus azimuth information of each frame frame of video in above-mentioned video, the convergent-divergent yardstick of the acceleration information of filming apparatus, camera and frame of video characteristic information;

(4-5) calculate the acceleration weights omega of each frame frame of video in video _ak: ω _ak=exp (λ ₁|| a _k|| ₂), λ wherein ₁for acceleration regulates parameter, || a _k|| ₂represent acceleration information a _ktwo norms of vector, λ ₁span can determine according to the order of magnitude of acceleration, generally 0.1～1;

(4-6) calculate each frame frame of video in video yardstick weights omega _sk: ω _sk=exp (λ ₂s _k), λ wherein ₂for yardstick regulates parameter, λ ₂span be: 0.5～1;

(4-7) calculate total weights omega of each frame frame of video in video _k: ω _k=ω _akω _sk;

(4-8) adopt K mean algorithm, all frame of video in above-mentioned video are taken to filming apparatus azimuth information constantly and carry out cluster, obtain C cluster centre, the parameter of C for choosing according to information such as video lengths, the span of C is: 1～T, T is the frame number of all frame of video in video, and all frame of video are referred in the class under the immediate cluster centre of azimuth information with corresponding filming apparatus;

(4-9) set up an optimization aim function as follows:

$J=\underset{k=1}{\overset{T}{\mathrm{\Σ}}}\underset{j=1}{\overset{C}{\mathrm{\Σ}}}{\mathrm{\ω}}_k{\mathrm{\μ}}_{\mathrm{kj}}^2{\left|\right|o_k-{\mathrm{\υ}}_j^{\left(p\right)}\left|\right||}_2^2,$

Constraints is: $\underset{j=1}{\overset{C}{\mathrm{\Σ}}}{\mathrm{\μ}}_{\mathrm{kj}}=\mathrm{1,0}\≤{\mathrm{\μ}}_{\mathrm{kj}}\≤1$

The sequence number that wherein k is frame of video, the classification that j is cluster centre, j ∈ [1, C], μ _kjparameter to be solved, υ _jcluster centre, p is current iteration number of times;

(4-10), during initialization, establish p=0,

the initial value vector that is j cluster centre;

(4-11) calculate μ _kj: ${\mathrm{\μ}}_{\mathrm{kj}}=1/{\left|\right|o_k-{\mathrm{\υ}}_j^{\left(p\right)}\left|\right|}_2^2$

(4-12), according to above-mentioned result of calculation, upgrade μ _kjvalue, calculate μ _kj:

${\mathrm{\μ}}_{\mathrm{kj}}={\mathrm{\μ}}_{\mathrm{kj}}/\underset{j=1}{\overset{C}{\mathrm{\Σ}}}{\mathrm{\μ}}_{\mathrm{kj}}$

(4-13) μ calculating according to step (4-12) _kj, calculate

(4-14) set an iteration and stop threshold epsilon, if

make p=p+1, and return to step (4-11), if

carry out step (4-15), the span of ε is: 0.01～0.001;

(4-15) by following formula, obtain an initial key frame set K={t ₁, t ₁..., t _c}: $t_j=\underset{k\∈[1,T]}{\arg \max }{\mathrm{\μ}}_{\mathrm{kj}},$ J ∈ [1, C] wherein;

(4-16) calculate the similarity of the frame of video characteristic information of any two width frame of video in above-mentioned initial key frame set K $S_{t_i,t_j}={\left|\right|f_{t_i}-f_{t_j}\left|\right|}_2^2,$ I wherein, j ∈ [1, C];

(4-17) set a similarity threshold, any two frames in the initial key frame set K that traversal step (4-16) calculates, calculate the similarity of the frame of video characteristic information of any two frames

compare with similarity threshold, if

and

from above-mentioned initial key frame set K, delete t _j; If and from above-mentioned initial key frame set K, delete t _i; If

in above-mentioned initial key frame set K, retain t _iand t _j, repeating this step, the set K obtaining is key frame of video, and the span of δ is: 0.2～0.3.

The method of extracting key frame of video from video that the present invention proposes, its advantage is, can be in user's capture video, the motion of records photographing video timer, towards and the variation of focal length, and according to the information of filming apparatus, infer and the intention of user when taking, for example the acceleration of device is less, and when keeping certain numerical value to continue for some time, can think that user is taking a certain scene targetedly, also this scene is crucial to user.Hypothesis can be extracted key frame of video more accurately by less amount of calculation from video accordingly.

Accompanying drawing explanation

Fig. 1 is the FB(flow block) of the method for extracting key frame of video from video that proposes of the present invention.

Embodiment

The method of extracting key frame of video from video that the present invention proposes, its FB(flow block) as shown in Figure 1, comprises the following steps:

(1) use video capture device, floor scene, obtains video, sets total T frame frame of video in video, and records each and take the convergent-divergent yardstick information of filming apparatus camera constantly;

(2) use the frequency identical with capture video, record respectively each and take moment filming apparatus linear acceleration information along x, y, z axle in rectangular coordinate system;

(3) use the frequency identical with capture video, with aspect sensor, record each shooting and constantly take and be equipped in the azimuth information in above-mentioned rectangular coordinate system;

(4) according to the azimuth information of above-mentioned record, linear acceleration information and yardstick information, from video, extract key frame, comprise the following steps:

(4-1) characteristic information of the device while taking k frame frame of video in extraction video, comprising: k frame frame of video is taken filming apparatus azimuth information o constantly _k=[o _x,k, o _y,k, o _z,k] ^t, o wherein _x,krepresent the k frame frame of video shooting rolling angle of filming apparatus constantly, install the angle of minor face and horizontal plane, o _y,krepresent the k frame frame of video shooting luffing angle of filming apparatus constantly, install the angle of long limit and horizontal plane, o _z,krepresent the k frame frame of video shooting angle of vacillating now to the left, now to the right of filming apparatus constantly, the direction that immediate shipment top set end points to and the angle of direct north; K frame frame of video is taken the acceleration information a of filming apparatus constantly _k=[a _x,k, a _y,k, a _z,k] ^t, a wherein _x,k, a _y,k, a _z,kfor device is respectively at the x of rectangular coordinate system, y, the acceleration on z axle, yardstick information s _kbe illustrated in the convergent-divergent yardstick of camera while taking k frame;

(4-2) adopt discrete cosine transform, video obtained above is carried out to feature information extraction, obtain the frame of video characteristic information f of k frame frame of video in video _k;

(4-3) repeating step (4-1) and step (4-2), obtain the filming apparatus azimuth information of each frame frame of video in above-mentioned video, the convergent-divergent yardstick of the acceleration information of filming apparatus, camera and frame of video characteristic information;

(4-5) calculate the acceleration weights omega of each frame frame of video in video _ak: ω _ak=exp (λ ₁|| a _k|| ₂), λ wherein ₁for acceleration regulates parameter, || a _k|| ₂represent acceleration information a _ktwo norms of vector, λ ₁span can determine according to the order of magnitude of acceleration, generally 0.1～1;

(4-6) calculate each frame frame of video in video yardstick weights omega _sk: ω _sk=exp (λ ₂s _k), λ wherein ₂for yardstick regulates parameter, λ ₂span be: 0.5～1;

(4-7) calculate total weights omega of each frame frame of video in video _k: ω _k=ω _akω _sk;

(4-8) adopt K mean algorithm, all frame of video in above-mentioned video are taken to filming apparatus azimuth information constantly and carry out cluster, obtain C cluster centre, the parameter of C for choosing according to information such as video lengths, the span of C is: 1～T, T is the frame number of all frame of video in video, and all frame of video are referred in the class under the immediate cluster centre of azimuth information with corresponding filming apparatus;

(4-9) set up an optimization aim function as follows:

$J=\underset{k=1}{\overset{T}{\mathrm{\Σ}}}\underset{j=1}{\overset{C}{\mathrm{\Σ}}}{\mathrm{\ω}}_k{\mathrm{\μ}}_{\mathrm{kj}}^2{\left|\right|o_k-{\mathrm{\υ}}_j^{\left(p\right)}\left|\right||}_2^2,$

Constraints is: $\underset{j=1}{\overset{C}{\mathrm{\Σ}}}{\mathrm{\μ}}_{\mathrm{kj}}=\mathrm{1,0}\≤{\mathrm{\μ}}_{\mathrm{kj}}\≤1$

The sequence number that wherein k is frame of video, the classification that j is cluster centre, j ∈ [1, C], μ _kjparameter to be solved, υ _jcluster centre, p is current iteration number of times;

(4-10), during initialization, establish p=0,

the initial value vector that is j cluster centre;

(4-11) calculate μ _kj: ${\mathrm{\μ}}_{\mathrm{kj}}=1/{\left|\right|o_k-{\mathrm{\υ}}_j^{\left(p\right)}\left|\right|}_2^2$

(4-12), according to above-mentioned result of calculation, upgrade μ _kjvalue, calculate μ _kj:

${\mathrm{\μ}}_{\mathrm{kj}}={\mathrm{\μ}}_{\mathrm{kj}}/\underset{j=1}{\overset{C}{\mathrm{\Σ}}}{\mathrm{\μ}}_{\mathrm{kj}}$

(4-13) μ calculating according to step (4-12) _kj, calculate

(4-14) set an iteration and stop threshold epsilon, if

make p=p+1, and return to step (4-11), if

carry out step (4-15), the span of ε is: 0.01～0.001;

(4-15) by following formula, obtain an initial key frame set K={t ₁, t ₁..., t _c}: $t_j=\underset{k\∈[1,T]}{\arg \max }{\mathrm{\μ}}_{\mathrm{kj}},$ J ∈ [1, C] wherein;

(4-16) calculate the similarity of the frame of video characteristic information of any two width frame of video in above-mentioned initial key frame set K $S_{t_i,t_j}={\left|\right|f_{t_i}-f_{t_j}\left|\right|}_2^2,$ I wherein, j ∈ [1, C];

(4-17) set a similarity threshold, any two frames in the initial key frame set K that traversal step (4-16) calculates, calculate the similarity of the frame of video characteristic information of any two frames

compare with similarity threshold, if

and

from above-mentioned initial key frame set K, delete t _j; If

and

from above-mentioned initial key frame set K, delete t _i; If

in above-mentioned initial key frame set K, retain t _iand t _j, repeating this step, the set K obtaining is key frame of video, and the span of δ is: 0.2～0.3.

The method of extracting key frame of video from video of the present invention, operating personnel carry out video capture by device to interested scene.In shooting process, install frame of video, acceleration information, azimuth information and the yardstick information of synchronous recording video.After completing, shooting directly utilize acceleration information, azimuth information and yardstick information to calculate its weight to each frame frame of video.Finally according to the number of key frames of weight and expectation, extract the key frame of video of expectation.

Claims (1)

1. from video, extract a method for key frame of video, it is characterized in that, the method comprises the following steps:

(1) use video capture device, floor scene, obtains video, sets total T frame frame of video in video, and records each and take the convergent-divergent yardstick information of filming apparatus camera constantly;

(2) use the frequency identical with capture video, record respectively each and take moment filming apparatus linear acceleration information along x, y, z axle in rectangular coordinate system;

(3) use the frequency identical with capture video, with aspect sensor, record each shooting and constantly take and be equipped in the azimuth information in above-mentioned rectangular coordinate system;

(4) according to the azimuth information of above-mentioned record, linear acceleration information and yardstick information, from video, extract key frame, comprise the following steps:

(4-1) characteristic information of the device while taking k frame frame of video in extraction video, comprising: k frame frame of video is taken filming apparatus azimuth information o constantly _k=[o _x,k, o _y,k, o _z,k] ^t, o wherein _x,krepresent the k frame frame of video shooting rolling angle of filming apparatus constantly, install the angle of minor face and horizontal plane, o _y,krepresent the k frame frame of video shooting luffing angle of filming apparatus constantly, install the angle of long limit and horizontal plane, o _z,krepresent the k frame frame of video shooting angle of vacillating now to the left, now to the right of filming apparatus constantly, the direction that immediate shipment top set end points to and the angle of direct north; K frame frame of video is taken the acceleration information α of filming apparatus constantly _k=[a _x,k, a _y,k, a _z,k] ^t, a wherein _x,k, a _y,k, a _z,kfor device is respectively at the x of rectangular coordinate system, y, the acceleration on z axle, yardstick information s _kbe illustrated in the convergent-divergent yardstick of camera while taking k frame;

(4-2) adopt discrete cosine transform, video obtained above is carried out to feature information extraction, obtain the frame of video characteristic information f of k frame frame of video in video _k;

(4-3) repeating step (4-1) and step (4-2), obtain the filming apparatus azimuth information of each frame frame of video in above-mentioned video, the convergent-divergent yardstick of the acceleration information of filming apparatus, camera and frame of video characteristic information;

(4-5) calculate the acceleration weights omega of each frame frame of video in video _ak: ω _ak=exp (λ ₁|| a _k|| ₂), λ wherein ₁for acceleration regulates parameter, || a _k|| ₂represent acceleration information a _ktwo norms of vector, λ ₁span can determine according to the order of magnitude of acceleration, generally 0.1～1;

(4-6) calculate each frame frame of video in video yardstick weights omega _sk: ω _sk=exp (λ ₂s _k), λ wherein ₂for yardstick regulates parameter, λ ₂span be: 0.5～1;

(4-7) calculate total weights omega of each frame frame of video in video _k: ω _k=ω _akω _sk;

(4-8) adopt K mean algorithm, all frame of video in above-mentioned video are taken to filming apparatus azimuth information constantly and carry out cluster, obtain C cluster centre, the parameter of C for choosing according to information such as video lengths, the span of C is: 1～T, T is the frame number of all frame of video in video, and all frame of video are referred in the class under the immediate cluster centre of azimuth information with corresponding filming apparatus;

(4-9) set up an optimization aim function as follows:

$J=\underset{k=1}{\overset{T}{\mathrm{\Σ}}}\underset{j=1}{\overset{C}{\mathrm{\Σ}}}{\mathrm{\ω}}_k{\mathrm{\μ}}_{\mathrm{kj}}^2{\left|\right|o_k-{\mathrm{\υ}}_j^{\left(p\right)}\left|\right||}_2^2,$

Constraints is: $\underset{j=1}{\overset{C}{\mathrm{\Σ}}}{\mathrm{\μ}}_{\mathrm{kj}}=\mathrm{1,0}\≤{\mathrm{\μ}}_{\mathrm{kj}}\≤1$

The sequence number that wherein k is frame of video, the classification that j is cluster centre, j ∈ [1, C], μ _kjparameter to be solved, υ _jcluster centre, p is current iteration number of times;

(4-10), during initialization, establish p=0, the initial value vector that is j cluster centre;

(4-11) calculate μ _kj: ${\mathrm{\μ}}_{\mathrm{kj}}=1/{\left|\right|o_k-{\mathrm{\υ}}_j^{\left(p\right)}\left|\right|}_2^2$

(4-12), according to above-mentioned result of calculation, upgrade μ _kjvalue, calculate μ _kj:

${\mathrm{\μ}}_{\mathrm{kj}}={\mathrm{\μ}}_{\mathrm{kj}}/\underset{j=1}{\overset{C}{\mathrm{\Σ}}}{\mathrm{\μ}}_{\mathrm{kj}}$

(4-13) μ calculating according to step (4-12) _kj, calculate

(4-14) set an iteration and stop threshold epsilon, if

make p=p+1, and return to step (4-11), if

carry out step (4-15), the span of ε is: 0.01～0.001;

(4-15) by following formula, obtain an initial key frame set K={t ₁, t ₁..., t _c}: $t_j=\underset{k\∈[1,T]}{\arg \max }{\mathrm{\μ}}_{\mathrm{kj}},$ J ∈ [1, C] wherein;

(4-16) calculate the similarity of the frame of video characteristic information of any two width frame of video in above-mentioned initial key frame set K $S_{t_i,t_j}={\left|\right|f_{t_i}-f_{t_j}\left|\right|}_2^2,$ I wherein, j ∈ [1, C];

(4-17) set a similarity threshold, any two frames in the initial key frame set K that traversal step (4-16) calculates, calculate the similarity of the frame of video characteristic information of any two frames

compare with similarity threshold, if from above-mentioned initial key frame set K, delete t _j; If

and

from above-mentioned initial key frame set K, delete t _i; If in above-mentioned initial key frame set K, retain t _iand t _j, repeating this step, the set K obtaining is key frame of video, and the span of δ is: 0.2～0.3.

Images