CN114866695A

CN114866695A - Anti-shake processing method and device for video, storage medium and electronic equipment

Info

Publication number: CN114866695A
Application number: CN202210467413.5A
Authority: CN
Inventors: 桑标; 肖龙报; 袁文亮; 朱力
Original assignee: Insta360 Innovation Technology Co Ltd
Current assignee: Insta360 Innovation Technology Co Ltd
Priority date: 2022-04-27
Filing date: 2022-04-27
Publication date: 2022-08-05
Anticipated expiration: 2042-04-27
Also published as: WO2023207899A1; CN114866695B

Abstract

The application discloses an anti-shake processing method and device for video, a storage medium and an electronic device, wherein the anti-shake processing method comprises the following steps: acquiring a video to be processed and a motion track of a shooting device when shooting the video to be processed; acquiring an out-of-range angle of a video frame in a video to be processed according to the motion track; determining a reference frame and a jitter frame corresponding to the reference frame from a video to be processed according to the out-of-range angle of the video frame; determining an anti-shake field angle of the shake frame according to the out-of-range angle of the reference frame corresponding to the shake frame; and performing anti-shake processing on the video to be processed according to the anti-shake field angle of the shake frame. The anti-shake viewing angle is obtained by adjusting the out-of-range angle of the shake frame according to the motion track of the shooting device, and anti-shake processing is performed on the video to be processed according to the anti-shake viewing angle of the shake frame, so that the anti-shake effect of the video and the display effect of video content are considered, and the video quality is improved.

Description

Anti-shake processing method and device for video, storage medium and electronic equipment

Technical Field

The present application relates to the field of video processing technologies, and in particular, to a method and an apparatus for anti-shake processing of a video, a storage medium, and an electronic device.

Background

In the shooting process, due to the fact that the pose of the shooting device changes, the shot video content is unstable, and the video quality is poor.

In order to improve the video quality, the shot video can be subjected to anti-shake processing. A common anti-shake processing method includes a trajectory smoothing algorithm, wherein a motion trajectory of the shooting device is smoothed by the trajectory smoothing algorithm to obtain an ideal motion trajectory, and then anti-shake processing is performed on a shot video according to the ideal motion trajectory.

However, the anti-shake processing method can only optimize the motion of the shooting device, and does not consider the display effect of the video content, which is likely to cause poor display effect of the processed video content and is not beneficial to improving the video quality.

Disclosure of Invention

The embodiment of the application provides an anti-shake processing method and device for a video, a storage medium and an electronic device, which can improve the video quality.

In a first aspect, an embodiment of the present application provides a method for anti-shake processing of a video, where the method includes:

acquiring a video to be processed and a motion track of a shooting device when shooting the video to be processed;

acquiring an out-of-range angle of a video frame in a video to be processed according to the motion track;

determining a reference frame and a jitter frame corresponding to the reference frame from a video to be processed according to the out-of-range angle of the video frame;

determining an anti-shake field angle of the shake frame according to the out-of-range angle of the reference frame corresponding to the shake frame;

and performing anti-shake processing on the video to be processed according to the anti-shake field angle of the shake frame.

In a second aspect, an embodiment of the present application further provides an anti-shake processing apparatus for a video, including:

the first data acquisition module is used for acquiring a video to be processed and a motion track of the shooting device when shooting the video to be processed;

the second data acquisition module is used for acquiring the out-of-range angle of the video frame in the video to be processed according to the motion track;

the video processing module is used for determining a reference frame and a jitter frame corresponding to the reference frame from the video to be processed according to the out-of-range angle of the video frame;

the field angle calculation module is used for determining the anti-shake field angle of the shake frame according to the out-of-range angle of the reference frame corresponding to the shake frame;

and the anti-shake processing module is used for carrying out anti-shake processing on the video to be processed according to the anti-shake field angle of the shake frame.

In a third aspect, embodiments of the present application further provide a computer-readable storage medium, on which a computer program is stored, and when the computer program runs on a computer, the computer is caused to execute the anti-shake processing method for video provided in any embodiment of the present application.

In a fourth aspect, an embodiment of the present application further provides an electronic device, which includes a processor and a memory, where the memory has a computer program, and the processor is configured to execute the anti-shake processing method for video according to any embodiment of the present application by calling the computer program.

According to the technical scheme, the out-of-range angle of the video frame in the video to be processed is obtained according to the motion track by obtaining the video to be processed and the motion track of the shooting device when the video to be processed is shot, and the reference frame and the jitter frame corresponding to the reference frame are determined from the video to be processed according to the out-of-range angle of the video frame. And then determining the anti-shake field angle of the shake frame according to the out-of-range angle of the reference frame corresponding to the shake frame, and performing anti-shake processing on the video to be processed according to the anti-shake field angle of the shake frame. The out-of-bound angle of the shaking frame can be adaptively adjusted according to the shaking condition of the shaking frame, so that the adjusted out-of-bound angle of the shaking frame can adapt to the motion change of the shooting device, and the anti-shaking processing effect of the video is improved. Secondly, when the jitter frame is adjusted, the anti-jitter view angle of the jitter frame is determined according to the out-of-range angle of the reference frame corresponding to the jitter frame and the motion condition of the jitter frame, so that the video content of the adjusted jitter frame is ensured not to be lost, the maximum utilization rate of the video content is ensured, and the video quality is improved.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.

Fig. 1 is a schematic flowchart of a video anti-shake processing method according to an embodiment of the present application.

Fig. 2 is a schematic diagram illustrating a relationship among a reference frame, a jittered frame and a video frame in the anti-shake processing method for a video according to the embodiment of the present application.

Fig. 3 is another schematic flow chart of a video anti-shake processing method according to an embodiment of the present disclosure.

Fig. 4 is a schematic diagram of a second view angle curve and a third view angle curve in the video anti-shake processing method according to the embodiment of the present application.

Fig. 5 is a schematic flowchart of a video anti-shake processing method according to an embodiment of the present application.

Fig. 6 is a schematic diagram of a reference frame in an anti-shake processing method for video according to an embodiment of the present application.

Fig. 7 is a flowchart illustrating step 107 of the video anti-shake processing method according to the embodiment of the present application.

Fig. 8 is a schematic structural diagram of an apparatus for processing video jitter prevention according to an embodiment of the present application.

Fig. 9 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

Detailed Description

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. It should be apparent that the described embodiments are only some of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without inventive step, are within the scope of the present application.

Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. It is explicitly and implicitly understood by one skilled in the art that the embodiments described herein can be combined with other embodiments.

An execution main body of the video anti-shake processing method may be the video anti-shake processing apparatus provided in the embodiment of the present application, or an electronic device integrated with the video anti-shake processing apparatus. The anti-shake processing device of the video can be realized in a hardware or software mode, and the electronic equipment can be equipment with a video playing function, such as a smart phone, a tablet personal computer, a palm computer, a television, a movie machine and the like, and can also be a shooting device with a shooting function, such as a camera, a video camera, an aircraft and the like. For example, when the electronic device is a smart phone, the to-be-processed video is shot by the smart phone, and the smart phone can perform anti-shake processing on the to-be-processed video shot by the smart phone; certainly, the video to be processed can also be obtained by shooting by a camera and transmitted to the smart phone through the camera, and the smart phone performs anti-shake processing on the video to be processed. For another example, when the electronic device is a camera, the video to be processed may be captured by the camera, and then the camera performs anti-shake processing on the video to be processed.

It should be noted that the scheme provided by the embodiment of the present application performs the anti-shake processing on the to-be-processed video obtained by shooting in the later period. In the related anti-shake technology, there are a mechanical stabilization method and an optical stabilization method, but both methods require improvement on hardware of the electronic device or addition of auxiliary devices, increase the cost of video anti-shake, and have high requirements on the performance of the camera. Compare in relevant anti-shake technique, the scheme of this application has avoided too much cost to drop into, and the suitability is higher, changes the popularization.

In order to better understand the scheme provided by the application, an application scenario is provided herein to explain the scheme provided by the application. For example, the implementation subject of the scheme of the application is a shooting device, and when the shooting device shoots a video, the shooting device can perform anti-shake processing on the video, and store the video after the anti-shake processing, so that the video quality can be improved. When the user plays the video, the video after anti-shake processing is stored in the shooting device, so that the video can not obviously shake during playing, the video display effect is improved, and the user experience is guaranteed.

Compared with the prior art, the scheme provided by the application can also acquire the out-of-bound angle of the video frame in the video to be processed according to the motion track of the shooting device, so that the jitter frame of the video to be processed and the motion condition of the jitter frame are determined according to the motion condition of the shooting device, and the out-of-bound angle of the jitter frame is adaptively adjusted according to the motion condition of the jitter frame, so that the adjusted out-of-bound angle of the jitter frame can adapt to the motion change of the shooting device, and the video anti-shake processing effect is improved. Meanwhile, the video content of the jittered frames is not lost in the adjusting process, the maximum utilization rate of the video content is ensured, and the video quality is improved. In the following embodiments, the scheme provided by the present application is explained in detail by taking a camera as an execution subject.

Referring to fig. 1, fig. 1 is a schematic flowchart illustrating an anti-shake processing method for video according to an embodiment of the present disclosure. The specific flow of the video anti-shake processing method provided by the embodiment of the application can be as follows:

101. the method comprises the steps of obtaining a video to be processed and shooting a motion track of a shooting device when the video to be processed is shot.

The gyroscope is arranged in the shooting device, the pose of the shooting device at each moment can be obtained through the gyroscope, and the pose at each moment can form the motion track of the shooting device.

The method comprises the steps of obtaining a to-be-processed video shot by a shooting device and a motion track of the shooting device when the to-be-processed video is shot, namely representing the shaking condition of a video frame in the to-be-processed video through the motion track of the shooting device, wherein the pose of each moment in the motion track corresponds to the video frame in the to-be-processed video.

102. And acquiring an out-of-range angle of a video frame in the video to be processed according to the motion track.

After the motion trail is obtained, the relative rotation amount between the video frames when the video frames are shot can be obtained according to the association between each pose in the motion trail and the corresponding video frame.

The relative rotation amount between the video frames can be represented by an angle, and the parameters which can be represented by the angle comprise rotation in the horizontal direction and rotation in the vertical direction.

According to the relative rotation amount between the video frames, the out-of-range angle of each video frame can be obtained. The out-of-bounds angle refers to a maximum angle at which image content in a video frame cannot be out of bounds after the video frame is subjected to angle rotation and the video frame subjected to angle rotation is cut. For example, taking horizontal rotation as an example, when a certain video frame is rotated, in order to ensure that the original image content in the video frame is complete, the maximum angle of rotation for the video frame may be the out-of-range angle of the video frame.

103. And determining a reference frame and a jitter frame corresponding to the reference frame from the video to be processed according to the out-of-range angle of the video frame. The video frames with the out-of-range angle smaller than the preset field angle in the video to be processed are determined as reference frames, while the video frames with the out-of-range angle greater than or equal to the preset field angle are non-reference frames.

For each reference frame, each reference frame corresponds to a different number of jittered frames, and the jittered frames corresponding to each reference frame are video frames distributed around the reference frame, which may include additional reference frames or non-reference frames. Therefore, for each of the jittered frames, each jittered frame may correspond to one reference frame or a plurality of reference frames, as shown in fig. 2, fig. 2 is a schematic diagram illustrating a relationship among the reference frames, the jittered frames, and the video frames in the anti-jittering processing method for video according to the embodiment of the present application, all the video frames in fig. 2 are divided into reference frames and non-reference frames, and the jittered frame corresponding to each reference frame may be another reference frame and/or non-reference frame. The following are exemplified:

for example, the 10 th frame video frame is a reference frame, the 15 th frame video frame is also a reference frame, and the jitter frames corresponding to the 10 th frame reference frame may be the 5 th to 9 th frames and the 11 th to 15 th frames, and the jitter frames corresponding to the 15 th frame reference frame may be the 13 th to 14 th frames and the 16 th to 17 th frames, at this time, the reference frame corresponding to the jitter frame 5 is the 10 th frame, and the reference frame corresponding to the jitter frame 13 is the 10 th frame and the 15 th frame.

104. And determining the anti-shake field angle of the shake frame according to the out-of-range angle of the reference frame corresponding to the shake frame.

According to the out-of-bound angle of the video frame, one or more reference frames can be determined from the video to be processed, wherein each reference frame corresponds to a jitter frame, the jitter frame is determined according to the out-of-bound angle of the reference frame, and the anti-jitter processing is performed on the jitter frame according to the out-of-bound angle of the corresponding reference frame.

105. And performing anti-shake processing on the video to be processed according to the anti-shake field angle of the shake frame.

The anti-shake view angle of the shake frame can be calculated according to the out-of-range angle of the reference frame corresponding to the shake frame, wherein the anti-shake view angle is used for carrying out anti-shake processing on the shake frame, so that obvious shake can not occur on the processed shake frame.

In particular implementation, the present application is not limited by the execution sequence of the described steps, and some steps may be performed in other sequences or simultaneously without conflict.

By the video anti-shake processing method in the embodiment of the application, the motion track of the shooting device when shooting the video to be processed can be obtained, the out-of-range angle of the video frame in the video to be processed can be obtained according to the motion track, the reference frame and the shake frame corresponding to the reference frame can be determined from the video to be processed according to the out-of-range angle, and the shake frame which shakes when the shooting device moves can be determined from the video to be processed. And then, an anti-shake field angle of the shake frame is determined according to the out-of-range angle of the reference frame, and then the shake frame is subjected to anti-shake processing according to the anti-shake field angle, so that the out-of-range angle of the shake frame is adaptively adjusted according to the shake condition of the shake frame, the maximum utilization rate of video content is ensured while anti-shake is carried out, and the video quality is improved.

The method according to the preceding embodiment is illustrated in further detail below by way of example.

In some embodiments, in the step of acquiring the motion trajectory of the shooting device when the to-be-processed video is shot, the motion trajectory may also be a motion trajectory that is processed by the trajectory smoothing algorithm and then smoothed, so that the influence of the motion of the shooting device on the to-be-processed video is reduced.

In some embodiments, the out-of-range angle of the video frame may be calculated by a quaternion, where each pose in the motion trajectory may represent a three-axis angular velocity, and the rotation of the video frame may be obtained by converting the three-axis angular velocity by the quaternion, and after obtaining the relative rotation amount between each video frame, the out-of-range angle of each video frame may be obtained according to the relative rotation amount.

Referring to fig. 3, fig. 3 is another schematic flow chart of a video anti-shake processing method according to an embodiment of the present disclosure. In some embodiments, the step of determining a reference frame and a jitter frame corresponding to the reference frame from the video to be processed according to the out-of-range angle of the video frame includes:

1031. determining a video frame with an out-of-range angle smaller than a preset field angle in a video to be processed as a reference frame;

1032. determining the number m of the jitter frames corresponding to the reference frame according to the out-of-range angle and the preset field angle of the reference frame;

1033. and determining m video frames before the reference frame and m video frames after the reference frame as the jitter frames corresponding to the reference frame.

In this embodiment, the video frames with the out-of-range angle smaller than the preset field angle in the video to be processed can be used as the reference frames according to the out-of-range angle of each video frame. The video to be processed may have one or more reference frames, and when there are multiple reference frames, for each reference frame, the method provided in this embodiment may be used to determine the jitter frame corresponding to each reference frame.

The preset field angle may be a larger field angle value, or may be a maximum field angle supported by the photographing device, and of course, the preset field angle may also be set by the user according to actual requirements. And the video frame with the out-of-range angle smaller than the preset field angle is taken as the reference frame, which shows that the out-of-range angle of the reference frame is smaller, the adjustable amplitude is smaller, and relatively speaking, the jitter of the reference frame is not obvious.

For example, when the number of the jittered frames corresponding to the reference frame is calculated according to the out-of-range angle and the preset field angle of the reference frame, the following formula may be adopted:

m＝(FOV _max -FOV _n )÷f

in this formula, m represents the number of dither frames, FOV _max Indicating a predetermined field of view, FOV _n Indicates the out-of-bounds angle of the reference frame and f indicates the field-of-view angle rate of change. Wherein f can be set by a user, or can be a value obtained empirically by default, for example, f can be a value of 0.1, 0.5, 1, etc.

After the number m of the jitter frames is obtained, according to the position of the reference frame in the video to be processed, the m frames of video frames before the reference frame and the m frames of video frames after the reference frame are selected, and then the selected 2m video frames are used as the jitter frames corresponding to the reference frame.

It can be understood that there may be a plurality of reference frames in the video to be processed, the number of the shaking frames corresponding to each reference frame may be sequentially calculated in sequence, and m video frames before and after the video are selected as the shaking frames corresponding to the reference frames according to the positions of the reference frames in the video to be processed. The number of the jitter frames corresponding to the reference frame is determined according to the out-of-range angle of the reference frame, and if the out-of-range angles of different reference frames are different, the number of the corresponding jitter frames is different. As can be seen from the above equation, the number of the jitter frames corresponding to the reference frame is proportional to the out-of-range angle of the reference frame.

In addition, the jitter frames corresponding to different reference frames may also partially overlap, for example, the 20 th frame and the 25 th frame in the video frames to be processed are reference frames, while the number of the jitter frames corresponding to the 20 th frame is 3, i.e., the 17 th, 18 th, 19 th, 21 th, 22 th, 23 th frame in the video to be processed is the jitter frame corresponding to the 20 th frame, and the number of the jitter frames corresponding to the 25 th frame is 5, i.e., the 20 th, 21 th, 22 th, 23 th, 24 th, 26 th, 27 th, 28 th, 29 th, 30 th frame in the video to be processed is the jitter frame corresponding to the 25 th frame, wherein the jitter frames where the 20 th frame and the 25 th frame overlap are the 22 th frame and the 23 th frame.

Next, there may be one or more additional reference frames in the shaking frame corresponding to a certain reference frame, for example, the 20 th frame in the shaking frame corresponding to the above-mentioned 25 th frame is the reference frame.

Referring to fig. 3, in some embodiments, the step of determining the anti-shake fov of the shake frame according to the out-of-range angle of the reference frame corresponding to the shake frame includes:

1041. determining the interval frame number of the jitter frame and the corresponding reference frame;

1042. and determining the anti-shake field angle of the shake frame according to the number of the interval frames and the out-of-range angle of the reference frame corresponding to the shake frame.

And obtaining the interval frame number of each jitter frame and the corresponding reference frame according to the positions of the jitter frame and the reference frame in the video to be processed, wherein the interval frame number is larger when the jitter frame is farther from the reference frame, and is smaller when the jitter frame is farther from the reference frame, otherwise.

When the anti-shake viewing angle of the shake frame is calculated, the calculation formula is based on the out-of-range angle of the corresponding reference frame, and is as follows:

FOV _m ＝FOV _n +i×f

in the formula, FOVm is an anti-shake field angle of a shake frame, FOVn is an out-of-range angle of a reference frame, f is a field angle change rate, i is an interval frame number, and i is 1 and 2.. m;

as described above, the numerical values of the anti-shake viewing angles of the shake frames are distributed in an arithmetic sequence on both sides of the reference frame with the out-of-range angle of the reference frame as a reference. For example, the out-of-range angle of the reference frame corresponding to the dither frame is 90, f is 0.1, m is 3, the reference frame is 20 th, the dither frames corresponding to the reference frames are 17 th, 18 th, 19 th, 21 th, 22 th, and 23 th frames, respectively, the anti-shake viewing angle of the dither frame 17 is 90+3 × 0.1 is 90.3 for the dither frame 17, and so on, the anti-shake viewing angle of the dither frame 18 is 90.2, the anti-shake viewing angle of the dither frame 19 is 90.1, the anti-shake viewing angle of the dither frame 21 is 90.1, the anti-shake viewing angle of the dither frame 22 is 90.2, and the anti-shake viewing angle of the dither frame 23 is 90.3.

It can be understood that, in the case that some of the reference frames correspond to shaking frames having overlap, since the anti-shaking viewing angles of the shaking frames corresponding thereto are calculated one by one in the order of the reference frames, for the overlapped shaking frames, the overlapped shaking frames have a plurality of corresponding anti-shaking viewing angles according to the number of the reference frames corresponding thereto, and the final anti-shaking viewing angle of the overlapped shaking frames is determined by analyzing the plurality of anti-shaking viewing angles corresponding to the overlapped shaking frames. For example, the 17 th, 18 th, 19 th, 21 th, 22 th and 23 th frame video frames in the video to be processed are the shaking frames corresponding to the 20 th frame, the 20 th, 21 th, 22 th, 23 th, 24 th, 26 th, 27 th, 28 th, 29 th and 30 th frame video frames in the video to be processed are the shaking frames corresponding to the 25 th frame, wherein the overlapped shaking frames are the 21 st, 22 nd and 23 th frames. Taking the overlapped shake frames 22 as an example, if the reference frames corresponding to the shake frames 22 are the 20 th frame and the 25 th frame, the shake frames 22 have one anti-shake viewing angle corresponding to the reference frame 20 and another anti-shake viewing angle corresponding to the reference frame 25, and the final anti-shake viewing angle of the shake frames 22 can be determined by analyzing the two anti-shake viewing angles.

Illustratively, the minimum value of a plurality of anti-shake field angles corresponding to the overlapped shake frames may be selected as the final anti-shake field angle. Continuing with the example of the overlapped shaking frame 22, when the reference frame is the 20 th frame, the anti-shaking viewing angle of the shaking frame 22 is 90.2, and when the reference frame is the 25 th frame, for example, the out-of-range angle of the reference frame 25 is 80, the anti-shaking viewing angle calculated by the shaking frame 22 in the above embodiment should be 80.3, wherein one anti-shaking viewing angle of the shaking frame 22 is 90.2, and the other anti-shaking viewing angle is 80.3, and a smaller anti-shaking viewing angle (80.3) is selected as the final anti-shaking viewing angle of the shaking frame 22.

It should be understood that, in the present embodiment, a minimum value is selected from a plurality of anti-shake viewing angles corresponding to a shake frame as a final anti-shake viewing angle, and as an expansion manner of the present embodiment, an average value, a median value, and the like may also be selected to replace the minimum value in the present embodiment to re-determine the final anti-shake viewing angle, and such expansion manners should be included in the protection scope covered by the present application.

In some embodiments, the step of performing anti-shake processing on the video to be processed according to the anti-shake field angle of the shake frame includes:

performing angle transformation processing on the first angle of view curve according to the anti-shake angle of view of the shake frame to obtain a second angle of view curve, wherein the first angle of view curve is constructed according to the out-of-range angle of each video frame in the video to be processed;

and performing anti-shake processing on the video to be processed according to the second field angle curve.

And obtaining the out-of-range angle of each video frame through the motion track, wherein each out-of-range angle forms a first view angle curve. When the anti-shake field angle of the shake frame is obtained, the out-of-range angle corresponding to the shake frame in the first field angle curve may be adjusted to the anti-shake field angle of the shake frame, thereby obtaining a second field angle curve.

In some embodiments, performing an angle transformation process on the first field angle curve according to the anti-shake field angle of the shake frame to obtain a second field angle curve includes:

determining a target shaking frame of which the anti-shaking field angle is not larger than the out-of-range angle from the shaking frames;

and performing angle transformation processing on the first angle of view curve according to the anti-shake angle of view of the target shaking frame to obtain a second angle of view curve.

When the anti-shake field angle of the shake frame is obtained through calculation, the anti-shake field angle is compared with the out-of-range angle of the shake frame, and when the anti-shake field angle is larger than the out-of-range angle, the out-of-range angle of the shake frame is not adjusted to be the anti-shake field angle, so that the defect of video content caused by overlarge adjustment range of the shake frame is avoided, and the video quality is guaranteed.

In some embodiments, the step of performing anti-shake processing on the video to be processed according to the second field angle curve includes:

performing iterative smoothing filtering processing on the second angle of view curve through a moving average filter until a first preset condition is met to obtain a third angle of view curve;

and performing anti-shake processing on the video to be processed according to the third field angle curve.

In this embodiment, after the second angle of view curve is obtained, iterative smoothing filtering is performed on the second angle of view curve through a moving average filter, so as to obtain a smooth third angle of view curve, and anti-shake processing is performed on the video to be processed through the third angle of view curve, so as to avoid a situation that a sharp video frame change occurs in the video to be processed, specifically referring to fig. 4, fig. 4 is a schematic diagram of the second angle of view curve and the third angle of view curve in the video anti-shake processing method provided by the embodiment of the present application, an uneven broken line in fig. 4 is the second angle of view curve before iterative smoothing filtering processing, and a relatively smooth curve is the processed third angle of view curve.

For convenience of expressing the out-of-range angle or the anti-shake viewing angle at the second viewing angle, the numerical value on the second viewing angle curve is collectively referred to as the viewing angle. When the second field angle curve is processed through the moving average filter, the frame number of the smooth window is set, the field angles of a plurality of video frames on two sides of each video frame are selected one by one according to the frame number to obtain an average value, and then the average value is assigned to each video frame, so that the effect of average filtering is achieved. For example, in filtering the 20 th frame, the number of frames in the smoothing window may be 21, and then the angles of view of the 10 th to 20 th frames and the angles of view of the 22 th to 32 th frames are selected, and the 21 angles of view are averaged to obtain the average value as the new angle of view of the 20 th frame.

The third viewing angle curve is obtained by performing the smoothing filtering process once on the second viewing angle curve, and the second viewing angle curve and the third viewing angle curve are compared so as to adjust the viewing angle of the portion of the third viewing angle curve larger than the second viewing angle curve to the original value, for example, when the viewing angle in the third viewing angle curve is 90 and the viewing angle in the second viewing angle curve is 20 for the 25 th frame, the viewing angle in the 25 th frame in the third viewing angle curve is adjusted from 90 to the original value of 20.

And after the part of the third angle of view curve, which is larger than the second angle of view curve, is adjusted to the original value, the smooth filtering processing can be continuously carried out on the adjusted third angle of view curve, and so on, the smooth filtering is carried out every time, and the angle of view of the part of the smooth filtered angle of view curve, which is larger than the angle of view curve before the smooth filtering, is adjusted to the original value, so that the angle of view is regarded as an adjustment period, and then the iteration processing is carried out in the adjustment period until the iteration times reach the preset times, or the filtered angle of view curve is smooth enough.

The first preset condition is a preset number of times or the curvature of the field angle curve. The preset number may be set according to actual requirements, for example, set to 20, 23, 25, 30, etc.

In some embodiments, before performing an iterative smoothing filtering process on the second view angle curve through the moving average filter until the first preset condition is met to obtain a third view angle curve, the method further includes:

and adjusting the out-of-range angle outside the preset field angle section in the second field angle curve according to the preset field angle section.

The preset field angle interval may refer to an angle range covered by the aforementioned preset field angle, and as can be seen from the above, the preset field angle may be a larger field angle value or a maximum field angle supported by the photographing device, and one boundary of the preset field angle interval may be the preset field angle, and the other boundary may be a smaller field angle value or a minimum field angle supported by the photographing device.

For example, taking a preset viewing angle section composed of a maximum viewing angle and a minimum viewing angle as an example, by adjusting the second viewing angle curve by the preset viewing angle, an out-bound angle larger than the maximum viewing angle in the second viewing angle curve can be adjusted to the maximum viewing angle, and an out-bound angle smaller than the minimum viewing angle in the second viewing angle curve can be adjusted to the minimum viewing angle, so that all the viewing angles in the adjusted second viewing angle curve are ensured to be in the preset viewing angle section, and the display effect of the video is ensured.

In some embodiments, the anti-shake processing is performed on the video to be processed according to the third field angle curve, and includes:

and after the third angle of view curve is obtained, comparing the third angle of view curve with the second angle of view curve, calculating a difference value by using each angle of view in the third angle of view curve and a corresponding angle of view in the second angle of view value, further selecting a maximum difference value from all the difference values, and subtracting the maximum difference value from each angle of view in the third angle of view curve, so that the overall filtering of the third angle of view curve is realized, the smoothing processing of the third angle of view curve is realized, and the video anti-shake effect is improved.

After the third view angle curve is obtained through the above embodiment, the anti-shake processing may be further performed on the video to be processed according to the third view angle curve, and during the processing, each video frame is rotated and cropped according to the view angle in the third view angle curve according to the corresponding relationship between each view angle in the third view angle curve and the video frame in the video frame to be processed, so as to obtain the processed video.

As can be seen from the above-mentioned contents in the embodiments, the solution provided in the embodiments of the present application determines the anti-shake viewing angles of a plurality of shake frames near each reference frame by taking each reference frame as a reference, and determines the anti-shake viewing angles of the shake frames according to the distances from the shake frames to the corresponding reference frame when determining the anti-shake viewing angles of the shake frames, where the anti-shake viewing angles are smaller when the distance from the reference frame is closer and larger when the distance from the reference frame is farther, and where the shake frames are closer to the reference frame, indicating that the shake of the shake frames is relatively larger, and adopting a smaller anti-shake viewing angle is not likely to cause the video content to go out of the field, thereby ensuring the video display effect, and when the distance from the shake frames to the reference frame is farther, indicating that the shake frames is relatively smaller, and adopting a larger anti-shake viewing angle can improve the anti-shake effect. Moreover, smooth filtering processing is carried out on the second view angle curve, so that abrupt view angle change cannot occur between video frames after anti-shake processing, and video quality is improved.

When determining the jitter frame corresponding to each reference frame, an overlap phenomenon may occur between the jitter frames, and in order to avoid repeatedly determining the anti-jitter view angle for a certain jitter frame, it may also be determined whether the interval between adjacent reference frames is close, and when the interval between adjacent reference frames is close, the anti-jitter view angle of the jitter frame between the adjacent reference frames is not determined in the above manner, specifically as follows:

referring to fig. 5, fig. 5 is a schematic flow chart of a video anti-shake processing method according to an embodiment of the present application, in some embodiments, after determining a video frame with an angle of field smaller than a preset angle of field in a video to be processed as a reference frame, the method further includes:

106. determining a reference frame of which the interval between two adjacent reference frames meets a second preset condition as a first type of reference frame, and determining reference frames except the first type of reference frame as a second type of reference frame;

107. if the reference frame is a first-type reference frame, determining a video frame between two adjacent first-type reference frames as a jitter frame corresponding to the two adjacent first-type reference frames;

108. and if the reference frame is the second type of reference frame, determining the number m of the jitter frames corresponding to the reference frame according to the out-of-range angle and the preset field angle of the reference frame.

In this embodiment, a reference frame whose interval between two adjacent reference frames is closer is determined as a first-type reference frame, and reference frames other than the first-type reference frame are regarded as second-type reference frames. For example, the video to be processed has a plurality of reference frames, an out-of-range angle of each reference frame is smaller than the preset field angle, and a video frame with an out-of-range angle not smaller than the preset field angle may exist between two adjacent reference frames, or a video frame with an out-of-range angle not smaller than the preset field angle may not exist.

As shown in fig. 6, fig. 6 is a schematic diagram of a reference frame provided in the present application. The unfilled rectangles represent reference frames, and the rectangles with oblique lines represent reference frames having an angle of boundary not less than a preset angle of view, wherein the reference frames are sorted, and then the reference frame with a closer interval between every two reference frames is determined as a first type of reference frame, and the reference frame with a farther interval between every two reference frames is determined as a second type of reference frame.

In some embodiments, prior to step 103, the method further comprises:

and when the reference frame is the first type reference frame, determining the video frame between two adjacent first type reference frames as the jitter frame corresponding to the two adjacent first type reference frames. It is understood that the out-of-range angle of the jitter frame between the two adjacent first-class reference frames is not smaller than the preset field angle.

When the reference frame is a reference frame of the second type, then, in the manner mentioned in the above embodiment: and determining the number m of the jitter frames corresponding to the reference frames according to the out-of-range angle and the preset field angle corresponding to each reference frame, and processing the jitter frames corresponding to the second type of reference frames according to the content mentioned in the subsequent steps. For details, reference is made to the above embodiments, which are not described herein again.

In some embodiments, the method further includes determining a reference frame, in which an interval between two adjacent reference frames meets a second preset condition, as the first type of reference frame, and determining a reference frame other than the first type of reference frame as the second type of reference frame, before:

calculating the predicted frame number between two adjacent reference frames according to the out-of-range angle and the preset field angle of the two adjacent reference frames, and judging that the interval between the two adjacent reference frames meets a second preset condition when the predicted frame number is greater than the actual frame number;

or when the interval between two adjacent reference frames is not more than the preset frame number, judging that the interval between two adjacent reference frames meets the second preset condition.

In this embodiment, whether the interval between two adjacent reference frames meets the second preset condition may be determined in any manner, and when the interval meets the second preset condition, the two adjacent reference frames are determined as the first type of reference frame.

In an embodiment, the predicted frame number of the video frame that should exist between two adjacent reference frames may be calculated according to the out-of-range angle and the preset field angle of the two adjacent reference frames, and the predicted frame number is compared with the actual frame number, when the predicted frame number is larger, it indicates that the actual frame number between the two adjacent reference frames is smaller, that is, the interval between the two adjacent reference frames meets the second preset condition, that is, the interval is smaller.

The formula for calculating the predicted frame number according to the out-of-range angle and the preset field angle is as follows:

x＝h·(2·FOV _MAX -FOV _n1 -FOV _n2 )/f

in this formula, x represents the predicted frame number, h represents the coefficient, FOV _max Indicating a predetermined field of view, FOV _n1 And FOV _n2 Two adjacent reference frames are shown, and f represents the angular rate of change of the field of view. H can be set according to actual requirements, for example, set to a value of 1.5, 2, 3, and the like.

In another embodiment, the actual frame number between two adjacent reference frames may be compared with the preset frame number, and when the actual frame number is smaller, it indicates that the interval between the two adjacent reference frames meets the second preset condition, i.e. the interval is smaller.

The preset frame number can be set according to actual requirements, and can also be a default value, such as 40, 50, 60, and the like.

In another embodiment, the predicted frame number may be calculated simultaneously, the predicted frame number is compared with a preset frame number, a maximum value is selected from the predicted frame number and the preset frame number, then the maximum value is compared with the actual frame number, and when the actual frame number is smaller than the maximum value, the interval between two adjacent reference frames is judged to meet the second preset condition.

For example, the 1 st reference frame (20 th frame of the original video) and the 2 nd reference frame (25 th frame of the original video) are two adjacent reference frames, the actual frame number between the 1 st reference frame and the 2 nd reference frame is 25-20 ═ 5, and the predicted frame number is x ═ 1.5 ═ 2 × 100-99-98)/0.1 ═ 45, where the predicted frame number is 45, the preset frame number is 60, the predicted frame number is less than the preset frame number, the actual frame number is compared with the preset frame number, and it is known that the actual frame number is 5 less than the preset frame number 60, the interval between the 1 st reference frame and the 2 nd reference frame is considered to meet the second preset condition, and the 1 st reference frame and the 2 nd reference frame are both the first type reference frame.

In some embodiments, the first type of reference frame corresponds to a shaking frame that is a first type of shaking frame, and the second type of reference frame corresponds to a shaking frame that is a second type of shaking frame;

the method comprises the following steps of determining a reference frame and a jitter frame corresponding to the reference frame from a video to be processed according to the out-of-range angle of the video frame, wherein the steps comprise:

determining an anti-shake field angle of the first type of shaking frames according to the out-of-bound angles of two adjacent first type of reference frames corresponding to the first type of shaking frames and the frame sequence of the first type of shaking frames in a sub-out-of-bound area, wherein the sub-out-bound area is formed by a video frame between the two adjacent first type of reference frames corresponding to the first type of shaking frames and the two adjacent first type of reference frames;

and determining the number of interval frames of the second type of shaking frames and the corresponding second type of reference frames, and determining the anti-shaking field angle of the second type of shaking frames according to the number of interval frames and the out-of-range angle of the second type of reference frames corresponding to the second type of shaking frames.

In this embodiment, two adjacent first reference frames and the video frames therebetween form a sub-out-of-bound region, and when determining the anti-shake field angle of the first type of shake frame corresponding to the first type of reference frame, the anti-shake field angle of the first type of shake frame is calculated according to the out-of-bound angles of the two adjacent first shake frames and the frame sequence of the first shake frame in the sub-out-of-bound region, that is, the anti-shake field angle of the first type of shake frame is associated with the out-of-bound angles of the two adjacent reference frames.

For the second type of shaking frames corresponding to the second type of reference frames, the anti-shaking view angle of each second type of shaking frame is determined according to the above mentioned manner, which can refer to the formula FOV _m ＝FOV _n + i × f, which will not be described in detail herein.

Referring to fig. 5, in some embodiments, determining an anti-shake view angle of a first type of shake frame according to an out-of-bound angle of two adjacent first type reference frames corresponding to the first type of shake frame and a frame sequence of the first type of shake frame in a sub-out-of-bound area includes:

1043. determining an offset coefficient according to the out-of-range angle and the actual frame number of two adjacent first-type reference frames corresponding to the first-type jitter frames;

1044. determining the anti-shake offset of the first type of shaking frame according to the offset coefficient and the frame sequence of the first type of shaking frame in the sub out-boundary area;

1045. and determining the anti-shake field angle of the first-class shaking frame according to the anti-shake offset and the out-of-range angles of the two adjacent first-class reference frames corresponding to the first-class shaking frame.

In this embodiment, the formula for determining the offset coefficient according to the out-of-range angle and the actual frame number of two adjacent first-type reference frames is as follows:

a＝(FOV _n2 -FOV _n1 )/(n ₂ -n ₁ )

in this formula, a represents the offset coefficient, FOV _n1 And FOV _n2 Representing two adjacent reference frames, n ₂ -n ₁ Representing the actual number of frames.

After obtaining the offset coefficient, determining the anti-shake offset according to the frame sequence of the first type of shaking frames and the offset coefficient, and determining the anti-shake offset as the product of the offset coefficient and the frame sequence, wherein the first type of reference frames and the first type of shaking frames are sequenced in each sub-out boundary area, the magnitude of the anti-shake offset is related to the sequencing of the first type of shaking frames in the corresponding sub-out boundary area, when the frame sequence of the first type of shaking frames is larger, the anti-shake offset of the first type of shaking frames is larger, and otherwise, the anti-shake offset is smaller.

When the anti-shake offset of the first-class shake frame is calculated, the anti-shake field of view is calculated by taking any one of the two corresponding adjacent first-class reference frames as a reference, and the anti-shake field of view of the first-class shake frame in each sub-boundary area is in an arithmetic sequence.

For example, taking the first type reference frame in each sub-bounding area as an example, the formula for calculating the anti-shake field angle of each first type shaking frame in the sub-bounding area is as follows:

FOV _n1+k ＝k·a+FOV _n1

in this formula, k represents a frame sequence, and k is 1, 2, …, n ₂ -n ₁ ，FOV _n1+k Denotes the first one firstThe anti-shake field angle of the kth first-type shake frame after the class reference frame, a represents an offset coefficient, k × a represents an anti-shake offset amount, FOV _n1 Indicating the out-of-bounds angle of the first type of reference frame.

Similarly, the anti-shake field angle of each first type of shaking frame in the sub-out-bound region may also be calculated according to the second first type of reference frame in the sub-out-bound region, which is not described herein again.

Referring to fig. 7, fig. 7 is a flowchart illustrating a step 107 in the anti-shake processing method for video according to an embodiment of the present application, in some embodiments, the step of determining, as a shake frame corresponding to two adjacent first-type reference frames, a video frame between two adjacent first-type reference frames for the first-type reference frames includes:

1071. for the first-class reference frames, determining a video frame between two adjacent first-class reference frames and the two adjacent first-class reference frames as a sub-out boundary area;

1072. for a sub-out-boundary area without an adjacent sub-out-boundary area, determining a video frame between two first-type reference frames in the sub-out-boundary area as a jitter frame corresponding to the two first-type reference frames;

1073. for a sub-out-of-bound area with adjacent sub-out-of-bound areas, combining a plurality of adjacent sub-out-of-bound areas into one out-of-bound area, and dividing the out-of-bound area into at least one section, wherein the length of the section is greater than that of the sub-out-of-bound area;

1074. for each block section, calculating a predicted out-of-range angle of a first class reference frame between the head and the tail of two first class reference frames in the block section according to the out-of-range angles of the head and the tail of the two first class reference frames in the block section;

1075. and if the predicted out-of-bounds angle is smaller than the actual out-of-bounds angle, determining the video frame between the first and last first-class reference frames as the jitter frame corresponding to the first and last first-class reference frames.

In this embodiment, when there are a plurality of adjacent sub-out-bound regions, the plurality of adjacent sub-out-bound regions may be merged into one out-bound region, and the number of the first type reference frames in each out-bound region is not less than 3, it can be understood that the length of any one out-bound region is greater than the length of the sub-out-bound region, where the number of the first type reference frames in the sub-out-bound region is 2.

For example, the out-of-bound region may be divided into at least one section, and the length of each section is greater than that of the sub-out-of-bound region, that is, at least 3 first-class reference frames exist in each section.

Each section may not overlap, may also permutation and combination at least 3 first kind reference frames. For example, after combining a plurality of adjacent sub-boundary regions into a boundary region, the sub-boundary region has 6 first-type reference frames, which are respectively (n) ₁ 、n ₂ 、n ₃ 、n ₄ 、n ₅ 、n ₆ ) When dividing the interval, (n) is selected ₁ 、n ₂ 、n ₃ ) And (n) ₄ 、n ₅ 、n ₆ ) One block segment, the two block segments do not overlap. If (n) is taken ₁ 、n ₂ 、n ₃ )、(n ₂ 、n ₃ 、n ₄ )、(n ₃ 、n ₄ 、n ₅ )、(n ₄ 、n ₅ 、n ₆ ) One block, respectively, the 4 blocks are partially overlapped. If (n) is taken ₁ 、n ₂ 、n ₃ )、(n ₂ 、n ₃ 、n ₄ )、(n ₃ 、n ₄ 、n ₅ )、(n ₄ 、n ₅ 、n ₆ )、(n ₁ 、n ₅ 、n ₆ )、(n ₂ 、n ₅ 、n ₆ )、(n ₃ 、n ₅ 、n ₆ )、(n ₁ 、n ₂ 、n ₄ )、(n ₁ 、n ₂ 、n ₅ )、(n ₁ 、n ₂ 、n ₆ )、(n ₂ 、n ₃ 、n ₅ ) If the number of the sections is one, the sections are selected according to a permutation and combination mode. In the specific embodiment, the dividing manner of the block section and the length of the block section may be selected according to actual requirements, which are not listed here.

When the block section is obtained, it is determined whether the video frame in the block section needs to be adjusted, that is, whether a jitter frame exists in the block section. The specific mode is as follows: and predicting the predicted out-of-range angle of the first class reference frame between the first class reference frame and the second class reference frame through the out-of-range angles of the first class reference frame and the second class reference frame in the interval, comparing the predicted field angle with the actual out-of-range angle, and determining that the distance between the first class reference frame and the second class reference frame is long when the predicted out-of-range angle is smaller than the actual out-of-range angle, so that the video frame between the first class reference frame and the second class reference frame can be determined as the corresponding first class jitter frame. At this time, the first type of shaking frames include both the first type of reference frames located between the first and last first type of reference frames and the video frames with the out-of-range angle not less than the preset field angle.

When the predicted out-of-bounds angle is not smaller than the actual out-of-bounds angle, the distance between the first type reference frame and the last type reference frame is short, no jitter frame exists between the first type reference frame and the last type reference frame, and the out-of-bounds angle of the video frame between the first type reference frame and the last type reference frame does not need to be adjusted.

the method for determining the anti-shake field angle of the shake frame according to the out-of-range angle of the reference frame corresponding to the shake frame comprises the following steps:

and determining the anti-shake field angle of the first type of shaking frame according to the out-of-range angles of the first and last two first type reference frames corresponding to the first type of shaking frame and the frame sequence of the first type of shaking frame in the interval.

And determining the number of interval frames of the second type of shaking frames and the corresponding second type of reference frames, and determining the anti-shaking field angle of the second type of shaking frames according to the number of interval frames and the out-of-range angle of the first reference frames corresponding to the second type of shaking frames.

In this embodiment, when calculating the anti-shake field angle of the first type of shaking frame of the block section, the formula FOV in the above embodiment may be referred to _n1+k ＝k·a+FOV _n1 Wherein, the first type of jitter frames in each block section are arranged in sequence, and for the first type of jitter frames in each block section, the first type of jitter frames are referenced according to the frame sequence in the block section and the first and the last two first types of referenceAnd calculating an anti-shake offset coefficient a of the first type of shake frame according to the out-of-range angle of the frame, then obtaining an anti-shake offset k & lta & gt according to the frame sequence k and the anti-shake offset coefficient a, and then calculating the anti-shake field angle of the first type of shake frame according to the first type of reference frame positioned at the head or the second type of reference frame positioned at the tail.

It can be understood that, when there are overlapping time segments, for the overlapped first-type shaking frames, the final anti-shaking view angle thereof selects the minimum value of the plurality of corresponding anti-shaking view angles, and may also select the average value of the plurality of corresponding anti-shaking view angles, which is specifically referred to the content mentioned in the above embodiments and will not be described herein again.

For calculating the anti-shake field angle of the second type of shake frame, reference may be made to the above-mentioned contents, and details thereof are not repeated here.

In some embodiments, the anti-shake processing is performed on the video to be processed according to the anti-shake field angle of the shake frame, and includes:

determining a first type of target shaking frame with an anti-shaking field angle not larger than an out-of-range angle from the first type of shaking frame, and determining a second type of target shaking frame with the anti-shaking field angle not larger than the out-of-range angle from the second type of shaking frame;

and performing anti-shake processing on the video to be processed according to the anti-shake field angle of the first type of target shaking frame and the anti-shake field angle of the second type of target shaking frame.

With reference to the above-mentioned contents of the embodiments, similarly, a first type target shaking frame with an anti-shaking field angle not greater than the out-of-range angle is determined from the first type shaking frame, the first type shaking frame is subjected to anti-shaking processing according to the anti-shaking field angle of the first type target shaking frame, a second type target shaking frame with an anti-shaking field angle not greater than the out-of-range angle is determined from the second type shaking frame, and the second type target shaking frame is subjected to anti-shaking processing according to the anti-shaking field angle of the second type target shaking frame.

When the anti-shake field angle is larger than the out-of-range angle, the out-of-range angle of the shake frame is not adjusted to be the anti-shake field angle, so that the defect of video content caused by overlarge adjustment range of the shake frame is avoided, and the video quality is ensured.

In some embodiments, the anti-shake processing is performed on the video to be processed according to the anti-shake field angle of the first type of target shake frame and the anti-shake field angle of the second type of target shake frame, and the processing includes:

performing angle transformation processing on the first field angle curve according to the anti-shake field angle of the first type of target shaking frame and the anti-shake field angle of the second type of target shaking frame to obtain a second field angle curve, wherein the first field angle curve is constructed according to the out-of-range angle of each video frame in the video to be processed;

adjusting an out-of-range angle, which is outside the preset field angle section, in the second field angle curve according to the preset field angle section to obtain a third field angle curve;

After the anti-shake field angle of the first type of target shaking frame and the anti-shake field angle of the second type of target shaking frame are obtained, the out-of-range angle corresponding to the first type of target shaking frame in the first field angle curve can be adjusted to be the anti-shake field angle of the first type of target shaking frame, and the out-of-range angle corresponding to the second type of target shaking frame can be adjusted to be the anti-shake field angle of the second type of target shaking frame, so that the second field angle curve is obtained.

As mentioned in the above embodiment, the second view angle curve may be processed through the preset view angle section, so that all the view angles in the third view angle curve are in the preset view angle section, thereby ensuring the display effect of the video.

In some embodiments, before adjusting the out-of-range angle of the second viewing angle curve outside the preset viewing angle interval according to the preset viewing angle interval, the method further includes:

and after the third angle of view curve is obtained, comparing the third angle of view curve with the second angle of view curve, calculating a difference value by using each angle of view in the third angle of view curve and the corresponding angle of view in the second angle of view value, further selecting a maximum difference value from all the difference values, and subtracting the maximum difference value from each angle of view in the third angle of view curve.

As can be seen from the above, the anti-shake processing method for video according to the embodiment of the present invention can determine the reference frame from the video to be processed and the shake frame corresponding to the reference frame by obtaining the motion trajectory when the to-be-processed video is photographed by the photographing device and obtaining the out-bound angle of the video frame in the to-be-processed video according to the motion trajectory, thereby determining the shake frame that shakes during the movement of the photographing device from the video to be processed, and adaptively adjusting the out-bound angle of the shake frame according to the movement condition of the shake frame, so that the adjusted out-bound angle of the shake frame can adapt to the movement change of the photographing device, that is, a larger angle is used when the movement change is small, the utilization rate of the field angle is ensured, a smaller angle is used when the movement change is large, the anti-shake effect is provided, and furthermore, by performing a smoothing process on the field angle curve, so that no abrupt angular changes occur between adjacent frames. Therefore, the rationalization of the utilization rate of the field angle is ensured while the anti-shake effect and the utilization rate of the video content are considered, the video display effect is ensured, and the video quality is improved.

In an embodiment, an apparatus 200 for processing video is also provided. Referring to fig. 8, fig. 8 is a schematic structural diagram of an anti-shake processing apparatus 200 for video according to an embodiment of the present application. Wherein the video anti-shake processing apparatus 200 is applied to an electronic device, the video anti-shake processing apparatus 200 includes:

a first data obtaining module 201, configured to obtain a video to be processed and a motion trajectory of a shooting device when shooting the video to be processed;

the second data acquisition module 202 is configured to acquire an out-of-range angle of a video frame in the video to be processed according to the motion trajectory;

the video processing module 203 is configured to determine a reference frame and a jitter frame corresponding to the reference frame from the video to be processed according to the out-of-range angle of the video frame;

the field angle calculation module 204 is configured to determine an anti-shake field angle of the shake frame according to the out-of-range angle of the reference frame corresponding to the shake frame;

and the anti-shake processing module 205 is configured to perform anti-shake processing on the video to be processed according to the anti-shake field angle of the shake frame.

In some embodiments, the video processing module 203 is further configured to:

determining a video frame with an interface angle smaller than a preset field angle in a video to be processed as a reference frame;

determining the number m of the jitter frames corresponding to the reference frame according to the out-of-range angle and the preset field angle of the reference frame;

and determining m video frames before the reference frame and m video frames after the reference frame as the jitter frames corresponding to the reference frame.

In some embodiments, the field angle calculation module 204 is further configured to:

determining the interval frame number of the jitter frame and the corresponding reference frame;

and determining the anti-shake field angle of the shake frame according to the number of the interval frames and the out-of-range angle of the reference frame corresponding to the shake frame.

In some embodiments, the anti-shake processing module 205 is further configured to:

determining a reference frame of which the interval between two adjacent reference frames meets a second preset condition as a first type of reference frame, and determining reference frames except the first type of reference frame as a second type of reference frame;

if the reference frame is a first-class reference frame, determining a video frame between two adjacent first-class reference frames as a jitter frame corresponding to the two adjacent first-class reference frames;

and if the reference frame is the second type of reference frame, determining the number m of the jitter frames corresponding to the reference frame according to the out-of-range angle and the preset field angle of the reference frame.

In some embodiments, before determining a reference frame whose interval between two adjacent reference frames meets a second preset condition as the first type of reference frame and determining a reference frame other than the first type of reference frame as the second type of reference frame, the view angle calculation module 204 is further configured to:

the field angle calculation module 204 is further configured to:

and determining the number of interval frames of the second type of jitter frames and the corresponding second type of reference frames, and determining the anti-shake field angle of the second type of jitter frames according to the number of interval frames and the out-of-range angle of the second type of reference frames corresponding to the second type of jitter frames.

determining an offset coefficient according to the out-of-range angle and the actual frame number of two adjacent first-type reference frames corresponding to the first-type jitter frames;

determining the anti-shake offset of the first type of shaking frame according to the offset coefficient and the frame sequence of the first type of shaking frame in the sub out-boundary area;

and determining the anti-shake field angle of the first-class shaking frame according to the anti-shake offset and the out-of-range angles of the two adjacent first-class reference frames corresponding to the first-class shaking frame.

for the first-class reference frames, determining a video frame between two adjacent first-class reference frames and the two adjacent first-class reference frames as a sub-out boundary area;

for a sub-out-boundary area without an adjacent sub-out-boundary area, determining a video frame between two first-type reference frames in the sub-out-boundary area as a jitter frame corresponding to the two first-type reference frames;

for a sub-out-of-bound area with adjacent sub-out-of-bound areas, combining a plurality of adjacent sub-out-of-bound areas into one out-of-bound area, and dividing the out-of-bound area into at least one section, wherein the length of the section is greater than that of the sub-out-of-bound area;

for each interval, calculating a predicted out-of-bound angle of a first type of reference frame between a head first type of reference frame and a tail first type of reference frame according to the out-of-bound angles of the head first type of reference frame and the tail first type of reference frame in the interval;

and if the predicted out-of-bounds angle is smaller than the actual out-of-bounds angle, determining the video frame between the first and last first-class reference frames as a jittering frame corresponding to the first and last first-class reference frames.

the field angle calculation module 204 is further configured to:

It should be noted that the anti-shake processing apparatus 200 for video provided in this embodiment of the present application and the anti-shake processing method for video in the foregoing embodiments belong to the same concept, and any method provided in the embodiment of the anti-shake processing method for video can be implemented by the anti-shake processing apparatus 200 for video, and the specific implementation process thereof is described in detail in the embodiment of the anti-shake processing method for video, and is not described herein again.

As can be seen from the above, the anti-shake processing apparatus for video provided in the embodiment of the present application can determine the reference frame from the video to be processed and the shake frame corresponding to the reference frame by obtaining the motion trajectory when the shooting apparatus shoots the video to be processed and obtaining the out-bound angle of the video frame in the video to be processed according to the motion trajectory, thereby determining the shake frame which shakes when the shooting apparatus moves from the video to be processed and adaptively adjusting the out-bound angle of the shake frame according to the motion condition of the shake frame, so that the adjusted out-bound angle of the shake frame can adapt to the motion change of the shooting apparatus, that is, a larger angle is used when the motion change is small, the utilization rate of the field angle is ensured, a smaller angle is used when the motion change is large, the anti-shake effect is provided, and furthermore, the field angle curve is smoothed, so that no abrupt angular changes occur between adjacent frames. Therefore, the rationalization of the utilization rate of the field angle is ensured while the anti-shake effect and the utilization rate of the video content are considered, the video display effect is ensured, and the video quality is improved.

The embodiment of the application further provides an electronic device, and the electronic device can be a smart phone, a tablet computer, a palm computer, a television, a movie machine, a camera, a video camera, an aircraft and other devices. As shown in fig. 9, fig. 9 is a schematic structural diagram of an electronic device according to an embodiment of the present application. The electronic device 300 includes a processor 301 having one or more processing cores, a memory 302 having one or more computer-readable storage media, and a computer program stored on the memory 302 and executable on the processor. The processor 301 is electrically connected to the memory 302. Those skilled in the art will appreciate that the electronic device configurations shown in the figures do not constitute limitations of the electronic device, and may include more or fewer components than shown, or some components in combination, or a different arrangement of components.

The processor 301 is a control center of the electronic device 300, connects various parts of the whole electronic device 300 by using various interfaces and lines, performs various functions of the electronic device 300 and processes data by running or loading software programs and/or modules stored in the memory 302, and calling data stored in the memory 302, thereby monitoring the electronic device 300 as a whole.

In this embodiment of the application, the processor 301 in the electronic device 300 loads instructions corresponding to processes of one or more application programs into the memory 302, and the processor 301 executes the application programs stored in the memory 302 according to the following steps, so as to implement various functions:

The above operations can be implemented in the foregoing embodiments, and are not described in detail herein.

In the foregoing embodiments, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.

As can be seen from the above, the electronic device provided in this embodiment can obtain the out-of-range angle of the video frame in the video to be processed according to the motion trajectory obtained when the shooting device shoots the video to be processed, thereby determining the reference frame and the jitter frame corresponding to the reference frame from the video to be processed according to the out-of-range angle, realizing the determination of the jitter frame jittered when the shooting device moves from the video to be processed, and adaptively adjusting the out-of-bound angle of the shaking frame according to the motion condition of the shaking frame so that the adjusted out-of-bound angle of the shaking frame can adapt to the motion change of the shooting device, the method uses a larger angle under the condition of small motion change to ensure the utilization rate of the angle of view, uses a smaller angle under the condition of large motion change to provide an anti-shake effect, and smoothes the angle of view curve to ensure that no abrupt angle change occurs between adjacent frames. Therefore, the rationalization of the utilization rate of the field angle is ensured while the anti-shake effect and the utilization rate of the video content are considered, the video display effect is ensured, and the video quality is improved.

It will be understood by those skilled in the art that all or part of the steps of the methods of the above embodiments may be performed by instructions or by associated hardware controlled by the instructions, which may be stored in a computer readable storage medium and loaded and executed by a processor.

To this end, the embodiments of the present application provide a computer-readable storage medium, and it can be understood by those skilled in the art that all or part of the steps in the method for implementing the embodiments described above can be implemented by hardware that is instructed to be implemented by a program, and the program can be stored in a computer-readable storage medium, and when executed, the program includes the following steps:

The storage medium may be ROM/RAM, magnetic disk, optical disk, etc. Since the computer program stored in the storage medium can execute the steps in any video anti-shake processing method provided in the embodiment of the present application, beneficial effects that can be achieved by any video anti-shake processing method provided in the embodiment of the present application can be achieved, which are detailed in the foregoing embodiments and will not be described herein again.

The method, the apparatus, the medium, and the electronic device for video anti-shake processing provided in the embodiments of the present application are described in detail above, and a specific example is applied in the present application to explain the principle and the implementation of the present application, and the description of the above embodiments is only used to help understand the method and the core idea of the present application; meanwhile, for those skilled in the art, according to the idea of the present application, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present application.

Claims

1. An anti-shake processing method for video, the method comprising:

acquiring an out-of-range angle of a video frame in the video to be processed according to the motion track;

determining a reference frame and a jitter frame corresponding to the reference frame from the video to be processed according to the out-of-range angle of the video frame;

and carrying out anti-shake processing on the video to be processed according to the anti-shake field angle of the shake frame.

2. The method according to claim 1, wherein the step of determining a reference frame and a jitter frame corresponding to the reference frame from the video to be processed according to the out-of-range angle of the video frame comprises:

determining a video frame with an out-of-range angle smaller than a preset field angle in the video to be processed as a reference frame;

determining the number m of the jitter frames corresponding to the reference frame according to the out-of-range angle of the reference frame and the preset field angle;

3. The method according to claim 1, wherein the step of determining the anti-shake field of view of the shake frame according to the out-of-range angle of the reference frame corresponding to the shake frame comprises:

4. The method according to claim 1, wherein the step of performing anti-shake processing on the video to be processed according to an anti-shake field angle of the shake frame comprises:

performing angle transformation processing on a first field angle curve according to the anti-shake field angle of the shake frame to obtain a second field angle curve, wherein the first field angle curve is constructed according to the out-of-range angle of each video frame in the video to be processed;

5. The method according to claim 4, wherein the step of performing anti-shake processing on the video to be processed according to the second view angle curve comprises:

6. The method according to claim 2, wherein after the step of determining the video frames with the out-of-range angle smaller than the preset field angle in the video to be processed as the reference frames, the method further comprises:

and if the reference frame is a second-class reference frame, determining the number m of the jitter frames corresponding to the reference frame according to the out-of-range angle of the reference frame and the preset field angle.

7. The method according to claim 6, wherein before determining the reference frame whose interval between two adjacent reference frames meets the second preset condition as the first type of reference frame and determining the reference frame other than the first type of reference frame as the second type of reference frame, the method further comprises:

calculating the predicted frame number between the two adjacent reference frames according to the out-of-range angle and the preset field angle of the two adjacent reference frames, and judging that the interval between the two adjacent reference frames meets a second preset condition when the predicted frame number is greater than the actual frame number;

or when the interval between two adjacent reference frames is not greater than the preset frame number, judging that the interval between the two adjacent reference frames meets a second preset condition.

8. The method according to claim 7, wherein the first type of reference frame corresponds to a jittered frame that is a first type of jittered frame, and the second type of reference frame corresponds to a jittered frame that is a second type of jittered frame;

the step of determining the anti-shake field angle of the shake frame according to the out-of-range angle of the reference frame corresponding to the shake frame includes:

determining an anti-shake field angle of the first type of shaking frames according to the out-of-bound angles of two adjacent first type of reference frames corresponding to the first type of shaking frames and the frame sequence of the first type of shaking frames in a sub-out-of-bound area, wherein the sub-out-bound area is formed by video frames between the two adjacent first type of reference frames corresponding to the first type of shaking frames and the two adjacent first type of reference frames;

and determining the number of interval frames of the second type of jittering frames and the corresponding second type of reference frames, and determining the anti-jittering field angle of the second type of jittering frames according to the number of interval frames and the out-of-range angle of the second type of reference frames corresponding to the second type of jittering frames.

9. The method according to claim 8, wherein the determining an anti-shake field angle of the first-class jittered frame according to the out-of-bounds angles of two adjacent first-class reference frames corresponding to the first-class jittered frames and the sequence of frames of the first-class jittered frames in the sub-out-of-bounds area comprises:

determining an offset coefficient according to the out-of-range angle of two adjacent first-class reference frames corresponding to the first-class jitter frames and the actual frame number;

determining the anti-shake offset of the first type of shaking frame according to the offset coefficient and the frame sequence of the first type of shaking frame in the sub-out boundary area;

and determining the anti-shake field angle of the first type of shaking frame according to the anti-shake offset and the out-of-range angles of the two adjacent first type of reference frames corresponding to the first type of shaking frame.

10. The method according to claim 6, wherein the step of determining, for the first-type reference frame, a video frame between two adjacent first-type reference frames as the dither frame corresponding to the two adjacent first-type reference frames comprises:

for a first-class reference frame, determining a video frame between two adjacent first-class reference frames and the two adjacent first-class reference frames as a sub-boundary region;

for a sub-out-bound area without an adjacent sub-out-bound area, determining a video frame between two first-class reference frames in the sub-out-bound area as a jitter frame corresponding to the two first-class reference frames;

for a sub-out-bound area with adjacent sub-out-bound areas, combining a plurality of adjacent sub-out-bound areas into one out-bound area, and dividing the out-bound area into at least one section, wherein the length of the section is greater than that of the sub-out-bound area;

for each block section, calculating a predicted out-of-bound angle of a first type reference frame between a head first type reference frame and a tail first type reference frame according to the out-of-bound angles of the head first type reference frame and the tail first type reference frame in the block section;

and if the predicted out-of-bounds angle is smaller than the actual out-of-bounds angle, determining the video frame between the head and the tail of the two first-type reference frames as the jitter frame corresponding to the head and the tail of the two first-type reference frames.

11. The method according to claim 10, wherein the first type of reference frame corresponds to a jittered frame that is a first type of jittered frame, and the second type of reference frame corresponds to a jittered frame that is a second type of jittered frame;

determining an anti-shake field angle of the first type of shaking frame according to the out-of-range angles of the first and last two first type of reference frames corresponding to the first type of shaking frame and the frame sequence of the first type of shaking frame in the interval;

and determining the number of interval frames of the second type of jittering frames and the corresponding second type of reference frames, and determining the anti-jittering field angle of the second type of jittering frames according to the number of interval frames and the out-of-range angle of the first reference frames corresponding to the second type of jittering frames.

12. The method according to any one of claims 6 to 11, wherein the anti-shake processing the video to be processed according to the anti-shake field angle of the shake frame comprises:

13. The method according to claim 12, wherein the performing anti-shake processing on the video to be processed according to the anti-shake field angle of the first type of target shake frame and the anti-shake field angle of the second type of target shake frame comprises:

performing angle transformation processing on a first field angle curve according to the anti-shake field angle of the first type of target shaking frame and the anti-shake field angle of the second type of target shaking frame to obtain a second field angle curve, wherein the first field angle curve is constructed according to the out-of-range angle of each video frame in the video to be processed;

adjusting an out-of-range angle, which is positioned outside the preset field angle section, in the second field angle curve according to the preset field angle section to obtain a third field angle curve;

14. An anti-shake processing apparatus for video, comprising:

the first data acquisition module is used for acquiring a video to be processed and shooting a motion track of a shooting device when the video to be processed is shot;

the second data acquisition module is used for acquiring an out-of-range angle of a video frame in the video to be processed according to the motion track;

15. A computer-readable storage medium having stored thereon a computer program for causing a computer to execute an anti-shake processing method for video according to any one of claims 1 through 13, when the computer program runs on the computer.

16. An electronic device comprising a processor and a memory, the memory storing a computer program, wherein the processor is configured to execute the anti-shake processing method for video according to any one of claims 1 to 13 by calling the computer program.