CN112734653A - Motion smoothing processing method, device and equipment for video image and storage medium - Google Patents
Motion smoothing processing method, device and equipment for video image and storage medium Download PDFInfo
- Publication number
- CN112734653A CN112734653A CN202011541615.7A CN202011541615A CN112734653A CN 112734653 A CN112734653 A CN 112734653A CN 202011541615 A CN202011541615 A CN 202011541615A CN 112734653 A CN112734653 A CN 112734653A
- Authority
- CN
- China
- Prior art keywords
- quaternion
- smooth
- queue
- current
- video frame
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000009499 grossing Methods 0.000 title claims abstract description 75
- 238000003672 processing method Methods 0.000 title description 4
- 238000012545 processing Methods 0.000 claims abstract description 68
- 238000001914 filtration Methods 0.000 claims abstract description 37
- 238000000034 method Methods 0.000 claims abstract description 33
- 238000004590 computer program Methods 0.000 claims description 15
- 230000008569 process Effects 0.000 claims description 3
- 238000012935 Averaging Methods 0.000 claims 1
- 230000000694 effects Effects 0.000 abstract description 11
- 238000005259 measurement Methods 0.000 description 5
- 230000006870 function Effects 0.000 description 4
- 238000010586 diagram Methods 0.000 description 3
- 230000007704 transition Effects 0.000 description 3
- 230000008859 change Effects 0.000 description 2
- 230000008030 elimination Effects 0.000 description 2
- 238000003379 elimination reaction Methods 0.000 description 2
- 230000009467 reduction Effects 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000010845 search algorithm Methods 0.000 description 1
- 230000000087 stabilizing effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/70—Denoising; Smoothing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/18—Image warping, e.g. rearranging pixels individually
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/68—Control of cameras or camera modules for stable pick-up of the scene, e.g. compensating for camera body vibrations
- H04N23/682—Vibration or motion blur correction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Studio Devices (AREA)
- Image Processing (AREA)
Abstract
The invention is suitable for the technical field of video image processing, and provides a method, a device, equipment and a storage medium for motion smoothing processing of video images, wherein the method comprises the following steps: the method comprises the steps of smoothing current original quaternion corresponding to an obtained current video frame of a video stream to obtain corresponding current smooth quaternion, carrying out smooth filtering processing on a smooth quaternion queue of a buffer frame according to the current original quaternion and the current smooth quaternion, and outputting a smooth video frame image corresponding to the smooth quaternion corresponding to a queue head in the processed smooth quaternion queue when the queue length of the processed smooth quaternion queue exceeds the frame number of the buffer frame, so that the smooth effect of the video image and the anti-shake real-time performance of the video image in various complex motion scenes are improved.
Description
Technical Field
The invention belongs to the technical field of video image processing, and particularly relates to a method, a device, equipment and a storage medium for motion smoothing processing of a video image.
Background
With the development of science and technology and the progress of society, mobile devices become an indispensable part of people's life, and people can take video shots through the mobile devices, record the drips in life to share with friends, so that video shots become a mode for many users to share and communicate socially. Although the camera configuration in the mobile device is higher and higher, so that the video image picture shot by the mobile device in a standing state is clear and high in quality, when a user holds the mobile device by hand to shoot videos in walking or running or other various complex motion scenes (for example, when the user drives a vehicle and places the mobile device on the vehicle), the mobile device is unstable, at the moment, the recorded video data often has a shaking phenomenon, the video image is easily blurred, and the shot video quality effect is not ideal.
In the invention patent application document with the patent publication number of US10027893B2, a method for stabilizing the real-time video of a mobile device based on vehicle-mounted motion sensing is disclosed, which realizes that the current angular speed can be timely adjusted according to the future motion situation, and the situation that the angular speed changes unsmooth due to sudden out-of-bounds is avoided, but the scheme is difficult to balance motion following and picture jitter elimination under the complex and violent motion conditions.
Disclosure of Invention
The invention aims to provide a motion smoothing processing method, a motion smoothing processing device, motion smoothing processing equipment and a storage medium of a video image, and aims to solve the problems of low jitter elimination real-time performance and poor smoothing effect of the video image in a complex motion scene in the prior art.
In one aspect, the present invention provides a method for motion smoothing of video images, the method comprising the steps of:
acquiring a current camera attitude corresponding to a current video frame of a video stream, and expressing the current camera attitude by a quaternion to obtain a current original quaternion corresponding to the current video frame;
obtaining a smooth quaternion of a previous video frame corresponding to the current video frame from a smooth quaternion queue of a buffer frame, and smoothing the current original quaternion according to the smooth quaternion of the previous video frame to obtain a current smooth quaternion corresponding to the current video frame;
according to the current original quaternion and the current smooth quaternion, performing smooth filtering processing on the smooth quaternion queue to obtain a processed smooth quaternion queue;
and when the queue length of the processed smooth quaternion queue exceeds the frame number of the buffer frame, outputting a smooth video frame image corresponding to a smooth quaternion corresponding to the queue head in the processed smooth quaternion queue.
Preferably, the step of obtaining the smooth quaternion of the previous video frame corresponding to the current video frame from the smooth quaternion queue of the buffer frame, and smoothing the current original quaternion according to the smooth quaternion of the previous video frame includes:
and performing spherical linear interpolation processing on the current original quaternion by using the smooth quaternion of the previous video frame.
Further preferably, the step of performing spherical linear interpolation processing on the current original quaternion by using the smoothed quaternion of the previous video frame includes:
and performing spherical linear interpolation processing on the current original quaternion by using the smooth quaternion of the previous video frame through a preset smoothing formula v (k) ═ slerp (v (k-1), p (k), weight), wherein p (k) is the original quaternion of the kth frame, v (k) is the smooth quaternion of the kth frame, v (k-1) is the smooth quaternion of the kth frame, weight is the interpolation weight of p (k) during the spherical linear interpolation processing, and k is an integer greater than 1.
Preferably, the step of performing smooth filtering processing on the smooth quaternion queue according to the current original quaternion and the current smooth quaternion to obtain a processed smooth quaternion queue includes:
according to an original quaternion queue of a buffer frame and a corresponding smooth quaternion queue, carrying out follow-up adjustment on a smooth quaternion corresponding to each video frame in the smooth quaternion queue to obtain an adjusted smooth quaternion queue;
updating the original quaternion queue and the adjusted smooth quaternion queue respectively according to the current original quaternion and the current smooth quaternion to obtain an updated smooth quaternion queue;
and carrying out mean filtering processing on the updated smooth quaternion queue to obtain a processed smooth quaternion queue.
Further preferably, the step of performing follow-up adjustment on the smooth quaternion corresponding to each video frame in the smooth quaternion queue includes:
according to the original quaternion corresponding to each video frame in the original quaternion queue, passing through a preset following formula v (m) -slerp (v (m), p (m))smooth,weightfollow) Performing follow-up adjustment on a smooth quaternion corresponding to each video frame in the smooth quaternion queue, wherein v (m) is the smooth quaternion of the mth frame in the smooth quaternion queue, and p (m)smoothRepresenting the original quaternion, weight, of the mth frame after the unconstrained iterative mean filtering of the original quaternion queuefollowTo follow the weights, m is an integer greater than 1.
Further preferably, the step of performing mean filtering processing on the updated smoothed quaternion queue includes:
and carrying out iterative mean filtering processing on the updated smooth quaternion queue.
Preferably, after the step of smoothing the video image in the video stream by the smoothing quaternion corresponding to the queue head in the smoothing quaternion queue, the method further includes:
and updating the original quaternion queue and the smooth quaternion queue.
In another aspect, the present invention provides an apparatus for motion smoothing of video images, the apparatus comprising:
the first parameter obtaining unit is used for obtaining a current camera attitude corresponding to a current video frame of a video stream, and expressing the current camera attitude through a quaternion to obtain a current original quaternion corresponding to the current video frame;
a second parameter obtaining unit, configured to obtain a smooth quaternion of a previous video frame corresponding to the current video frame from a smooth quaternion queue of a buffer frame, and perform smoothing processing on the current original quaternion according to the smooth quaternion of the previous video frame to obtain a current smooth quaternion corresponding to the current video frame;
the queue smoothing filtering unit is used for performing smoothing filtering processing on the smoothing quaternion queue according to the current original quaternion and the current smoothing quaternion to obtain a processed smoothing quaternion queue; and
and the smooth image output unit is used for outputting a smooth video frame image corresponding to the smooth quaternion corresponding to the queue head in the processed smooth quaternion queue when the queue length of the processed smooth quaternion queue exceeds the frame number of the buffer frame.
Preferably, the second parameter obtaining unit includes:
and the parameter obtaining subunit is used for performing spherical linear interpolation processing on the current original quaternion by using the smooth quaternion of the previous video frame.
Preferably, the queue smoothing filter unit includes:
the queue following and adjusting unit is used for following and adjusting the smooth quaternion corresponding to each video frame in the smooth quaternion queue according to the original quaternion queue of the buffer frame and the corresponding smooth quaternion queue to obtain an adjusted smooth quaternion queue;
a first queue updating unit, configured to update the original quaternion queue and the adjusted smooth quaternion queue respectively according to the current original quaternion and the current smooth quaternion, so as to obtain an updated smooth quaternion; and
and the queue filtering processing unit is used for carrying out mean filtering processing on the updated smooth quaternion queue to obtain a processed smooth quaternion queue.
Further preferably, the queue following adjustment unit includes:
a following adjustment subunit, configured to, according to an original quaternion corresponding to each video frame in the original quaternion queue, follow a preset following formula v (m) ═ slerp (v (m), p (m))smooth,weightfollow) Performing follow-up adjustment on a smooth quaternion corresponding to each video frame in the smooth quaternion queue, wherein v (m) is the smooth quaternion of the mth frame in the smooth quaternion queue, and p (m)smoothRepresenting the original quaternion, weight, of the mth frame after the unconstrained iterative mean filtering of the original quaternion queuefollowTo follow the weights, m is an integer greater than 1.
Further preferably, the queue filter processing unit includes:
and the filtering processing subunit is used for performing iterative mean filtering processing on the updated smooth quaternion queue.
Preferably, the apparatus further comprises:
and the second queue updating unit is used for updating the original quaternion queue and the smooth quaternion queue.
In another aspect, the present invention further provides a video image processing apparatus, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method for motion smoothing of video images as described above when executing the computer program.
In another aspect, the present invention further provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the computer program implements the steps of the method for motion smoothing of video images.
The method comprises the steps of smoothing the current original quaternion corresponding to the current video frame of the acquired video stream to obtain the corresponding current smooth quaternion, carrying out smooth filtering processing on a smooth quaternion queue of a buffer frame according to the current original quaternion and the current smooth quaternion, and outputting a smooth video frame image corresponding to the smooth quaternion corresponding to the queue head in the processed smooth quaternion queue when the queue length of the processed smooth quaternion queue exceeds the frame number of the buffer frame, so that the smooth effect of the video image and the anti-shake real-time performance of the video image in various complex motion scenes are improved.
Drawings
Fig. 1 is a flowchart of an implementation of a method for motion smoothing of video images according to an embodiment of the present invention;
fig. 2 is a flowchart of an implementation of a motion smoothing method for video images according to a second embodiment of the present invention;
fig. 3 is a schematic structural diagram of a motion smoothing apparatus for video images according to a third embodiment of the present invention;
fig. 4 is a schematic structural diagram of a motion smoothing device for video images according to a third embodiment of the present invention; and
fig. 5 is a schematic structural diagram of a video image processing apparatus according to a fourth embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
The following detailed description of specific implementations of the present invention is provided in conjunction with specific embodiments:
the first embodiment is as follows:
fig. 1 shows an implementation flow of a method for motion smoothing processing of a video image according to a first embodiment of the present invention, and for convenience of description, only the relevant portions of the embodiment of the present invention are shown, and the following details are described below:
in step S101, a current camera pose corresponding to a current video frame of the video stream is obtained, and the current camera pose is represented by a quaternion to obtain a current original quaternion corresponding to the current video frame.
The embodiment of the invention is suitable for video image processing equipment for video display, acquisition and the like. In the embodiment of the invention, after the video recording is started, the current camera attitude corresponding to the current video frame of the video stream is obtained, the camera attitude is three-dimensional space information, and the quaternion is used for representing the three-dimensional space information to obtain the current original quaternion corresponding to the current video frame.
In step S102, a smooth quaternion of a previous video frame corresponding to the current video frame is obtained from the smooth quaternion queue of the buffer frame, and the current original quaternion is smoothed according to the smooth quaternion of the previous video frame, so as to obtain a current smooth quaternion corresponding to the current video frame.
In the embodiment of the invention, the original quaternion and the smooth quaternion related to the video frame are buffered in a queue mode by setting the buffer frame. After the current original quaternion corresponding to the current video frame is obtained, the smooth quaternion of the previous video frame corresponding to the current video frame is obtained from the smooth quaternion queue of the buffer frame, and the current original quaternion is subjected to smoothing processing according to the smooth quaternion of the previous video frame to obtain the current smooth quaternion corresponding to the current video frame, wherein the previous video frame is one frame before the current video frame in two adjacent frames. And when the current video frame is a first frame video image of the video stream, the current smooth quaternion after the smoothing processing is the current original quaternion, namely v (1) is p (1), p (1) is the original quaternion of the first video frame, and v (1) is the smooth quaternion of the first video frame.
When the current original quaternion is smoothed according to the smooth quaternion of the previous video frame, preferably, the smooth quaternion of the previous video frame is used for performing spherical linear interpolation processing on the current original quaternion, so that the smoothness of the transition of the front and rear video images is improved, and the presented video image is more stable.
In step S103, according to the current original quaternion and the current smooth quaternion, the smooth quaternion queue is subjected to a smooth filtering process to obtain a processed smooth quaternion queue.
In the embodiment of the present invention, a specific implementation flow of performing the smoothing filtering process on the smoothed quaternion queue according to the current original quaternion and the current smoothed quaternion is described in detail in the second embodiment.
In step S104, when the queue length of the processed smooth quaternion queue exceeds the number of frames of the buffer frame, the smooth video frame image corresponding to the smooth quaternion corresponding to the queue head in the processed smooth quaternion queue is output.
In the embodiment of the invention, the queue length of the smooth quaternion queue is detected in real time, and when the queue length of the processed smooth quaternion queue exceeds the number of preset buffer frames, the smooth video frame image corresponding to the smooth quaternion corresponding to the queue head in the smooth quaternion queue is output.
Preferably, the number of buffered frames may be set to 5, so as to meet the hardware device requirement that the number of buffered frames is small.
After outputting the smooth video frame image corresponding to the smooth quaternion corresponding to the queue head in the processed smooth quaternion queue, preferably, dequeuing the original quaternion queue head, dequeuing the processed smooth quaternion queue head, and finishing updating the original quaternion queue and the smooth quaternion queue, so that the anti-shake real-time performance of the video image is improved by setting the queue in the buffer frame.
In the embodiment of the invention, the current original quaternion corresponding to the current video frame of the acquired video stream is smoothed to obtain the corresponding current smooth quaternion, the smooth quaternion queue of the buffer frame is smoothed according to the current original quaternion and the current smooth quaternion, and when the queue length of the smoothed quaternion queue exceeds the frame number of the buffer frame, the smoothed video frame image corresponding to the smooth quaternion corresponding to the queue head in the smoothed quaternion queue is output, so that the smoothing effect of the video image and the anti-shake real-time performance of the video image in various complex motion scenes are improved.
Example two:
fig. 2 shows an implementation flow of a motion smoothing method for video images according to a second embodiment of the present invention, and for convenience of description, only the relevant parts related to the second embodiment of the present invention are shown, which is detailed as follows:
in step S201, a current camera pose corresponding to a current video frame of the video stream is obtained, and the current camera pose is represented by a quaternion, so as to obtain a current original quaternion corresponding to the current video frame.
The embodiment of the invention is suitable for video image processing equipment for video display, acquisition and the like. In the embodiment of the invention, after the video recording is started, the current camera attitude corresponding to the current video frame of the video stream is obtained, the camera attitude is three-dimensional space information, and the quaternion is used for representing the three-dimensional space information to obtain the current original quaternion corresponding to the current video frame.
In step S202, a smooth quaternion of a previous video frame corresponding to the current video frame is obtained from the smooth quaternion queue of the buffer frame, and the smooth quaternion of the previous video frame is used to perform spherical linear difference processing on the current original quaternion, so as to obtain a current smooth quaternion corresponding to the current video frame.
In the embodiment of the invention, the original quaternion and the smooth quaternion related to the video frame are buffered in a queue mode by setting the buffer frame. After the current original quaternion corresponding to the current video frame is obtained, the smooth quaternion of the last video frame corresponding to the current video frame is obtained from the smooth quaternion queue of the buffer frame, and Spherical linear difference (Slerp for short) processing is carried out on the current original quaternion according to the smooth quaternion of the last video frame to obtain the current smooth quaternion corresponding to the current video frame, wherein the last video frame is one frame before the current video frame in two adjacent frames. And when the current video frame is a first frame video image of the video stream, the current smooth quaternion after the smoothing processing is the current original quaternion, namely v (1) is p (1), p (1) is the original quaternion of the first video frame, and v (1) is the smooth quaternion of the first video frame.
When the smooth quaternion of the previous video frame is used to perform spherical linear difference processing on the current original quaternion, preferably, the smooth quaternion of the previous video frame is used to perform spherical linear interpolation processing on the current original quaternion through a preset smoothing formula v (k) -slerp (v (k-1), p (k), weight), where p (k) is the original quaternion of the kth frame, v (k) is the smooth quaternion of the kth frame, v (k-1) is the smooth quaternion of the kth-1 frame, weight is the interpolation weight of p (k) when performing spherical linear interpolation processing, and k is an integer greater than 1, so as to improve the smoothness of the transition of the previous and next two frames of video images, and make the presented video image more stable.
When the current original quaternion is processed by spherical linear interpolation according to a preset smoothing formula v (k) -slope (v (k-1), p (k), weight), it is further preferable that the relative quaternion Δ q between the smoothed quaternion v (k-1) of the previous frame and the original quaternion p (k) is calculated as an initial value, and then a metric value rate of the current video frame from the boundary of the original picture is calculated when the relative rotation amount is Δ q, wherein the rate is 0 when the polygon is clipped in the inner frame, the rate is 1 when the polygon is out of the outer frame, and the metric value is larger when the polygon is closer to the outer frame between the inner frame and the outer frame, and then the metric value is calculated according to the metric value rate and the weight calculation formula
Calculating weight, wherein min _ weight represents the minimum weight, generally between 0.001 and 0.4, inner _ rate represents a metric value corresponding to an inner frame, the value range is between 0 and 1, the larger the inner _ rate value is, the higher the smoothness is but the following is slower, the larger the value is, when the value is out of bounds, larger jitter occurs, the smaller the value is, the smoothness is relatively lower but the following is tighter, α is the sharpness for controlling weight change, and the value generally ranges between 0.5 and 4, and finally, smoothing is performed on the current original quaternion by using a smoothing formula v (k) slope (v (k-1), p (k), and weight), so as to obtain the current smooth quaternion corresponding to the current video frame, thereby further improving the smoothness of transition of the two frames of video images before and after, and enabling the presented video image to be more stable.
In step S203, according to the original quaternion queue of the buffer frame and the corresponding smooth quaternion queue, the smooth quaternion corresponding to each video frame in the smooth quaternion queue is followed and adjusted to obtain an adjusted smooth quaternion queue.
In the embodiment of the invention, according to the original quaternion queue of the buffer frame and the corresponding smooth quaternion queue, the smooth quaternion corresponding to each video frame in the smooth quaternion queue is adjusted in a following manner, so that the smooth quaternion is close to the corresponding original quaternion, and the adjusted smooth quaternion queue is obtained.
When the smooth quaternion corresponding to each video frame in the smooth quaternion queue is followed and adjusted, preferably, according to the original quaternion corresponding to each video frame in the original quaternion queue, the following formula v (m) is slerp (v (m), p (m)smooth,weightfollow) Follow-up adjustment is carried out on the smooth quaternion corresponding to each video frame in the smooth quaternion queue, wherein v (m) is the smooth quaternion of the mth frame in the smooth quaternion queue, p (m)smoothRepresenting the original quaternion, weight, of the mth frame after the unconstrained iterative mean filtering of the original quaternion queuefollowTo follow the weights, m is an integer greater than 1, thereby improving the smoothing effect of subsequent video images.
After passing through a preset following formula v (m) ═ slerp (v (m), p (m)smooth,weightfollow) When the smooth quaternion corresponding to each video frame in the smooth quaternion queue is adjusted in a following mode, further preferably, a relative quaternion delta q between the smooth quaternion of each video frame in the smooth quaternion queue and the original quaternion corresponding to the original quaternion queue is calculated, then the measurement value of the polygon to the original picture boundary is calculated when the relative rotation amount of each video frame in the cache frame is the corresponding delta q, the measurement value average value rate _ avr is calculated according to the measurement values of all frames in the cache frame, and then the measurement value average value rate _ avr and a following weight formula are used for following the measurement value average value
Calculating weight following the original quaternionfollowWherein, min _ weightfollowRepresents the minimum follow-up weight, generally between 0.002 and 0.4, beta is the sharpness of the change of the control weight, generally between 0.5 and 4, and finally follows the formula v (m) slerp (v (m), p (m)smooth,weightfollow) Adjusting buffer frame smooth quaternion queue to further improve the flatness of subsequent video imagesAnd (4) a sliding effect.
In step S204, the original quaternion queue and the adjusted smooth quaternion queue are updated according to the current original quaternion and the current smooth quaternion, respectively, so as to obtain an updated smooth quaternion queue.
In the embodiment of the invention, the current original quaternion of the current video frame is added into an original quaternion queue, the current smooth quaternion is added into an adjusted smooth quaternion queue, the two queues are updated, and the updated smooth quaternion queue is obtained.
In step S205, mean filtering processing is performed on the updated smooth quaternion queue to obtain a processed smooth quaternion queue.
In the embodiment of the invention, mean filtering processing is carried out on the updated smooth quaternion queue to obtain the processed smooth quaternion queue, so that the noise reduction of the video image corresponding to each smooth quaternion in the smooth quaternion queue and the smoothness of the camera posture are realized.
When the mean filtering processing is performed on the updated smooth quaternion queue, preferably, the iterative mean filtering processing is performed on the updated smooth quaternion queue, and whether the smooth quaternion of each video frame in the queue can cause out-of-bounds is judged after each iterative smoothing, namely whether the projection of the cutting area of each video frame in the original picture after smoothing exceeds the picture effective range is judged, if so, the quaternion which is closest to the smooth quaternion and is ensured not to be out of bounds is obtained through a binary search algorithm, and the quaternion is used as the smoothed result of the frame corresponding to the smooth quaternion, so that the smoothing and noise reduction effects of the video image corresponding to the smooth quaternion are improved.
In step S206, when the queue length of the processed smooth quaternion queue exceeds the number of frames of the buffer frame, the smooth video frame image corresponding to the smooth quaternion corresponding to the queue head in the processed smooth quaternion queue is output.
In the embodiment of the invention, the queue length of the smooth quaternion queue is detected in real time, and when the queue length of the processed smooth quaternion queue exceeds the number of preset buffer frames, the smooth video frame image corresponding to the smooth quaternion corresponding to the queue head in the smooth quaternion queue is output.
Preferably, the number of buffered frames is set to 5, so as to meet the requirement of hardware equipment with less buffered frames.
After outputting the smooth video frame image corresponding to the smooth quaternion corresponding to the queue head in the processed smooth quaternion queue, preferably, dequeuing the original quaternion queue head, dequeuing the processed smooth quaternion queue head, and finishing updating the original quaternion queue and the smooth quaternion queue, so that the anti-shake real-time performance of the video image is improved by setting the queue in the buffer frame.
In the embodiment of the invention, the current original quaternion corresponding to the current video frame of the acquired video stream is smoothed to obtain the corresponding current smooth quaternion, and according to the original quaternion corresponding to each video frame in the original quaternion queue of the buffer frame, the smooth quaternion corresponding to each video frame in the smooth quaternion queue is adjusted along with the current original quaternion and the current smooth quaternion, the original quaternion queue and the adjusted smooth quaternion queue are respectively updated, the updated smooth quaternion queue is processed by iterative mean filtering, when the queue length of the processed smooth quaternion queue exceeds the frame number of the buffer frame, outputting the smooth video frame image corresponding to the smooth quaternion corresponding to the queue head in the processed smooth quaternion queue, therefore, the smoothing effect of the video image and the anti-shake real-time performance of the video image in various complex motion scenes are improved.
Example three:
fig. 3 shows a structure of a video image motion smoothing apparatus according to a third embodiment of the present invention, and for convenience of description, only the portions related to the third embodiment of the present invention are shown, which include:
the first parameter obtaining unit 31 is configured to obtain a current camera pose corresponding to a current video frame of the video stream, and express the current camera pose through a quaternion to obtain a current original quaternion corresponding to the current video frame.
The second parameter obtaining unit 32 is configured to obtain a smooth quaternion of a previous video frame corresponding to the current video frame from the smooth quaternion queue of the buffer frame, and perform smoothing processing on the current original quaternion according to the smooth quaternion of the previous video frame to obtain a current smooth quaternion corresponding to the current video frame.
And a queue smoothing filtering unit 33, configured to perform smoothing filtering processing on the smooth quaternion queue according to the current original quaternion and the current smooth quaternion, so as to obtain a processed smooth quaternion queue.
And a smooth image output unit 34, configured to output a smooth video frame image corresponding to the smooth quaternion corresponding to the head of the queue in the processed smooth quaternion queue when the queue length of the processed smooth quaternion queue exceeds the number of frames of the buffer frame.
As shown in fig. 4, preferably, the motion smoothing processing apparatus for video images according to the embodiment of the present invention further includes:
and a second queue updating unit 35, configured to update the original quaternion queue and the smoothed quaternion queue.
The second parameter obtaining unit 32 includes:
and a parameter obtaining subunit 321, configured to perform spherical linear interpolation processing on the current original quaternion by using the smoothed quaternion of the previous video frame.
The queue smoothing filter unit 33 includes:
a queue following adjustment unit 331, configured to perform following adjustment on a smooth quaternion corresponding to each video frame in a smooth quaternion queue according to an original quaternion queue of the buffer frame and a corresponding smooth quaternion queue, to obtain an adjusted smooth quaternion queue;
a first queue updating unit 332, configured to update the original quaternion queue and the adjusted smooth quaternion queue respectively according to the current original quaternion and the current smooth quaternion, so as to obtain an updated smooth quaternion; and
and a queue filter processing unit 333, configured to perform mean filter processing on the updated smooth quaternion queue to obtain a processed smooth quaternion queue.
The queue following adjustment unit 331 includes:
a following adjustment subunit 3311, configured to follow the original quaternion corresponding to each video frame in the original quaternion queue according to a preset following formula v (m) ═ slerp (v (m), p (m))smooth,weightfollow) Follow-up adjustment is carried out on the smooth quaternion corresponding to each video frame in the smooth quaternion queue, wherein v (m) is the smooth quaternion of the mth frame in the smooth quaternion queue, p (m)smoothRepresenting the original quaternion, weight, of the mth frame after the unconstrained iterative mean filtering of the original quaternion queuefollowTo follow the weights, m is an integer greater than 1.
The queue filter processing unit 333 includes:
and the filtering processing subunit 3331 is configured to perform iterative mean filtering processing on the updated smooth quaternion queue.
In the embodiment of the present invention, each unit of the motion smoothing device for video images may be implemented by a corresponding hardware or software unit, and each unit may be an independent software or hardware unit, or may be integrated into a software or hardware unit, which is not limited herein. Specifically, the implementation of each unit can refer to the description of the foregoing method embodiment, and is not repeated herein.
Example four:
fig. 5 shows a configuration of a video image processing apparatus according to a fourth embodiment of the present invention, and for convenience of explanation, only a part related to the embodiment of the present invention is shown.
The video image processing apparatus 5 of the embodiment of the present invention includes a processor 50, a memory 51, and a computer program 52 stored in the memory 51 and executable on the processor 50. The processor 50, when executing the computer program 52, implements the steps in the above-described embodiment of the method for motion smoothing of video images, such as the steps S101 to S104 shown in fig. 1. Alternatively, the processor 50, when executing the computer program 52, implements the functions of the units in the above-described device embodiments, such as the functions of the units 31 to 34 shown in fig. 3.
In the embodiment of the invention, the current original quaternion corresponding to the current video frame of the acquired video stream is smoothed to obtain the corresponding current smooth quaternion, the smooth quaternion queue of the buffer frame is smoothed according to the current original quaternion and the current smooth quaternion, and when the queue length of the smoothed quaternion queue exceeds the frame number of the buffer frame, the smoothed video frame image corresponding to the smooth quaternion corresponding to the queue head in the smoothed quaternion queue is output, so that the smoothing effect of the video image and the anti-shake real-time performance of the video image in various complex motion scenes are improved.
The video image processing equipment provided by the embodiment of the invention can be a smart phone with a camera and a tablet computer. The steps implemented when the processor 50 in the video image processing apparatus 5 executes the computer program 52 to implement the motion smoothing processing method for the video image can refer to the description of the foregoing method embodiments, and are not repeated herein.
Example five:
in an embodiment of the present invention, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps in the above-described embodiment of the method for motion smoothing of video images, for example, steps S101 to S104 shown in fig. 1. Alternatively, the computer program may be adapted to perform the functions of the units of the above-described device embodiments, such as the functions of the units 31 to 34 shown in fig. 3, when executed by the processor.
In the embodiment of the invention, the current original quaternion corresponding to the current video frame of the acquired video stream is smoothed to obtain the corresponding current smooth quaternion, the smooth quaternion queue of the buffer frame is smoothed according to the current original quaternion and the current smooth quaternion, and when the queue length of the smoothed quaternion queue exceeds the frame number of the buffer frame, the smoothed video frame image corresponding to the smooth quaternion corresponding to the queue head in the smoothed quaternion queue is output, so that the smoothing effect of the video image and the anti-shake real-time performance of the video image in various complex motion scenes are improved.
The computer readable storage medium of the embodiments of the present invention may include any entity or device capable of carrying computer program code, a recording medium, such as a ROM/RAM, a magnetic disk, an optical disk, a flash memory, or the like.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents and improvements made within the spirit and principle of the present invention are intended to be included within the scope of the present invention.
Claims (12)
1. A method for motion smoothing of video images, the method comprising the steps of:
acquiring a current camera attitude corresponding to a current video frame of a video stream, and expressing the current camera attitude by a quaternion to obtain a current original quaternion corresponding to the current video frame;
obtaining a smooth quaternion of a previous video frame corresponding to the current video frame from a smooth quaternion queue of a buffer frame, and smoothing the current original quaternion according to the smooth quaternion of the previous video frame to obtain a current smooth quaternion corresponding to the current video frame;
according to the current original quaternion and the current smooth quaternion, performing smooth filtering processing on the smooth quaternion queue to obtain a processed smooth quaternion queue;
and when the queue length of the processed smooth quaternion queue exceeds the frame number of the buffer frame, outputting a smooth video frame image corresponding to a smooth quaternion corresponding to the queue head in the processed smooth quaternion queue.
2. The method of claim 1, wherein the step of obtaining the smoothed quaternion of the previous video frame corresponding to the current video frame from the smoothed quaternion queue of the buffered frame, and the step of smoothing the current original quaternion according to the smoothed quaternion of the previous video frame comprises:
and performing spherical linear interpolation processing on the current original quaternion by using the smooth quaternion of the previous video frame.
3. The method of claim 2, wherein the step of performing spherical linear interpolation on the current raw quaternion using the smoothed quaternion of the previous video frame comprises:
and performing spherical linear interpolation processing on the current original quaternion by using the smooth quaternion of the previous video frame through a preset smoothing formula v (k) ═ slerp (v (k-1), p (k), weight), wherein p (k) is the original quaternion of the kth frame, v (k) is the smooth quaternion of the kth frame, v (k-1) is the smooth quaternion of the kth frame, weight is the interpolation weight of p (k) during the spherical linear interpolation processing, and k is an integer greater than 1.
4. The method of claim 1, wherein the step of performing a smoothing filtering process on the smoothed quaternion queue according to the current original quaternion and the current smoothed quaternion to obtain a processed smoothed quaternion queue comprises:
according to an original quaternion queue of a buffer frame and a corresponding smooth quaternion queue, carrying out follow-up adjustment on a smooth quaternion corresponding to each video frame in the smooth quaternion queue to obtain an adjusted smooth quaternion queue;
updating the original quaternion queue and the adjusted smooth quaternion queue respectively according to the current original quaternion and the current smooth quaternion to obtain an updated smooth quaternion queue;
and carrying out mean filtering processing on the updated smooth quaternion queue to obtain a processed smooth quaternion queue.
5. The method of claim 4, wherein the step of performing a follow-up adjustment on the smoothed quaternion corresponding to each video frame in the smoothed quaternion queue comprises:
according to the original quaternion corresponding to each video frame in the original quaternion queue, passing through a preset following formula v (m) -slerp (v (m), p (m))smooth,weightfollow) Performing follow-up adjustment on a smooth quaternion corresponding to each video frame in the smooth quaternion queue, wherein v (m) is the smooth quaternion of the mth frame in the smooth quaternion queue, and p (m)smoothRepresenting the original quaternion, weight, of the mth frame after the unconstrained iterative mean filtering of the original quaternion queuefollowTo follow the weights, m is an integer greater than 1.
6. The method of claim 4, wherein the step of averaging the updated smoothed quaternion queue comprises:
and carrying out iterative mean filtering processing on the updated smooth quaternion queue.
7. The method of claim 1, wherein after the step of smoothing the video image in the video stream by the corresponding smoothing quaternion at the head of the queue in the smoothing quaternion queue, the method further comprises:
and updating the original quaternion queue and the smooth quaternion queue.
8. An apparatus for motion smoothing of video images, the apparatus comprising:
the first parameter obtaining unit is used for obtaining a current camera attitude corresponding to a current video frame of a video stream, and expressing the current camera attitude through a quaternion to obtain a current original quaternion corresponding to the current video frame;
a second parameter obtaining unit, configured to obtain a smooth quaternion of a previous video frame corresponding to the current video frame from a smooth quaternion queue of a buffer frame, and perform smoothing processing on the current original quaternion according to the smooth quaternion of the previous video frame to obtain a current smooth quaternion corresponding to the current video frame;
the queue smoothing filtering unit is used for performing smoothing filtering processing on the smoothing quaternion queue according to the current original quaternion and the current smoothing quaternion to obtain a processed smoothing quaternion queue; and
and the smooth image output unit is used for outputting a smooth video frame image corresponding to the smooth quaternion corresponding to the queue head in the processed smooth quaternion queue when the queue length of the processed smooth quaternion queue exceeds the frame number of the buffer frame.
9. The apparatus of claim 8, wherein the second parameter obtaining unit comprises:
and the parameter obtaining subunit is used for performing spherical linear interpolation processing on the current original quaternion by using the smooth quaternion of the previous video frame.
10. The apparatus of claim 8, wherein the queue smoothing filter unit comprises:
the queue following and adjusting unit is used for following and adjusting the smooth quaternion corresponding to each video frame in the smooth quaternion queue according to the original quaternion queue of the buffer frame and the corresponding smooth quaternion queue to obtain an adjusted smooth quaternion queue;
a first queue updating unit, configured to update the original quaternion queue and the adjusted smooth quaternion queue respectively according to the current original quaternion and the current smooth quaternion, so as to obtain an updated smooth quaternion; and
and the queue filtering processing unit is used for carrying out mean filtering processing on the updated smooth quaternion queue to obtain a processed smooth quaternion queue.
11. A video image processing apparatus comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that the processor implements the steps of the method according to any of claims 1 to 7 when executing the computer program.
12. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the steps of the method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011541615.7A CN112734653A (en) | 2020-12-23 | 2020-12-23 | Motion smoothing processing method, device and equipment for video image and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011541615.7A CN112734653A (en) | 2020-12-23 | 2020-12-23 | Motion smoothing processing method, device and equipment for video image and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112734653A true CN112734653A (en) | 2021-04-30 |
Family
ID=75604600
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011541615.7A Pending CN112734653A (en) | 2020-12-23 | 2020-12-23 | Motion smoothing processing method, device and equipment for video image and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112734653A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115866162A (en) * | 2021-09-26 | 2023-03-28 | 中移雄安信息通信科技有限公司 | Video stream generation method and device, electronic equipment and storage medium |
Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2010011050A (en) * | 2008-06-26 | 2010-01-14 | Canon Inc | Moving picture processing unit, image recording device integrated with camera, moving picture reproducing device, remote control system, moving picture processing program, storage medium, and moving picture processing system |
| WO2017038541A1 (en) * | 2015-09-03 | 2017-03-09 | ソニー株式会社 | Video processing device, video processing method, and program |
| WO2017112800A1 (en) * | 2015-12-22 | 2017-06-29 | Mobile Video Corporation | Macro image stabilization method, system and devices |
| US20180082428A1 (en) * | 2016-09-16 | 2018-03-22 | Qualcomm Incorporated | Use of motion information in video data to track fast moving objects |
| CN108632501A (en) * | 2017-03-23 | 2018-10-09 | 展讯通信（上海）有限公司 | Video anti-fluttering method and device, mobile terminal |
| CN109547692A (en) * | 2018-10-26 | 2019-03-29 | 华为技术有限公司 | Image posture antidote, equipment and terminal |
| US20190104255A1 (en) * | 2017-10-03 | 2019-04-04 | Google Llc | Video Stabilization |
| CN110049246A (en) * | 2019-04-22 | 2019-07-23 | 联想(北京)有限公司 | Video anti-fluttering method, device and the electronic equipment of electronic equipment |
| CN110430365A (en) * | 2019-08-26 | 2019-11-08 | Oppo广东移动通信有限公司 | Anti-fluttering method, device, computer equipment and storage medium |
| CN110431507A (en) * | 2018-05-31 | 2019-11-08 | 深圳市大疆创新科技有限公司 | A kind of cloud platform control method and holder |
| CN110519507A (en) * | 2019-07-23 | 2019-11-29 | 深圳岚锋创视网络科技有限公司 | A kind of camera lens smoothing processing method, device and portable terminal |
| WO2020014953A1 (en) * | 2018-07-20 | 2020-01-23 | 深圳市大疆创新科技有限公司 | Image processing method and device |
| CN111712857A (en) * | 2019-06-25 | 2020-09-25 | 深圳市大疆创新科技有限公司 | Image processing method, device, holder and storage medium |
-
2020
- 2020-12-23 CN CN202011541615.7A patent/CN112734653A/en active Pending
Patent Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2010011050A (en) * | 2008-06-26 | 2010-01-14 | Canon Inc | Moving picture processing unit, image recording device integrated with camera, moving picture reproducing device, remote control system, moving picture processing program, storage medium, and moving picture processing system |
| WO2017038541A1 (en) * | 2015-09-03 | 2017-03-09 | ソニー株式会社 | Video processing device, video processing method, and program |
| WO2017112800A1 (en) * | 2015-12-22 | 2017-06-29 | Mobile Video Corporation | Macro image stabilization method, system and devices |
| US20180082428A1 (en) * | 2016-09-16 | 2018-03-22 | Qualcomm Incorporated | Use of motion information in video data to track fast moving objects |
| CN108632501A (en) * | 2017-03-23 | 2018-10-09 | 展讯通信（上海）有限公司 | Video anti-fluttering method and device, mobile terminal |
| US20190104255A1 (en) * | 2017-10-03 | 2019-04-04 | Google Llc | Video Stabilization |
| CN110431507A (en) * | 2018-05-31 | 2019-11-08 | 深圳市大疆创新科技有限公司 | A kind of cloud platform control method and holder |
| WO2020014953A1 (en) * | 2018-07-20 | 2020-01-23 | 深圳市大疆创新科技有限公司 | Image processing method and device |
| CN109547692A (en) * | 2018-10-26 | 2019-03-29 | 华为技术有限公司 | Image posture antidote, equipment and terminal |
| CN110049246A (en) * | 2019-04-22 | 2019-07-23 | 联想(北京)有限公司 | Video anti-fluttering method, device and the electronic equipment of electronic equipment |
| CN111712857A (en) * | 2019-06-25 | 2020-09-25 | 深圳市大疆创新科技有限公司 | Image processing method, device, holder and storage medium |
| CN110519507A (en) * | 2019-07-23 | 2019-11-29 | 深圳岚锋创视网络科技有限公司 | A kind of camera lens smoothing processing method, device and portable terminal |
| CN110430365A (en) * | 2019-08-26 | 2019-11-08 | Oppo广东移动通信有限公司 | Anti-fluttering method, device, computer equipment and storage medium |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115866162A (en) * | 2021-09-26 | 2023-03-28 | 中移雄安信息通信科技有限公司 | Video stream generation method and device, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9979889B2 (en) | Combined optical and electronic image stabilization | |
| US11095833B2 (en) | Automatic composition of composite images or videos from frames captured with moving camera | |
| US9426409B2 (en) | Time-lapse video capture with optimal image stabilization | |
| CN108886584B (en) | Method and apparatus for generating high fidelity zoom for mobile video | |
| US9674438B2 (en) | Low light video image stabilization strength modulation | |
| JP6092255B2 (en) | Method and system for processing video for image stabilization using dynamic cropping | |
| US10984583B2 (en) | Reconstructing views of real world 3D scenes | |
| US9300871B2 (en) | Stationary camera detection and virtual tripod transition for video stabilization | |
| JP7224526B2 (en) | Camera lens smoothing method and mobile terminal | |
| US20130155264A1 (en) | Motion sensor based virtual tripod method for video stabilization | |
| CN108681439B (en) | Uniform display method based on frame rate control | |
| CN106550187A (en) | For the apparatus and method of image stabilization | |
| CN110572534A (en) | Digital video image stabilization method, device, equipment and storage medium of panoramic image | |
| CN115176455A (en) | Power efficient dynamic electronic image stabilization | |
| CN109561254A (en) | A kind of method, apparatus and portable terminal of panoramic video stabilization | |
| CN112734653A (en) | Motion smoothing processing method, device and equipment for video image and storage medium | |
| US10911677B1 (en) | Multi-camera video stabilization techniques | |
| WO2020125130A1 (en) | Panoramic video anti-shake method and portable terminal | |
| CN113099106B (en) | Video processing method, device, equipment and storage medium | |
| CN116489514A (en) | Video image motion smoothing method, device, equipment and storage medium | |
| CN114339101B (en) | Video recording method and equipment | |
| CN112804444A (en) | Video processing method and device, computing equipment and storage medium | |
| CN114463213A (en) | Video processing method, video processing device, terminal and storage medium | |
| US20240169498A1 (en) | Joint Video Stabilization and Motion Deblurring | |
| CN116347232A (en) | Real-time video image stabilizing method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |