CN111292357A - Video inter-frame rapid motion estimation method based on correlation filtering - Google Patents

Video inter-frame rapid motion estimation method based on correlation filtering Download PDF

Info

Publication number
CN111292357A
CN111292357A CN202010136402.XA CN202010136402A CN111292357A CN 111292357 A CN111292357 A CN 111292357A CN 202010136402 A CN202010136402 A CN 202010136402A CN 111292357 A CN111292357 A CN 111292357A
Authority
CN
China
Prior art keywords
reference frame
block
frame
search area
motion estimation
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202010136402.XA
Other languages
Chinese (zh)
Other versions
CN111292357B (en
Inventor
孙宏滨
米云琪
汪航
张旭翀
郑南宁
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Xian Jiaotong University
Original Assignee
Xian Jiaotong University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Xian Jiaotong University filed Critical Xian Jiaotong University
Priority to CN202010136402.XA priority Critical patent/CN111292357B/en
Publication of CN111292357A publication Critical patent/CN111292357A/en
Application granted granted Critical
Publication of CN111292357B publication Critical patent/CN111292357B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/262—Analysis of motion using transform domain methods, e.g. Fourier domain methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Mathematical Physics (AREA)
  • Image Analysis (AREA)
  • Compression Or Coding Systems Of Tv Signals (AREA)

Abstract

The invention discloses a video inter-frame rapid motion estimation method based on correlation filtering. The method comprises the steps of firstly preprocessing a reference frame and a reference frame, improving the image contrast by carrying out logarithmic transformation, secondly dividing the reference frame into grids with equal size, and sequentially extracting a reference block BcurAnd extracting a reference area in the reference frame according to the corresponding position of the reference block, sequentially extracting the reference block from the reference area, and calculating the motion vector through related filtering. The invention can obviously improve the robustness of the traditional motion estimation method, overcome the defect that the traditional motion estimation method can not accurately acquire the motion vector for the non-rigid motion, and reduce the operation complexity by improving the search strategy.

Description

Video inter-frame rapid motion estimation method based on correlation filtering
Technical Field
The invention belongs to the technical field of digital video image processing and display, and particularly relates to a video inter-frame rapid motion estimation method based on related filtering.
Background
With the continuous development of video processing technology and the continuous improvement of the requirements of people on video display quality, ultrahigh-definition display equipment with high refresh rate has been widely popularized and applied. The motion estimation is used as a core module of a video processing technology and is responsible for tracking the motion conditions of all objects in a video frame. The method mainly adopts an efficient block matching mode, namely, a reference frame is divided into non-overlapped reference blocks with a certain granularity, and the best matching block is searched within a certain range of the corresponding block position of the reference frame by taking the reference blocks as the reference. There are two main block matching methods: full search and fast motion estimation. A full search requires traversing all positions within the search range to find the best matching motion vector, resulting in a significant amount of computation. Therefore, a plurality of rapid motion estimation algorithms are provided, and only some key candidate positions are compared through a certain rule, so that the operation amount is greatly reduced. However, these fast motion estimation algorithms are easy to obtain a locally optimal solution, and the accuracy of the obtained motion vector is not as good as that of a full search. Current motion estimation algorithms therefore require a trade-off between computational resources and computational accuracy. The existing motion estimation algorithm usually uses the Sum of Absolute Differences (SAD) as a loss function, the SAD calculates the sum of absolute differences of pixel differences at corresponding positions in the reference block and the search block, and if non-rigid transformation (such as scaling, rotation, affine transformation, etc.) or large displacement occurs in two frames, the motion estimation method using SAD as a loss function often obtains wrong motion vectors. Meanwhile, images in the real world often have problems of low contrast, complex scene and the like, and the problems further challenge the motion estimation algorithm.
Disclosure of Invention
The invention aims to provide a video inter-frame rapid motion estimation method based on correlation filtering. The method utilizes the relevant filtering as a loss function, improves the contrast by carrying out logarithmic conversion on the input image, reduces the calculated amount by improving the searching step, and realizes more accurate and rapid motion estimation.
The invention is realized by adopting the following technical scheme:
a video interframe quick motion estimation method based on correlation filtering is characterized in that firstly, reference frames extracted from original video streams are summedPreprocessing a reference frame, improving the image contrast by carrying out logarithmic transformation, dividing the reference frame into grids with equal size, and sequentially extracting a reference block BcurAnd extracting a reference area in the reference frame according to the corresponding position of the reference block, sequentially extracting the reference block from the reference area, and calculating the motion vector through related filtering.
The invention has the further improvement that the method specifically comprises the following implementation steps:
5) sequentially extracting a reference frame and a reference frame from an original video stream, and respectively preprocessing the reference frame and the reference frame;
6) sequentially extracting a reference block B from the preprocessed reference frame and the reference framecurAnd a search area;
7) the reference block B is processed according to the searching flowcurConverting the corresponding search block in the search area into a frequency domain, and calculating the correlation through correlation filtering;
8) and finding out the coordinates (x, y) of the point with the maximum correlation, and taking the displacement between the coordinates (x, y) and the center point of the search area as a motion vector.
The further improvement of the invention is that the specific implementation method of the step 1) is as follows:
101) sequentially extracting a reference frame and a reference frame from the video stream, taking the current frame of the video stream as the reference frame, and taking the next frame as the reference frame;
102) and respectively carrying out logarithmic change on the reference frame and the reference frame so as to improve the contrast of the image.
The further improvement of the invention is that the specific implementation method of the step 2) is as follows:
201) dividing the reference frame into several rectangles with equal size and without overlapping, making the rectangles as reference block Bcur；
202) And taking the corresponding position in the reference frame and a block-shaped area taking the corresponding position as the center and s as the radius as a search area, wherein 0 is complemented for the part of the search area exceeding the size of the reference frame.
The further improvement of the invention is that the specific implementation method of the step 3) is as follows:
304) reference block BcurTransferring to a frequency domain;
305) taking a central block of the search area as a central block, and performing motion estimation;
306) transferring the center block and the peripheral eight blocks of the search area into a frequency domain, calculating a response value through a correlation formula of correlation filtering, transferring the response value into the time domain, and if the maximum response value G ismaxIn the central block, the next step is carried out, otherwise the maximum response value G is led tomaxThe block in which it is located is the center block and step 302) is entered.
The further improvement of the invention is that the specific implementation method of the step 4) is as follows:
403) according to the maximum response value G calculated in the step 3)maxRecording its coordinates (x, y) in the search area;
404) the difference between (x, y) and the center point (0, 0) of the search area is calculated and output as a motion vector.
Compared with the prior art, the invention has at least the following beneficial technical effects:
according to the invention, through carrying out logarithmic conversion on the input image, the contrast of the original image is improved, and the problem of low image contrast in a real scene is solved. Because the related filtering is used as a loss function for motion estimation, the problem that the accurate motion vector can not be obtained when the image is subjected to non-rigid transformation in the conventional motion estimation is well solved. Because the filter at each position is learned from the filter at the corresponding position, the problem of acquiring wrong motion vectors when the displacement is large is solved. Meanwhile, the invention improves the search strategy of motion estimation, reduces the calculated amount and improves the calculation efficiency while ensuring the output of high-precision motion vectors.
Drawings
FIG. 1 is a general framework of motion estimation between video frames based on correlation filtering according to the present invention;
FIG. 2 is a schematic diagram of a correlation filtering principle;
FIG. 3 is a diagram illustrating uni-directional motion estimation;
FIG. 4 is a schematic diagram of the search process of the present invention;
FIG. 5 is a schematic diagram of a related filtering process according to the present invention.
Detailed Description
The invention is described in further detail below with reference to the figures and examples.
Referring to fig. 1-5, the method for estimating fast motion between video frames based on correlation filtering according to the present invention includes the following steps:
1) and sequentially extracting a reference frame and a reference frame from the original video stream, wherein the current frame in the video stream is taken as the reference frame, and the next frame in the video stream is taken as the reference frame. The method comprises the steps of preprocessing a reference frame and a reference frame, namely carrying out logarithmic transformation on an image to improve the image contrast and solve the problem that an accurate motion vector cannot be obtained when the original image contrast is low.
2) Dividing the reference frame into non-overlapping macro blocks with consistent size as reference blocks BcurAnd taking the corresponding position in the reference frame and a block-shaped area taking the corresponding position as the center and s as the radius as a search area, wherein 0 is complemented for the part of the search area exceeding the size of the reference frame, and motion estimation is carried out.
3) In the process of motion estimation, correlation filtering is used as a loss function, and the principle of the correlation filtering is shown in fig. 2. The relevant filtering of the image can be described as: and finding a filter h, and obtaining the correlation with the input image f to obtain a response graph g, wherein the response graph g describes the target response and is larger when being closer.
4) The search strategy of the present invention is based on a one-way motion estimation algorithm, as shown in fig. 3, fig. 3 is a schematic diagram of one-way motion estimation, and one-way motion estimation is usually performed between two consecutive frames, with one frame being a reference frame and the other frame being a reference frame. And searching a corresponding block in the reference frame according to a certain search strategy by using each reference block in the reference frame to obtain the motion vector. Fig. 4 is a schematic diagram of the search strategy of the present invention, where each point in the diagram represents the upper left corner of the corresponding search block, the distance between each point is the size of the reference block, and the arrow represents the search direction. The circular point is the search block of the first search, the most central circular point is at the center of the search area, and the position of the reference block in the frame is the same. Take fig. 4 as an exampleFirst, according to the principles of the present invention, the response between the search block represented by nine circular points and the reference block is calculated, and these search blocks are recorded as Bref,i(i ═ 1,2, 3.., 9), find the point G with the largest responsemax,1And if the block position is in the center, stopping, and returning to the corresponding coordinate. G in FIG. 4max,1Above the center search block Bref,2Found, so proceed to the second step, calculate with Bref,2Response between the nine search blocks at the center and the reference block, since B is now presentref,i(i ═ 4,5, 6.., 9) has been calculated, and only three search blocks, represented as square points in the calculation graph, are used, and the point G with the largest response at this time is similarly recordedmax,2And if the block position is in the center, stopping, and returning to the corresponding coordinate. G in FIG. 4max,2In Bref,3Found, so proceed to the third step, with Bref,3As a center, the response between the surrounding nine search blocks and the reference block is calculated, since B is now presentref,i(i is 4,5,7,8), and only five search blocks indicated as pentagonal points in the figure are calculated, and if G is the casemax,3If the coordinate appears in the center, the corresponding coordinate is returned, otherwise, the G found in the first step is returnedmax,1And calculating a motion vector according to the corresponding coordinates. In this way, the amount of computation is still low while high precision motion vectors are satisfied.
5) FIG. 5 is a schematic diagram of the principle of correlation filtering, referring to a reference block BcurConversion to frequency domain to generate frequency domain graph BFcurThen, the conjugate image BF is obtained* curThen filter H at a certain position in the jth framejIt can be initialized as:
Figure BDA0002397473600000051
wherein G is an ideal Gaussian distribution, σ is a learning rate, and Hj-1The filter for the corresponding block in the previous frame.
Extracting each search block B in the search area in turn according to the search strategyref,iWill search for block Bref,iConversion to frequency domain to generate frequency domain graph BFref,iThen it is firstThe response map GF for a certain position in the j frame can be obtained by the following equation:
GF＝BFref,i·Hj
the conversion of the response map GF into the time domain yields a response map G in which the maximum G is foundmaxRecord GmaxCoordinates (x, y) of (a) and the block position at that time. The above process is repeated until the search step is completed. If G ismaxIf the corresponding value is on the central block, the search is stopped, and the coordinate at the moment is returned to calculate the motion vector. Otherwise with GmaxAnd taking the corresponding block as a center block, and continuing the searching step.

Claims (6)

1. A video inter-frame fast motion estimation method based on correlation filtering is characterized in that firstly, a reference frame and a reference frame extracted from an original video stream are preprocessed, the image contrast is improved by carrying out logarithmic transformation, secondly, the reference frame is divided into grids with equal size, and a reference block B is sequentially extractedcurAnd extracting a reference area in the reference frame according to the corresponding position of the reference block, sequentially extracting the reference block from the reference area, and calculating the motion vector through related filtering.
2. The method according to claim 1, wherein the method specifically comprises the following steps:
1) sequentially extracting a reference frame and a reference frame from an original video stream, and respectively preprocessing the reference frame and the reference frame;
2) sequentially extracting a reference block B from the preprocessed reference frame and the reference framecurAnd a search area;
3) the reference block B is processed according to the searching flowcurConverting the corresponding search block in the search area into a frequency domain, and calculating the correlation through correlation filtering;
4) and finding out the coordinates (x, y) of the point with the maximum correlation, and taking the displacement between the coordinates (x, y) and the center point of the search area as a motion vector.
3. The method according to claim 2, wherein the step 1) is implemented as follows:
101) sequentially extracting a reference frame and a reference frame from the video stream, taking the current frame of the video stream as the reference frame, and taking the next frame as the reference frame;
102) and respectively carrying out logarithmic change on the reference frame and the reference frame so as to improve the contrast of the image.
4. The method according to claim 3, wherein the step 2) is implemented as follows:
201) dividing the reference frame into several rectangles with equal size and without overlapping, making the rectangles as reference block Bcur；
202) And taking the corresponding position in the reference frame and a block-shaped area taking the corresponding position as the center and s as the radius as a search area, wherein 0 is complemented for the part of the search area exceeding the size of the reference frame.
5. The method according to claim 4, wherein the step 3) is implemented as follows:
301) reference block BcurTransferring to a frequency domain;
302) taking a central block of the search area as a central block, and performing motion estimation;
303) transferring the center block and the peripheral eight blocks of the search area into a frequency domain, calculating a response value through a correlation formula of correlation filtering, transferring the response value into the time domain, and if the maximum response value G ismaxIn the central block, the next step is carried out, otherwise the maximum response value G is led tomaxThe block in which it is located is the center block and step 302) is entered.
6. The method according to claim 5, wherein the step 4) is implemented as follows:
401) calculated according to step 3)To the maximum response value GmaxRecording its coordinates (x, y) in the search area;
402) the difference between (x, y) and the center point (0, 0) of the search area is calculated and output as a motion vector.
CN202010136402.XA 2020-03-02 2020-03-02 Video inter-frame rapid motion estimation method based on correlation filtering Active CN111292357B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010136402.XA CN111292357B (en) 2020-03-02 2020-03-02 Video inter-frame rapid motion estimation method based on correlation filtering

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010136402.XA CN111292357B (en) 2020-03-02 2020-03-02 Video inter-frame rapid motion estimation method based on correlation filtering

Publications (2)

Publication Number Publication Date
CN111292357A true CN111292357A (en) 2020-06-16
CN111292357B CN111292357B (en) 2023-04-14

Family

ID=71020192

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010136402.XA Active CN111292357B (en) 2020-03-02 2020-03-02 Video inter-frame rapid motion estimation method based on correlation filtering

Country Status (1)

Country Link
CN (1) CN111292357B (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113542588A (en) * 2021-05-28 2021-10-22 上海第二工业大学 Anti-interference electronic image stabilization method based on visual saliency

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5731850A (en) * 1995-06-07 1998-03-24 Maturi; Gregory V. Hybrid hierarchial/full-search MPEG encoder motion estimation
CN1633184A (en) * 2005-01-14 2005-06-29 北京航空航天大学 Multi-reference frame rapid movement estimation method based on effective coverage
US20060203914A1 (en) * 2005-03-09 2006-09-14 Pixart Imaging Inc. Motion estimation method utilizing a distance-weighted search sequence
CN101888546A (en) * 2010-06-10 2010-11-17 北京中星微电子有限公司 Motion estimation method and device

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5731850A (en) * 1995-06-07 1998-03-24 Maturi; Gregory V. Hybrid hierarchial/full-search MPEG encoder motion estimation
CN1633184A (en) * 2005-01-14 2005-06-29 北京航空航天大学 Multi-reference frame rapid movement estimation method based on effective coverage
US20060203914A1 (en) * 2005-03-09 2006-09-14 Pixart Imaging Inc. Motion estimation method utilizing a distance-weighted search sequence
CN101888546A (en) * 2010-06-10 2010-11-17 北京中星微电子有限公司 Motion estimation method and device

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
余志雄;孙洪;贺军;: "基于低频子图的运动估计算法" *
李绍滋;苏松志;成运;孙岩;郭锋;: "H.264中的运动估计和帧内预测算法研究" *

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113542588A (en) * 2021-05-28 2021-10-22 上海第二工业大学 Anti-interference electronic image stabilization method based on visual saliency

Also Published As

Publication number Publication date
CN111292357B (en) 2023-04-14

Similar Documents

Publication Publication Date Title
CN103325112B (en) Moving target method for quick in dynamic scene
Jiang et al. Video object detection with locally-weighted deformable neighbors
CN106210449B (en) Multi-information fusion frame rate up-conversion motion estimation method and system
CN102098440B (en) Electronic image stabilizing method and electronic image stabilizing system aiming at moving object detection under camera shake
Ding et al. Spatio-temporal recurrent networks for event-based optical flow estimation
CN110796010A (en) Video image stabilization method combining optical flow method and Kalman filtering
CN110753181A (en) Video image stabilization method based on feature tracking and grid path motion
CN105872345A (en) Full-frame electronic image stabilization method based on feature matching
CN102427505B (en) Video image stabilization method and system on the basis of Harris Corner
CN110163887B (en) Video target tracking method based on combination of motion interpolation estimation and foreground segmentation
CN108460792B (en) Efficient focusing stereo matching method based on image segmentation
CN108280844B (en) Video target positioning method based on area candidate frame tracking
CN110796662A (en) Real-time semantic video segmentation method
CN111160295A (en) Video pedestrian re-identification method based on region guidance and space-time attention
CN115619826A (en) Dynamic SLAM method based on reprojection error and depth estimation
CN101272450B (en) Global motion estimation exterior point removing and kinematic parameter thinning method in Sprite code
CN111292357B (en) Video inter-frame rapid motion estimation method based on correlation filtering
CN114973399A (en) Human body continuous attitude estimation method based on key point motion estimation
Zhang et al. Dyna-depthformer: Multi-frame transformer for self-supervised depth estimation in dynamic scenes
Zhu et al. A Pose Estimation Method in Dynamic Scene with Yolov5, Mask R-CNN and ORB-SLAM2
CN202931463U (en) Characteristic block based video image stabilization device
Puglisi et al. Fast block based local motion estimation for video stabilization
CN112561001A (en) Video target detection method based on space-time feature deformable convolution fusion
Shi et al. Motion-compensated temporal frame interpolation algorithm based on global entirety unidirectional motion estimation and local fast bidirectional motion estimation
CN113205544B (en) Space attention reinforcement learning tracking method based on cross-over ratio estimation

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant