CN104200494B - Real-time visual target tracking method based on light streams - Google Patents

Abstract

The invention discloses a real-time visual target tracking method based on light streams. The method includes: screening feature points rich in textural features, and tracking the feature points between every two frames of images to obtain feature point matching relations; filtering the feature points with large matching errors through various filter algorithms such as normalized correlation coefficients, forward-reverse tracking errors and random consistency detection to retain most reliable feature points so as to obtain the target motion speed in images; using the target motion speed as observation quantity, and estimating through Kalman filtering to generate the target position. Compared with a tracking algorithm, the method has the advantages that the method is unaffected by light, the target can be stably tracked under the conditions that the target moves, a camera moves, the target is blocked for a short term and the like. The method is high in calculation efficiency, and the tracking algorithm is verified for the first time on an ARM platform.

Description

Real-time visual target tracking method based on optical flow

Technical Field

The invention relates to a real-time visual target tracking method based on optical flow, and belongs to the technical field of target tracking.

Background

The visual target tracking is an important research direction in the field of machine vision, detects, extracts, identifies and tracks an interested target in an image sequence, so as to provide motion state parameters of the target, such as position, speed, acceleration and motion trail, further process and analyze the motion state parameters, realize behavior understanding of the moving target, provide original data for subsequent applications (such as visual navigation, pose estimation, motion analysis and the like), and have wide application in the fields of intelligent monitoring, human-computer interaction, robot navigation and the like. In addition to the civil field, the visual target tracking technology is also greatly concerned in the military field, such as field reconnaissance, cruise missile guidance, ground attack helicopters, main battle tank fire control systems and other scenes. In the above application, object tracking is the basis for the robot to sense the external environment and react, and is the key to understanding the image.

Currently, the following methods are commonly used for visual tracking:

(1) the Meanshift algorithm is a method of matching images of a fixed-size area using color information. The Meanshift algorithm is a data-driven non-parameter estimation algorithm, which is also called a kernel density estimation algorithm, and mainly finds the posterior probability local optimum through a mean shift vector, namely, the Meanshift algorithm determines a matching region through a region with the most similar color, and the Meanshift algorithm does not support the updating of a target.

(2) The template matching algorithm is a method for comparing each region of an image with a fixed template. And selecting a target template during initialization, and finally selecting an image area with the minimum distance from the template as the position of the target in the current image by continuously comparing the image area with the template in the tracking process. The algorithm is simple in structure, but the calculation amount is large, the condition of target change cannot be processed, and when the target is changed greatly or rotates by more than 45 degrees, the algorithm fails.

(3) The machine learning method is used for training a classifier by respectively extracting the features of a target and a background, and the image is evaluated by the classifier. In the tracking process, the position of the target is determined through the posterior probability given by the classifier. Meanwhile, the classifier can be retrained through the calculation result, so that continuous tracking is realized under the condition that the target is changed. The algorithm has large calculation amount, unstable tracking effect and easy divergence.

In summary, the problems of the prior art are as follows: the tracking of the target is greatly influenced by the environment (such as illumination change, target motion and camera motion), and the shape updating of the target is not supported. Meanwhile, the calculation amount is large, and the real-time requirement is difficult to meet.

Disclosure of Invention

The invention aims to solve the problems and provides a real-time visual target tracking method based on optical flow. And screening out feature points rich in texture features during initialization, and tracking the feature points between every two frames of images so as to obtain a feature point matching relation. And filtering the characteristic points with large matching errors through a plurality of filtering algorithms including normalized correlation coefficients, positive and negative tracking errors, random consistency detection and the like, and reserving the most reliable characteristic points so as to obtain the movement speed of the target in the image. And taking the motion speed of the target as an observed quantity, and generating the target position by Kalman filtering estimation.

A real-time visual target tracking method based on optical flow comprises the following steps:

acquiring an initial position of a first frame image of a tracking target;

step two, acquiring a characteristic point sequence;

step three: obtaining an image I of a new moment of a tracked target_t+1；

Step four: tracking the characteristic points;

step five: screening the point pairs;

step six: eliminating the influence of background characteristic points;

step seven: system state estimation

The invention has the advantages that:

(1) target tracking is carried out in a characteristic point tracking mode, the calculated amount is small, and the running speed is high;

(2) by using Kalman filtering, tracking errors can be controlled, and meanwhile, the method has the capability of resisting short-time shielding;

(3) a background elimination method based on random consistency detection is provided, and the influence of background special points on a tracking result can be eliminated.

Drawings

FIG. 1 is a flow chart of a method of the present invention;

FIG. 2 is a schematic illustration of background elimination of the present invention;

FIG. 3 is a schematic illustration of face tracking;

FIG. 4 is a schematic view of a track-and-shoot slow-running large vehicle;

FIG. 5 is a schematic view of tracking an aerial highway vehicle.

Detailed Description

The present invention will be described in further detail with reference to the accompanying drawings and examples.

The invention relates to a real-time visual target tracking method based on optical flow, the flow of the method is shown in figure 1, and the method comprises the following steps:

acquiring an initial position of a first frame image of a tracking target;

the initial position of the tracking target is known, a rectangular frame is given to represent the position of the target, and the position of the target in each frame of image is represented by the rectangular frame in the target tracking process.

Step two, acquiring a characteristic point sequence

The optical flow field of a general solution is a dense motion field, and is generally used for navigation applications such as obstacle avoidance. However, in the target tracking problem, a dense optical flow field is not required, and the motion velocity of the target can be obtained only by optical flows of partial pixel points on the target. Furthermore, solving for dense optical flow consumes a significant amount of time, which is too burdensome for real-time algorithms. Furthermore, not all points are suitable for tracking, and for those points lacking texture information, the tracking error will be larger, and will cause the tracking error, therefore, the present invention firstly uses the image I_tIn a rectangular frame of the position of the middle target, screening characteristic points rich in texture information, wherein the method comprises the following steps:

let p be ═ p_x，p_y]Is any point in a rectangular frame, p_x、p_yThe coordinates of a point p in a pixel coordinate system are represented, a search area is defined as a range with a radius w and a center of p, the size of the search area is (2w +1) × (2w +1), and a matrix M is defined as:

wherein: i is_xAnd I_yIs the differential of all points in the search area in the x and y directions under the pixel coordinate system.

For each point in the rectangular frame, two eigenvalues λ of the matrix M are calculated₁、λ₂Let the minimum eigenvalue be λ_minSetting a threshold lambda of the characteristic value_{threshold}，1≤λ_{threshold}Less than or equal to 10, discarding lambda_min＜λ_{threshold}The remaining dots are at λ_minArranging the points in sequence from big to small to be set as a pre-selected point set, wherein the positions of some points in the pre-selected point set generally converge together, and selecting lambda in the pre-selected point set to balance the distribution of the points_minMaximum point, set minimum distance, discard and λ_minThe maximum point distance is less than the minimum distance, will be λ_minAdding the largest point to the characteristic point sequence, forming a new pre-selected point set by the rest points, and selecting lambda from the new pre-selected point set_minMaximum point, discard and λ_minThe maximum point distance is less than the minimum distance, will be λ_minAnd adding the largest point into the characteristic point sequence, forming a new preselected point set by the rest points again, circulating in sequence until the preselected point set is empty, and finally obtaining the characteristic point sequence.

Wherein if the minimum distance is too large, fewer points remain, otherwise the number of points is too large and too concentrated. The number of feature points that are ultimately retained can be controlled by adjusting the minimum spacing. After going through the above steps, the remaining points are points that are rich in texture information and suitable for tracking, while being more evenly distributed within the rectangular box.

Step three: obtaining an image I of a new moment of a tracked target_t+1；

Step four: tracking the characteristic points;

according to image I_tThe image I is obtained by the Lucas-Kanade method through the characteristic point sequence in the method_tIn image I_t+1The corresponding points form N point pairs.

In the invention, the Lucas-Kanade method is used for matching the characteristic points, so that the movement of the characteristic points between two images is obtained, namely the characteristic points are tracked.

Step five: screening the point pairs;

after the characteristic point sequence tracking is completed, N point pairs are obtained, wherein some are reliable, and some have larger tracking errors, and the point pairs are screened by respectively adopting positive and negative tracking errors and normalized correlation coefficients.

(1) Screening the N point pairs by adopting positive and negative tracking errors;

positive and negative tracking errors the error of the characteristic point tracking is evaluated by using a time positive and negative consistent constraint method, and the essential idea is to assume that the correct characteristic point tracking should be independent of the direction of time flow.

Image I set at time t_tPoint p in (1)_tImage I matched to time t +1_t+1Point in (1) is denoted as p_t+1Let is from p_t+1Point retracing back to image I_tWhen the corresponding point is p'_tIdeally, p'_tAnd p_tLet the distance between these two points be the error, which should be the same point:

e＝dis(p′_t，p_t)，

obtaining errors of N point pairs, and making the errors largerTo a small rank, the reserve error is smallA point pair.

(2) Using pairs of normalized correlation coefficientsScreening individual point pairs;

for each feature point, a small image block representation centered on it may be used. Due to the assumption of smooth motion, the appearance of an object between two adjacent images does not change drastically. Therefore, if the point matching is correct, the image blocks corresponding to each pair of feature points should be similar. Therefore, the degree of similarity between a pair of image blocks can be measured by normalizing the correlation coefficients.

Let feature points be represented by image blocks of w × w, which may be represented by a vector q ∈ R^w×wIndicating that at time t, image I is acquired_tPoint p in (1)_tImage block A of_tTo obtain a vector q_tAt time t +1, image I is acquired_t+1Point p in (1)_t+1Image block A of_t+1To obtain a vector q_t+1Obtaining normalized correlation coefficients of two vectors:

wherein,representing a vector q_tIs determined by the average value of (a) of (b),representing a vector q_t+1Is determined by the average value of (a) of (b),<·，·>denotes the inner product of two vectors, ncc (q)_t，q_t+1) Indicating the normalized correlation coefficient, a normalized correlation coefficient threshold value ncc is set_{threshold}，0.7≤ncc_{threshold}Less than or equal to 0.9, and making the normalized correlation coefficient less than ncc_{threshold}The point pairs of (2) are discarded.

And screening the point pairs obtained in the step four by the two methods to obtain the screened high-reliability point pairs.

Step six: eliminating the influence of background characteristic points;

in addition to the target to be tracked, a background is often included in the rectangular frame, and feature points from the background may adversely affect the tracking result.

And eliminating the interference of the background characteristic points by adopting a random consensus detection (RANSAC) based method. Let P be (P)₁，p₂...p_m，p_m+1...p_n) Is a high reliability set of point pairs after screening, where the first m point pairs come from the background, m ≦ n/2 (in a correct tracking procedure, the target should be the main part of the rectangular box, so we agree that m ≦ n/2). Let D ═ D₁，d₂...d_m，d_m+1...d_n) Indicating that each point pair in P is displaced between two frame images, then d_iIndicating the relative displacement of the ith point pair. In the image coordinate system, the velocity of the background is assumed to be V_bThe speed of the object being V_oThe following can be obtained:

d_i～N(V_b，σ_b)，i∈[1，m]

d_j～N(V_o，σ_o)，j∈[m+1，n]

wherein: d_i～N(V_b，σ_b) Denotes d_iObey mean value of V_bVariance is σ_bNormal distribution of (d)_j～N(V_o，σ_o) Denotes d_jObey mean value of V_oVariance is σ_oIs normally distributed.

The method for eliminating the background feature points comprises the following steps:

(1) randomly extracting K elements from the D to form an array V;

(2) taking the median value of V and recording as EV_iFinding out all elements in V and EV_iThe error of (2) is recorded as ∈_i；

(3) Executing the steps (1) and (2), repeating for T times, and solving T errors;

(4) EV with minimum error_iThe value is recorded as EV, and the EV is the displacement of the target;

(5) solving errors of all elements in D and EV, and removing point pairs with the errors larger than a threshold value, wherein the error is more than or equal to 1 and less than or equal to 3;

through the steps, the point pairs with the influence of the background characteristic points eliminated are obtained.

As shown in fig. 2 below, when tracking a car on a highway, two points in the background are eliminated (gray) and the remaining feature points (white) are all located on the car.

The method for eliminating background feature points in the sixth step has the main advantages that the method is easy to apply and is suitable for a series of tracking algorithms based on feature points, and the calculation method of the estimation and the error of the speed can be changed according to the application.

Step seven: system state estimation

Although the remaining feature points have strong reliability after being filtered for many times, the error of the algorithm still accumulates gradually during the operation process. Therefore, kalman filtering is adopted to estimate the position of the target by receiving the displacement of the target in the step six as the observed quantity, and simultaneously control the noise. In addition, feature point based tracking algorithms can fail when a tracked object disappears from view or features are lost. With kalman filtering, when the tracker fails without an output of velocity, the algorithm can maintain the state of the object by no measurement update of the tracking model.

Let the state vector X at time t_t＝[P_t，V_t]^TIn which P is_tIndicating the position of the rectangular frame, V_tIndicating the displacement of the rectangular frame in the current frame image and the previous frame image. Because the motion of the target between any two frames of images is not too large, the motion of the target between each frame can be taken as a constant speed model. Thus, the state transition matrix of the target is as follows:

wherein: x_t+1Representing the state vector at time t +1, X_tRepresenting the state vector at time t, x_t+1、y_t+1Representing the coordinates, x, of the object in the image coordinate system at time t +1_t、y_tRepresenting the coordinates of the object in the image coordinate system at time t, u_t+1、v_t+1Representing the speed of the target in two directions in the image coordinate system from time t to time t +1, u_t、v_tRepresents the speed of the target from t-1 time to t time in two directions of the image coordinate system, and u is set₀＝v₀When the number of frames is 0, Δ t represents a time interval between the acquisition of two frames of images.

The observation vector for kalman filtering is represented as:

wherein,

V～N(0，R)

wherein V to N (0, R) represent a normal distribution in which V follows a mean value of 0 and a variance of R.

And estimating the observation vector through Kalman filtering, so as to obtain the position of the target t +1 in the image, wherein the position is a rectangular frame, and returning to the step two until the target tracking is stopped.

The method of the invention is applied to track human faces and automobiles, wherein the result of tracking the human faces is shown in figure 3, and people enter a bright environment from a dark environment, change the positions of the faces, take off glasses and other operations.

Claims (4)

1. A real-time visual target tracking method based on optical flow comprises the following steps:

acquiring an initial position of a first frame image of a tracking target;

the initial position of a tracking target is known, and a rectangular frame is given to represent the position of the target;

step two, acquiring a characteristic point sequence;

in picture I_tIn a rectangular frame of the position of the middle target, screening characteristic points rich in texture information, wherein the method comprises the following steps:

let p be ═ p_x,p_y]Is any point in a rectangular frame, p_x、p_yThe coordinates of a point p in a pixel coordinate system are represented, a search area is defined as a range with a radius w and a center of p, the size of the search area is (2w +1) × (2w +1), and a matrix M is defined as:

$M=\underset{x=p_x-w}{\overset{p_x+w}{\&Sigma;}}\underset{y=p_y-w}{\overset{x=p_y+w}{\&Sigma;}}\left[\begin{array}{cc}{I}_x{I}_x& I_x{I}_y\\ I_x{I}_y& I_y{I}_y\end{array}\right]$

wherein: i is_xAnd I_yThe differential of all points in the search area in the x and y directions under the pixel coordinate system is obtained;

for each point in the rectangular frame, two eigenvalues λ of the matrix M are calculated₁、λ₂Let the minimum eigenvalue be λ_minSetting a threshold lambda of the characteristic value_{{thres hold}}Discard λ_min<λ_{{thres hold}}The remaining dots are at λ_minArranging in descending order, setting as a pre-selected point set, and selecting lambda in the pre-selected point set_minMaximum point, set minimum distance, discard and λ_minThe maximum point distance is less than the minimum distance, will be λ_minAdding the largest point to the characteristic point sequence, forming a new pre-selected point set by the rest points, and selecting lambda from the new pre-selected point set_minMaximum point, discard and λ_minThe maximum point distance is less than the minimum distance, will be λ_minAdding the largest point into the characteristic point sequence, forming a new preselected point set by the rest points again, circulating in sequence until the preselected point set is empty, and finally obtaining the characteristic point sequence;

step three: obtaining an image I of a new moment of a tracked target_t+1；

Step four: tracking the characteristic points;

according to image I_tThe image I is obtained by the Lucas-Kanade method through the characteristic point sequence in the method_tIn image I_t+1The corresponding points form N point pairs;

step five: screening the point pairs;

(1) screening the N point pairs by adopting positive and negative tracking errors;

image set at time tI_tPoint p in (1)_tImage I matched to time t +1_t+1Point in (1) is denoted as p_t+1Let is from p_t+1Point retracing back to image I_tWhen the corresponding point is p'_tLet the distance between these two points be the error:

e＝dis(p′_t,p_t)，

obtaining errors of N point pairs, sorting the errors from large to small, and keeping the errors smallPoint pairs;

(2) using pairs of normalized correlation coefficientsScreening individual point pairs;

at time t, image I is acquired_tPoint p in (1)_tImage block A of_tTo obtain a vector q_tAt time t +1, image I is acquired_t+1Point p in (1)_t+1Image block A of_t+1To obtain a vector q_t+1Obtaining normalized correlation coefficients of two vectors:

$v_1=q_t-\stackrel{\&OverBar;}{q_t},$

$v_2=q_{t+1}-\stackrel{\&OverBar;}{q_{t+1}},$

$ncc(q_t,q_{t+1})=<\frac{v_1}{\left|v_1\right|},\frac{v_2}{\left|v_2\right|}>$

wherein,representing a vector q_tIs determined by the average value of (a) of (b),representing a vector q_t+1Is determined by the average value of (a) of (b),<·,·>denotes the inner product of two vectors, ncc (q)_t,q_t+1) Indicating the normalized correlation coefficient, a normalized correlation coefficient threshold value ncc is set_{{thres hold}}Normalized correlation coefficient is less than ncc_{{thres hold}}Discarding the point pairs;

screening the point pairs obtained in the step four by the two methods to obtain screened high-reliability point pairs;

step six: eliminating the influence of background characteristic points;

let P be (P)₁,p₂...p_m,p_m+1...p_n) Is a high reliability set of point pairs after screening, where the first m point pairs come from the backN/2 is less than or equal to m; let D ═ D₁,d₂...d_m,d_m+1...d_n) Indicating that each point pair in P is displaced between two frame images, then d_iRepresenting the relative displacement of the ith point pair; in the image coordinate system, the velocity of the background is assumed to be V_bThe speed of the object being V_oObtaining:

d_i～N(V_b,σ_b),i∈[1,m]

d_j～N(V_o,σ_o),j∈[m+1,n]

wherein: d_i～N(V_b,σ_b) Denotes d_iObey mean value of V_bVariance is σ_bNormal distribution of (d)_j～N(V_o,σ_o) Denotes d_jObey mean value of V_oVariance is σ_oNormal distribution of (2);

the method for eliminating the background feature points comprises the following steps:

(1) randomly extracting K elements from the D to form an array V;

(2) taking the median value of V and recording as EV_iFinding out all elements in V and EV_iThe error of (2) is recorded as ∈_i；

(3) Executing the steps (1) and (2), repeating for T times, and solving T errors;

(4) EV with minimum error_iThe value is recorded as EV, and the EV is the displacement of the target;

(5) solving errors between all elements in D and EV, and removing point pairs with the errors larger than a threshold value;

obtaining a point pair for eliminating the influence of the background characteristic points through the steps;

step seven: estimating the state of the system;

let the state vector X at time t_t＝[P_t,V_t]^TIn which P is_tIndicating the position of the rectangular frame, V_tRepresenting the displacement of a rectangular frame in a current frame image and a previous frame image, taking the motion of a target between each frame as a constant speed model, wherein a state transition matrix of the target is as follows:

$X_{t+1}=\left[\begin{array}{c}{x}_{t+1}\\ y_{t+1}\\ u_{t+1}\\ v_{t+1}\end{array}\right]=F_x\&CenterDot;X_t=\left[\begin{array}{cccc}1& 0& \mathrm{\&Delta;}t& 0\\ 0& 1& 0& \mathrm{\&Delta;}t\\ 0& 0& 1& 0\\ 0& 0& 0& 1\end{array}\right]\left[\begin{array}{c}{x}_t\\ y_t\\ u_t\\ v_t\end{array}\right]$

wherein: x_t+1Representing the state vector at time t +1, X_tRepresenting the state vector at time t, x_t+1、y_t+1Representing the coordinates, x, of the object in the image coordinate system at time t +1_t、y_tRepresenting the coordinates of the object in the image coordinate system at time t, u_t+1、v_t+1Representing the speed of the target in two directions in the image coordinate system from time t to time t +1, u_t、v_tRepresents the speed of the target from t-1 time to t time in two directions of the image coordinate system, and u is set₀＝v₀When the image is obtained, Δ t represents a time interval between two frames of images;

the observation vector for kalman filtering is represented as:

$Z=\left[\begin{array}{c}{u}_t\\ v_t\end{array}\right]=H\&CenterDot;X_t+V,$

wherein,

$H=\left[\begin{array}{cccc}0& 0& 1& 0\\ 0& 0& 0& 1\end{array}\right],$

V～N(0,R)

wherein V-N (0, R) represents that V obeys normal distribution with mean value of 0 and variance of R;

and estimating the observation vector through Kalman filtering to obtain the position of the target t +1 in the image, wherein the position is a rectangular frame, and returning to the step two to circulate until the target tracking is stopped.

2. The method for optical flow-based real-time visual target tracking as claimed in claim 1, wherein λ is 1 ≦ λ in said second step_{{thres hold}}≤10。

3. The method for optical flow-based real-time visual target tracking according to claim 1, wherein 0.7 ≦ ncc in said step five_{{thres hold}}≤0.9。

4. The method for optical flow-based real-time visual target tracking according to claim 1, wherein 1. ltoreq. 3 in said sixth step.