CN107194308B - Method and device for tracking ball players in video - Google Patents
Method and device for tracking ball players in video Download PDFInfo
- Publication number
- CN107194308B CN107194308B CN201710210009.9A CN201710210009A CN107194308B CN 107194308 B CN107194308 B CN 107194308B CN 201710210009 A CN201710210009 A CN 201710210009A CN 107194308 B CN107194308 B CN 107194308B
- Authority
- CN
- China
- Prior art keywords
- player
- frame
- value
- tracking
- coordinate
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/22—Image preprocessing by selection of a specific region containing or referencing a pattern; Locating or processing of specific regions to guide the detection or recognition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/40—Scenes; Scene-specific elements in video content
- G06V20/41—Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items
- G06V20/42—Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items of sport video content
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Theoretical Computer Science (AREA)
- Computational Linguistics (AREA)
- Software Systems (AREA)
- Human Computer Interaction (AREA)
- Image Analysis (AREA)
Abstract
The invention relates to a method and a device for tracking a player in a video, wherein the method comprises the steps of obtaining a court area in a current frame of the video; obtaining player pixels in the court area; generating a player mark frame according to the player pixels; when the area of the player marking frame in the current frame is increased, acquiring a segmentation point of the player marking frame with the increased area; dividing the player marking frame according to the dividing points to form a new player marking frame; and tracking and displaying the mark frame of the player in the segmented current frame through Kalman filtering. According to the method and the device for tracking the football players in the video, whether the football players are shielded is judged by detecting whether the area of the marker frame of the football players is increased, the processing is simple, and when the shielding exists, the marker frame of the football players can be divided through the dividing points, so that the marker frame of the football players can be accurately tracked through Kalman filtering, and the condition that the information of the football players is lost due to the shielding of the football players cannot occur.
Description
Technical Field
The invention relates to the field of image processing, in particular to a method and a device for tracking a player in a video.
Background
In the field of football video, Bebie et al realized a semi-automatic video clip of a football match, focused on the reproduction of player movements and postures, and Bobick et al proposed the concept of "closed-world" and applied it to the tracking of American football. However, the above method has at least the following problems: and once the blocking condition appears to the player, complex detection mechanisms are needed to be carried out for retracing the blocked player and the like.
Disclosure of Invention
Based on this, it is necessary to provide a player tracking method and device in video for the problem that the player is occluded.
A method of player tracking in a video, the method comprising:
acquiring a court area in a current frame of a video;
obtaining player pixels in the court area;
generating a player mark frame according to the player pixels;
when the area of the player marking frame in the current frame is increased, acquiring a segmentation point of the player marking frame with the increased area;
dividing the player marking frame according to the dividing points to form a new player marking frame;
and tracking and displaying the mark frame of the player in the segmented current frame through Kalman filtering.
In one embodiment, the step of obtaining the segmentation point of the player mark box with the increased area comprises:
acquiring the number of player pixels on two preset and opposite edges of the player marking frame with the increased area;
when the number of the pixels of the player on the two preset opposite sides is not equal, the length and the width of each player marking frame before the player marking frame is not shielded are obtained according to the coordinates of the player marking frames with the increased areas;
calculating the dividing points of the player marking frames according to the length and the width of each player marking frame before the player marking frames are not shielded;
and when the number of the player pixels on the two preset opposite edges is equal, acquiring the central point of the player mark frame as the segmentation point.
In one embodiment, the step of performing tracking display on the player mark frame in the segmented current frame through kalman filtering includes:
tracking the player mark frame in the segmented current frame through Kalman filtering;
acquiring the coordinate of the tracked player mark frame;
acquiring a first player projection coordinate of the coordinate projection of the player mark frame to a target plane;
and in the target plane, tracking and displaying the player identification of the position where the corresponding player coordinate in the previous frame is projected to the second player projection coordinate in the target plane according to the first player projection coordinate.
In one embodiment, the step of tracking the player mark frame in the segmented current frame through kalman filtering includes:
calculating an error covariance matrix between the predicted value and the true value of the player mark box in the current frame by the following formula:
wherein the content of the first and second substances,
is a predicted value at the time k, wherein,
is an estimate of the time k-1, uk-1For the current frame at the time k-1, A is a conversion matrix for converting the predicted value at the previous time into the predicted value at the current time, B is a conversion matrix for converting the input value at the previous time into the predicted value at the current time, Pk' is the error covariance matrix between the predicted value and the true value at the k time, xkThe real value at the moment k is obtained, and Q is a covariance matrix of system noise;
the kalman gain is calculated by the following equation:
Kk＝P′kHT(HP′kHT+R)-1and is and
wherein, KkFor the Kalman gain at time k, H is the conversion matrix for converting the predicted value to the measured value at the same time, R is the covariance matrix for measuring the noise variance, zkIs the measured value at the time k;
calculating an error covariance matrix between the estimated value and the real value based on the Kalman gain and the error covariance matrix between the predicted value and the real value by the following formula:
and obtaining the real value according to an error covariance matrix between the estimated value and the real value, and tracking the real value.
In one embodiment, the method further comprises:
obtaining a court line in the court area through Hough transform;
and determining the coordinate projection relation between the current frame and a target plane according to the spherical field lines.
A player tracking device in a video, the device comprising:
the court area acquisition module is used for acquiring a court area in a current frame of the video;
the player mark frame generation module is used for acquiring player pixels in the ball field area and generating a player mark frame according to the player pixels;
a division point acquisition module, configured to acquire, when the area of the player mark frame in the current frame increases, a division point of the player mark frame whose area increases;
the segmenting module is used for segmenting the player marking frame according to the segmenting point to form a new player marking frame;
and the tracking module is used for tracking and displaying the segmented player mark frames in the current frame through Kalman filtering.
In one embodiment, the segmentation point obtaining module includes:
the number acquisition unit is used for acquiring the number of player pixels on two preset and opposite edges of the player marking frame with the increased area;
the dividing point determining unit is used for acquiring the length and the width of each player marking frame before the player marking frames are not shielded according to the coordinates of the player marking frames with increased areas when the number of the player pixels on the two preset and opposite edges is not equal; calculating the dividing points of the player marking frames according to the length and the width of each player marking frame before the player marking frames are not shielded; and when the number of the player pixels on the two preset opposite edges is equal, acquiring the central point of the player mark frame as the segmentation point.
In one embodiment, the tracking module comprises:
the tracking unit is used for tracking the player mark frame in the segmented current frame through Kalman filtering;
the first coordinate calculation unit is used for acquiring the coordinates of the tracked player mark frame;
the second coordinate calculation unit is used for acquiring a first player projection coordinate of the coordinate projection of the player mark frame to the target plane;
and the display unit is used for tracking and displaying the player identification of the coordinate projection of the corresponding player in the previous frame to the projection coordinate of the second player in the target plane according to the projection coordinate of the first player in the target plane.
In one embodiment, the tracking unit comprises:
a first matrix calculation unit for calculating an error covariance matrix between the predicted value and the true value of the player mark frame in the current frame by the following formula:
wherein the content of the first and second substances,
is a predicted value at the time k, wherein,
is an estimate of the time k-1, uk-1For the current frame at the time k-1, A is a conversion matrix for converting the predicted value at the previous time into the predicted value at the current time, B is a conversion matrix for converting the input value at the previous time into the predicted value at the current time, Pk' is the error covariance matrix between the predicted value and the true value at the k time, xkThe real value at the moment k is obtained, and Q is a covariance matrix of system noise;
a kalman gain calculating unit for calculating a kalman gain by the following formula:
Kk＝P′kHT(HP′kHT+R)-1and is and
wherein, KkFor the Kalman gain at time k, H is the conversion matrix for converting the predicted value to the measured value at the same time, R is the covariance matrix for measuring the noise variance, zkIs the measured value at the time k;
a second matrix calculation unit for calculating an error covariance matrix between the estimated value and the true value based on the kalman gain and the error covariance matrix between the predicted value and the true value by the following formula:
and the real value calculating unit is used for obtaining the real value according to the error covariance matrix between the estimated value and the real value and tracking the real value.
In one embodiment, the apparatus further comprises:
the coordinate projection relation calculation unit is used for obtaining a court line in the court area through Hough transformation; and determining the coordinate projection relation between the current frame and a target plane according to the spherical field lines.
According to the method and the device for tracking the football players in the video, whether the football players are shielded is judged by detecting whether the area of the marker frame of the football players is increased, the processing is simple, and when the shielding exists, the marker frame of the football players can be divided through the dividing points, so that the marker frame of the football players can be accurately tracked through Kalman filtering, and the condition that the information of the football players is lost due to the shielding of the football players cannot occur.
Drawings
FIG. 1 is a flow diagram of a method for player tracking in a video according to one embodiment;
FIG. 2 is a flow chart of step S108 in the embodiment shown in FIG. 1;
FIG. 3 is a schematic view of a player marking box of the occluded player of the embodiment shown in FIG. 2;
FIG. 4 is a flowchart of step S112 in the embodiment shown in FIG. 1;
FIG. 5 is a schematic view of the projected coordinates of a first player in the target plane in the embodiment of FIG. 1;
FIG. 6 is a flowchart of step S112 in the embodiment shown in FIG. 1;
FIG. 7 is a flowchart illustrating a coordinate projection relationship obtaining step in an embodiment;
FIG. 8 is a histogram of the gray scale of the R channel of the current frame in the embodiment shown in FIG. 1;
FIG. 9 is a histogram of the gray levels of the G channel of the current frame in the embodiment shown in FIG. 1;
FIG. 10 is a histogram of the gray scale of the B channel of the current frame in the embodiment shown in FIG. 1;
fig. 11 is a schematic view of a court area in the embodiment shown in fig. 1;
FIG. 12 is a schematic illustration of a white line region in a field area in one embodiment;
FIG. 13 is a diagram of the embodiment of FIG. 12 after Hough transform;
FIG. 14 is a diagram of a player tracking device in a video according to one embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
Before describing in detail embodiments that are in accordance with the present invention, it should be observed that the embodiments reside primarily in combinations of steps and system components related to data ordering methods and apparatus. Accordingly, the system components and method steps have been represented where appropriate by conventional symbols in the drawings, showing only those details that are pertinent to understanding the embodiments of the present invention so as not to obscure the disclosure with details that will be readily apparent to those of ordinary skill in the art having the benefit of the description herein.
In this document, relational terms such as left and right, top and bottom, front and back, first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. The terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus.
Referring to fig. 1, fig. 1 is a flowchart of a player tracking method in a video according to an embodiment, the method includes:
s102: a field region in a current frame of a video is acquired.
Specifically, since the soccer field is generally composed of a grass, the color of the grass is green, the periphery of the grass is a track, and the color of the track is red, the field area can be extracted from the current frame by the color.
S104: player pixels in the field area are obtained.
Specifically, the player pixels refer to pixels representing players, and because colors of uniforms of players of different teams on the court are different and are different from colors of the grasslands, the player pixels in the court area can be extracted from the current frame according to different colors.
S106: and generating a player mark frame according to the player pixels.
Specifically, the player mark frame may be determined according to the size of the connected domain of the player pixels, and for example, a minimum rectangular frame including the player pixels in a certain area may be generated as the player mark frame according to a certain player pixel in a breadth-first search manner.
S108: when the area of the player mark frame in the current frame is increased, the division point of the player mark frame with the increased area is obtained.
Specifically, since the shape of the player is unlikely to change suddenly within a certain time, whether the player is blocked can be determined by the area of the player mark frame, and when the player is blocked, the player mark frame is divided.
The division point is a division point of a division line of a rectangular frame of a player for which a blocking phenomenon can be determined, and may be a center point of a mark frame of the player for which the blocking phenomenon exists, or the like.
S110: and dividing the player mark frame according to the dividing points to form a new player mark frame.
Specifically, when the player mark frame is a player rectangular frame, when the division point of the player rectangular frame is obtained, the corresponding division line may be obtained according to the division point, for example, two lines perpendicular to each other may be determined as the division line of the player rectangular frame according to the division point.
S112: and tracking and displaying the mark frame of the player in the segmented current frame through Kalman filtering.
Specifically, after the sportsman rectangle frame that shelters from exists is cut apart, sportsman's mark frame in the current frame after the rethread kalman filter is cut apart tracks the demonstration, can avoid because the sportsman shelters from, and sportsman's tracking lost condition in the kalman filter appears.
According to the method for tracking the football players in the video, whether the football players are shielded is judged by detecting whether the area of the marker frame of the football players is increased or not, the processing is simple, and when the shielding exists, the marker frame of the football players can be divided through the dividing points, so that the marker frame of the football players can be accurately tracked through Kalman filtering, and the condition that the information of the football players is lost due to the shielding of the football players cannot occur.
In one embodiment, referring to fig. 2, fig. 2 is a flowchart of step S108 in the embodiment shown in fig. 1, and the step S108 of obtaining the dividing point of the player mark frame with the increased area may include:
s202: and acquiring the number of player pixels on two preset and opposite edges of the player marking frame with the increased area.
Specifically, referring to fig. 3, fig. 3 is a schematic diagram of the player marking frame for blocking players in the embodiment shown in fig. 2, where the sides AB and CD are two preset and opposite sides, and both the two sides have player pixels, and when the numbers of the player pixels on the two sides are different, it can be considered that the blocked player has a front-back positional relationship under the current viewing angle, and therefore, in order to achieve the accuracy of the division, the previous player marking frame needs to be divided first.
S204: when the numbers of the pixels of the player on the two preset opposite edges are not equal, the length and the width of each player marking frame before the player marking frame is not shielded are obtained according to the coordinates of the player marking frame with the increased area.
Specifically, as shown in fig. 3, the area of the player mark frame ABCD is increased, and the length L and the width W of the player mark frame in which the occlusion exists and the length and the width of the player mark frame before the occlusion exist can be obtained, as shown in fig. 3, the original first player mark frame AEKF has the length AF and the width AE, and the original second player mark frame GHCL has the length HC and the width GH.
S206: and calculating the division points of the player marking frames according to the length and the width of each player marking frame before the player marking frames are not shielded.
The abscissa of the division point is obtained by ((AE + GH) -W)/2, and the ordinate of the division point is obtained by ((AF + HC) -L)/2, such as the O point in FIG. 3, through which division lines perpendicular to AB and AC, such as PQ and RS, are drawn.
And S208, when the number of the pixels of the player on the two preset opposite edges is equal, acquiring the central point of the player mark frame as a segmentation point.
When the numbers of the pixels of the player on the two edges are the same, the player with the shielding can be considered to have no front-back position relation under the current visual angle, and therefore the central point of the player marking frame with the shielding can be directly used as a dividing point.
In this embodiment, the number of the player pixels on the two preset and opposite edges is used for judging whether the blocked player has a front-back position relation or not under the current visual angle, and when the front-back position relation exists, the length and the width of each player mark frame before the blocking are obtained through the coordinates of the player mark frame with the increased area, so that the segmentation point can be accurately determined, and the segmentation precision is improved.
In one embodiment, referring to fig. 4, fig. 4 is a flowchart of step S112 in the embodiment shown in fig. 1, and the step S112 of performing the tracking display on the marker box of the player in the segmented current frame through the kalman filter may include:
s402: and tracking the marker frames of the players in the segmented current frame through Kalman filtering.
Specifically, when the blocking player exists, the blocking player is divided, and then the marking frame of the player in the divided current frame is tracked through Kalman filtering.
S404: and acquiring the coordinates of the tracked player mark frame.
Specifically, the player coordinates may be calculated by a player marking frame, for example, the center point of the player marking frame may be used as the player coordinates, and a certain corner point of the player marking frame may also be used as the marking frame, which is not limited herein.
S406: and acquiring a first player projection coordinate of the coordinate projection of the player mark frame to the target plane.
In particular, the target plane may be a two-dimensional plane, which corresponds to the pitch, which may be understood as a scaled down pitch plane. And in order to visually reveal the positions of the players on the course to the user watching the video, the determined player coordinates in the current frame are projected to the first player projection coordinates in the object plane, as shown in fig. 5.
S408: and in the target plane, tracking and displaying the player identification of the position where the corresponding player coordinate in the last frame is projected to the second player projection coordinate in the target plane according to the first player projection coordinate.
Specifically, the player identifier is an icon that can uniquely identify the player, and may be, for example, a circular icon, a triangular icon, or a figure icon with a number. In order to realize the tracking of the player, the coordinate of the corresponding player in the previous frame can be projected to the player identification at the projection coordinate of the second player in the target plane and moved to the projection coordinate of the first player, so that the position of the player is visually represented to be changed, and the tracking of the player is realized.
In the embodiment, the player mark frames in the current frame are tracked through the kalman filter, and in order to provide visual representation for the user, the tracking image under the view angle of the camera is converted into the target plane, so that the user experience is improved.
In one embodiment, please refer to fig. 6, fig. 6 is a flowchart of step S112 in the embodiment shown in fig. 1, and the step S112 of tracking the marker frame of the player in the segmented current frame through kalman filtering may include:
s602: calculating an error covariance matrix between the predicted value and the true value of the player mark box in the current frame by the following formula:
wherein the content of the first and second substances,
is a predicted value at the time k, wherein,
is an estimate of the time k-1, uk-1For the current frame at the time k-1, A is a conversion matrix for converting the predicted value at the previous time into the predicted value at the current time, B is a conversion matrix for converting the input value at the previous time into the predicted value at the current time, Pk' is the error covariance matrix between the predicted value and the true value at the k time, xkThe real value at the moment k is obtained, and Q is a covariance matrix of system noise;
s604: the kalman gain is calculated by the following equation:
Kk＝P′kHT(HP′kHT+R)-1and is and
wherein, KkFor the Kalman gain at time k, H is the conversion matrix for converting the predicted value to the measured value at the same time, R is the covariance matrix for measuring the noise variance, zkIs the measured value at the time k;
s606: calculating an error covariance matrix between the estimated value and the real value based on the Kalman gain and the error covariance matrix between the predicted value and the real value by the following equation:
s608: and obtaining a real value according to the error covariance matrix between the estimated value and the real value, and tracking the real value.
Specifically, the state difference equation in kalman filtering is:
xk＝Axk-1+Buk-1+wk-1(4)
wherein xkMarking boxes, x, for players at time kk-1The boxes are marked for the players at time k-1 and are n x 1 in size. A is a conversion matrix for converting the predicted value at the previous moment into the predicted value at the current moment, and the size of the conversion matrix is n multiplied by n. u. ofk-1The current frame at time k-1, the size is k × 1. B is the previous momentThe input value is converted into a conversion matrix of a predicted value at the current moment, and the size of the conversion matrix is n multiplied by k. Random variable wk-1The system noise at time k-1. This equation shows that the latest state of the system is equal to the previous state plus the effect of the system inputs, plus the noise of the system.
However, the measurement is often subject to errors, and in most cases it is difficult to construct an accurate model of a real moving system, thus introducing feedback. The measured values are mapped by system state variables, and the equation is in the form of:
zk＝Hxk+vk(5)
wherein z iskThe measured value at the time k is m multiplied by 1, H is a conversion matrix for converting the predicted value at the same time into the measured value, vkIs the measurement noise at time k.
For systematic noise w in the equation of statek-1And measuring the noise vkAssume that the following multivariate gaussian distribution is followed, and w, v are independent of each other. Q is the covariance matrix of the system noise and R is the covariance matrix of the measured noise variance.
p(w)～N(0，Q)
p(v)～N(0，R) (6)
Wherein the content of the first and second substances,
is a predicted value at the time k,
is an estimate of the time at the time k,
for the prediction of the measured value at time k, an estimate is obtained from the general feedback idea:
wherein the content of the first and second substances,
is the residual, i.e. the difference between the predicted and measured values. If this term equals 0, it indicates that the predicted and measured values are equal.
Solving for KkAnd constructing a covariance matrix of errors between the estimated values and the true values, wherein the covariance matrix is obtained by calculating the covariance matrix, namely the mean square error by using the sum of diagonal elements of the covariance matrix, because the diagonal elements of the covariance matrix are the variances.
Substituting the previously obtained estimate into this can be simplified:
after simplification, the following is obtained:
similarly, a covariance matrix of the error between the predicted value and the true value can be obtained:
and finally obtaining:
continue to unfold
Next, the minimum mean square error is used, and the diagonal elements of the covariance matrix are the variances. Thus, the diagonal elements of the matrix P are summed, and the operator is represented by the letter T.
The minimum mean square error is to minimize the above equation for the unknown KkBy taking the derivative, and making the derivative function equal to 0, K can be foundkThe value of (c).
Kk＝P′kHT(HP′kHT+R)-1
Note that this equation KkThe transformation matrix H is a constant and the measurement noise covariance R is also a constant. Thus KkWill be related to the error covariance of the predicted values. Assuming that the matrix dimensions in the above equation are all 1 × 1 in size, and assuming that H is 1, Pk' is not equal to 0. Then Pk' can be written as follows:
so PkThe larger the ` then K `kThe larger the weight will be, the more important the feedback, if Pk' equal to 0, i.e. the predicted value and the true value are equal, then KkWhen 0, the estimated value is equal to the predicted value.
Will calculate KkInverse substitution of PkIn (3), P can be simplifiedk：
Thus K for each step in the recursion formulakIt is calculated and the estimated covariance of each step can be calculated. But KkThe formula of (1) also adds a covariance matrix of errors between predicted values and true values.
First note that the recurrence form of the predicted values is:
and is
Thus, P 'is also obtained'kA recurrence formula of (c). Therefore, only the initial P is setkAnd the process can be continued.
In the above embodiment, by kalman filtering, as long as the state of the player mark frame at the previous time is known, the state of the rectangular frame of the player at the current time can be obtained, the processing amount is small, and the phenomena such as jamming and the like cannot occur.
In one embodiment, referring to fig. 7, fig. 7 is a flowchart of a coordinate projection relationship obtaining step in an embodiment, where the coordinate projection relationship obtaining step may be performed before the embodiment shown in fig. 1, and the coordinate projection relationship obtaining step may include:
s702: the spherical field lines in the spherical field region are obtained by hough transform.
Specifically, referring to fig. 8 to 10, the histogram of the gray levels of the R channel (red channel), the G channel (green channel), and the B channel (blue channel) of the current frame is shown, wherein the abscissa identifies the corresponding gray level after the gray level is normalized to 0 to 1, wherein the area occupying a larger area is the field area, specifically, the area having a higher gray level and a higher peak in the histogram of the R channel is the runway area, and the part having a more dispersed distribution is the field area; the part with lower gray value and relatively steep in the G channel histogram is a tree area around the runway, and the part with more dispersed and higher gray value is a ball field area; in the B channel histogram, the distribution of blue color is more uniform. After selecting a proper threshold value according to the histogram, segmenting to obtain a binary image; then removing salt and pepper noise by using median filtering, and selecting the maximum connected region in the image to separate the maximum connected region from the surrounding environment; finally, a function of filling holes by using Matlab and mathematical morphology operations such as opening and closing are used to obtain a complete court area mask as shown in fig. 11.
Referring to fig. 12, a schematic diagram of a white line region in a field area in an embodiment is shown, wherein the region of the spherical field line is segmented by a threshold value using the color characteristics of the white line, and the image edge can be found by using the canny operator. The effect of extracting the white line area of the court after denoising by using median filtering and morphological operations is shown in fig. 12. The effect of determining marginality using the hough transform is shown in fig. 13.
S704: and determining the coordinate projection relation between the current frame and the target plane according to the spherical field lines.
Referring to fig. 13, boundary points of the field area and proportions of respective line segments in the real field area may be obtained. Taking the coordinate system created in fig. 13 as an example, the coordinates of the X point may be obtained by finding the intersection point where the Y value is the largest in the field area, and after obtaining the coordinates of the X point, the coordinates of the U point and the V point may be obtained from the intersection points of the line segments, and then, the Z ═ VU/XU may be calculated, where V 'U'/X 'U' ═ a and Y 'U'/X 'a in the real field area are assumed, and h ═ Z/a is assumed, and since the perspective deformation is continuous, the wide-angle distortion that may be generated by the camera may be ignored, and Y' U '/X' U ═ h × a is calculated, so that the field area shown in fig. 13 has the following relationship YU/XU ═ h × a, and thus, the coordinates of the Y point may be determined. And then, calculating coordinates of the point Z and the point T, calculating the coordinates of the point W, and obtaining the coordinates of the point Z by combining YU/XU (h × a) because the values of X 'W'/Z 'W' in the real spherical area are known, and obtaining the coordinates of the point Z by combining with the straight line XZ/XU ═ h × a, and calculating the intersection point of the straight line XZ and the straight line L1 because the straight line XZ, the straight line YT and the straight line L1 intersect at one point in the perspective relation, so that obtaining the equation of the straight line YT and the equation of the straight line ZT, and obtaining the coordinates of the point T by solving the intersection point of the straight line YT and the straight line.
After the coordinates of the X point, the Y point, the Z point and the T point are determined, the coordinate projection relation between the current frame and the target plane can be obtained according to coordinate transformation, the two-dimensional coordinate transformation can be expressed as a 3X 3 matrix, the degree of freedom is 8, and the obtained coordinates of the X point, the Y point, the Z point and the T point are brought into the matrix to be solved, so that the matrix to be solved, namely the coordinate projection relation between the current frame and the target plane, can be obtained.
In the embodiment, the coordinate projection relation between the current frame and the target plane can be obtained by obtaining the court line in the court area and according to the coordinates of the key points of the court line, so that the conversion from the video image to the target plane can be realized, the tracked players can be presented on the target plane in real time in the subsequent player tracking process, and the method is simple and visual.
In one embodiment, multi-channel gray scale joint comparison can be further adopted to identify players of different teams, for example, when one team of players is a red coat player and the other team of players is an orange coat player, the goalkeeper has low gray scales in the R channel and the G channel and has a high gray scale in the B channel; the red clothing team member is high in R channel gray scale, but low in G channel B channel gray scale; the orange-clothing player has high gray level in the R channel G channel, but has low gray level in the B channel. Different players can thus still be divided according to the color-based threshold segmentation. Thus, after the player is tracked by the Kalman filtering, the player can be divided according to the color.
In this embodiment, only can realize the division to the sportsman through the colour, easy operation, the rate of accuracy is high.
Referring to fig. 14, fig. 14 is a schematic diagram of an embodiment of a player tracking device in a video, the device comprising:
a court area obtaining module 100, configured to obtain a court area in a current frame of the video.
And the player mark frame generation module 200 is used for acquiring player pixels in the ball field area and generating the player mark frame according to the player pixels.
The dividing point obtaining module 300 is configured to, when the area of the player marking frame in the current frame increases, obtain a dividing point of the player marking frame with the increased area.
And a dividing module 400 for dividing the player marking frame according to the dividing points to form a new player marking frame.
And the tracking module 500 is used for tracking and displaying the player mark frame in the segmented current frame through kalman filtering.
In one embodiment, the segmentation point obtaining module 300 may include:
and the number acquisition unit is used for acquiring the number of the player pixels on two preset and opposite edges of the player marking frame with the increased area.
The division point determining unit is used for acquiring the length and the width of each player marking frame before the player marking frames are not shielded according to the coordinates of the player marking frames with the increased areas when the number of the player pixels on the two preset and opposite edges is not equal; calculating the dividing points of the player marking frames according to the length and the width of each player marking frame before the player marking frames are not shielded; when the number of the player pixels on the two preset opposite edges is equal, the central point of the player mark frame is obtained as a segmentation point.
In one embodiment, the tracking module 500 may include:
and the tracking unit is used for tracking the mark frame of the player in the segmented current frame through Kalman filtering.
And the first coordinate calculation unit is used for acquiring the coordinates of the tracked player mark frame.
And the second coordinate calculation unit is used for acquiring the first player projection coordinate of the coordinate projection of the player marking frame to the target plane.
And the display unit is used for tracking and displaying the player identification of the position of the coordinate projection of the corresponding player in the previous frame to the coordinate projection of the second player in the target plane according to the coordinate projection of the first player in the target plane.
In one embodiment, the tracking unit may include:
a first matrix calculation unit for calculating an error covariance matrix between the predicted value and the true value of the player mark frame in the current frame by the following formula:
wherein the content of the first and second substances,
is a predicted value at the time k, wherein,
estimate for time k-1Evaluation of uk-1For the current frame at the time k-1, A is a conversion matrix for converting the predicted value at the previous time into the predicted value at the current time, B is a conversion matrix for converting the input value at the previous time into the predicted value at the current time, Pk' is the error covariance matrix between the predicted value and the true value at the k time, xkAnd Q is the covariance matrix of the system noise, which is the true value of the k moment.
A kalman gain calculating unit for calculating a kalman gain by the following formula:
Kk＝P′kHT(HP′kHT+R)-1and is and
wherein, KkFor the Kalman gain at time k, H is the conversion matrix for converting the predicted value to the measured value at the same time, R is the covariance matrix for measuring the noise variance, zkIs the measured value at time k.
A second matrix calculation unit for calculating an error covariance matrix between the estimated value and the true value based on the Kalman gain and the error covariance matrix between the predicted value and the true value by the following formula:
and the real value calculating unit is used for obtaining a real value according to the error covariance matrix between the estimated value and the real value and tracking the real value.
In one embodiment, the apparatus may further include: the coordinate projection relation calculation unit is used for obtaining a court line in the court area through Hough transformation; and determining the coordinate projection relation between the current frame and the target plane according to the spherical field lines.
The above definition of the player tracking device in video can refer to the above definition of the player tracking method in video, and is not described herein again.
The technical features of the embodiments described above may be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the embodiments described above are not described, but should be considered as being within the scope of the present specification as long as there is no contradiction between the combinations of the technical features.
The above-mentioned embodiments only express several embodiments of the present invention, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the inventive concept, which falls within the scope of the present invention. Therefore, the protection scope of the present patent shall be subject to the appended claims.
Claims (8)
1. A method for tracking a player in a video, the method comprising:
acquiring a court area in a current frame of a video;
obtaining player pixels in the court area;
generating a player mark frame according to the player pixels;
when the area of the player marking frame in the current frame is increased, acquiring a segmentation point of the player marking frame with the increased area;
dividing the player marking frame according to the dividing points to form a new player marking frame;
tracking and displaying the player mark frame in the segmented current frame through Kalman filtering;
the step of obtaining the division point of the player mark frame with the increased area includes:
acquiring the number of player pixels on two preset and opposite edges of the player marking frame with the increased area;
when the number of the pixels of the player on the two preset opposite sides is not equal, the length and the width of each player marking frame before the player marking frame is not shielded are obtained according to the coordinates of the player marking frames with the increased areas;
calculating the dividing points of the player marking frames according to the length and the width of each player marking frame before the player marking frames are not shielded;
and when the number of the player pixels on the two preset opposite edges is equal, acquiring the central point of the player mark frame as the segmentation point.
2. The method of claim 1, wherein the step of performing a tracking display of the player mark box in the segmented current frame through kalman filtering includes:
tracking the player mark frame in the segmented current frame through Kalman filtering;
acquiring the coordinate of the tracked player mark frame;
acquiring a first player projection coordinate of the player marking frame projected to a target plane, wherein the first player projection coordinate refers to a coordinate of the player marking frame tracked in the current frame projected to the target plane;
in the target plane, according to the first player projection coordinate, tracking and displaying a player identifier of which the corresponding player coordinate in the previous frame is projected to a second player projection coordinate in the target plane, wherein the second player projection coordinate refers to a coordinate of which the corresponding player coordinate in the previous frame is projected to the target plane.
3. The method of claim 2, wherein the step of tracking the player marker box in the segmented current frame by kalman filtering comprises:
calculating an error covariance matrix between the predicted value and the true value of the player mark box in the current frame by the following formula:
wherein the content of the first and second substances,
is k atThe predicted value of the moment, wherein,
is an estimate of the time k-1, uk-1For the current frame at the time k-1, A is a conversion matrix for converting the predicted value at the previous time into the predicted value at the current time, B is a conversion matrix for converting the input value at the previous time into the predicted value at the current time, Pk' is the error covariance matrix between the predicted value and the true value at the k time, xkThe real value at the moment k is obtained, and Q is a covariance matrix of system noise;
the kalman gain is calculated by the following equation:
Kk＝P′kHT(HP′kHT+R)-1and is and
wherein, KkFor the Kalman gain at time k, H is the conversion matrix for converting the predicted value to the measured value at the same time, R is the covariance matrix for measuring the noise variance, zkIs the measured value at the time k;
calculating an error covariance matrix between the estimated value and the real value based on the Kalman gain and the error covariance matrix between the predicted value and the real value by the following formula:
and obtaining the real value according to an error covariance matrix between the estimated value and the real value, and tracking the real value.
4. The method of claim 2, further comprising:
obtaining a court line in the court area through Hough transform;
and determining the coordinate projection relation between the current frame and a target plane according to the spherical field lines.
5. A player tracking device in a video, the device comprising:
the court area acquisition module is used for acquiring a court area in a current frame of the video;
the player mark frame generation module is used for acquiring player pixels in the ball field area and generating a player mark frame according to the player pixels;
a division point acquisition module, configured to acquire, when the area of the player mark frame in the current frame increases, a division point of the player mark frame whose area increases;
the segmenting module is used for segmenting the player marking frame according to the segmenting point to form a new player marking frame;
the tracking module is used for tracking and displaying the player mark frame in the segmented current frame through Kalman filtering;
the segmentation point acquisition module comprises:
the number acquisition unit is used for acquiring the number of player pixels on two preset and opposite edges of the player marking frame with the increased area;
the dividing point determining unit is used for acquiring the length and the width of each player marking frame before the player marking frames are not shielded according to the coordinates of the player marking frames with increased areas when the number of the player pixels on the two preset and opposite edges is not equal; calculating the dividing points of the player marking frames according to the length and the width of each player marking frame before the player marking frames are not shielded; and when the number of the player pixels on the two preset opposite edges is equal, acquiring the central point of the player mark frame as the segmentation point.
6. The apparatus of claim 5, wherein the tracking module comprises:
the tracking unit is used for tracking the player mark frame in the segmented current frame through Kalman filtering;
the first coordinate calculation unit is used for acquiring the coordinates of the tracked player mark frame;
the second coordinate calculation unit is used for acquiring a first player projection coordinate of the player marking frame projected to a target plane, wherein the first player projection coordinate refers to the coordinate of the player marking frame tracked in the current frame projected to the target plane;
and the display unit is used for tracking and displaying the player identification of a position where the coordinate of the corresponding player in the previous frame is projected to the projection coordinate of a second player in the target plane according to the projection coordinate of the first player in the target plane, wherein the projection coordinate of the second player refers to the coordinate of the corresponding player in the previous frame projected to the target plane.
7. The apparatus of claim 6, wherein the tracking unit comprises:
a first matrix calculation unit for calculating an error covariance matrix between the predicted value and the true value of the player mark frame in the current frame by the following formula:
wherein the content of the first and second substances,
is a predicted value at the time k, wherein,
is an estimate of the time k-1, uk-1For the current frame at the time k-1, A is a conversion matrix for converting the predicted value at the previous time into the predicted value at the current time, B is a conversion matrix for converting the input value at the previous time into the predicted value at the current time, Pk' is the error covariance matrix between the predicted value and the true value at the k time, xkThe real value at the moment k is obtained, and Q is a covariance matrix of system noise;
a kalman gain calculating unit for calculating a kalman gain by the following formula:
Kk＝P′kHT(HP′kHT+R)-1and is and
wherein, KkFor the Kalman gain at time k, H is the conversion matrix for converting the predicted value to the measured value at the same time, R is the covariance matrix for measuring the noise variance, zkIs the measured value at the time k;
a second matrix calculation unit for calculating an error covariance matrix between the estimated value and the true value based on the kalman gain and the error covariance matrix between the predicted value and the true value by the following formula:
and the real value calculating unit is used for obtaining the real value according to the error covariance matrix between the estimated value and the real value and tracking the real value.
8. The apparatus of claim 6, further comprising:
the coordinate projection relation calculation unit is used for obtaining a court line in the court area through Hough transformation; and determining the coordinate projection relation between the current frame and a target plane according to the spherical field lines.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710210009.9A CN107194308B (en) | 2017-03-31 | 2017-03-31 | Method and device for tracking ball players in video |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710210009.9A CN107194308B (en) | 2017-03-31 | 2017-03-31 | Method and device for tracking ball players in video |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107194308A CN107194308A (en) | 2017-09-22 |
| CN107194308B true CN107194308B (en) | 2020-06-09 |
Family
ID=59871759
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710210009.9A Active CN107194308B (en) | 2017-03-31 | 2017-03-31 | Method and device for tracking ball players in video |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107194308B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108924627A (en) * | 2018-08-23 | 2018-11-30 | 北京字节跳动网络技术有限公司 | Position distribution display methods, device, equipment and the storage medium of Moving Objects |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101833771A (en) * | 2010-06-03 | 2010-09-15 | 北京智安邦科技有限公司 | Tracking device and method for solving multiple-target meeting dodging |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6781566B2 (en) * | 2001-08-22 | 2004-08-24 | Aurora Systems, Inc. | Residual DC bias correction in a video imaging device |
-
2017
- 2017-03-31 CN CN201710210009.9A patent/CN107194308B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101833771A (en) * | 2010-06-03 | 2010-09-15 | 北京智安邦科技有限公司 | Tracking device and method for solving multiple-target meeting dodging |
Non-Patent Citations (1)
| Title |
|---|
| 基于卡尔曼滤波的橘小实蝇成虫运动轨迹优化跟踪;文韬等;《农业工程学报》;20140830;第30卷(第15期);第197-205页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107194308A (en) | 2017-09-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Crabb et al. | Real-time foreground segmentation via range and color imaging | |
| CN109903312B (en) | Football player running distance statistical method based on video multi-target tracking | |
| CN109076198B (en) | Video-based object tracking occlusion detection system, method and equipment | |
| Xu et al. | Tracking football players with multiple cameras | |
| CN109271937B (en) | Sports ground marker identification method and system based on image processing | |
| JP5036084B2 (en) | Video processing apparatus, video processing method, and program | |
| CN106991686B (en) | A kind of level set contour tracing method based on super-pixel optical flow field | |
| CN107240120B (en) | Method and device for tracking moving target in video | |
| CN107507222B (en) | Anti-occlusion particle filter target tracking method based on integral histogram | |
| Cvejic et al. | The effect of pixel-level fusion on object tracking in multi-sensor surveillance video | |
| JP6850751B2 (en) | Object tracking device, object tracking method, and computer program | |
| Walton et al. | Accurate real-time occlusion for mixed reality | |
| KR20110133416A (en) | Video processing method for 3d display based on multi-thread scheme | |
| CN107194308B (en) | Method and device for tracking ball players in video | |
| CN113240656B (en) | Visual positioning method and related device and equipment | |
| CN107194306B (en) | Method and device for tracking ball players in video | |
| Mauthner et al. | Visual tracking of athletes in beach volleyball using a single camera | |
| CN107194954B (en) | Player tracking method and device of multi-view video | |
| Xu et al. | Head tracking using particle filter with intensity gradient and color histogram | |
| CN106778822B (en) | Image straight line detection method based on funnel transformation | |
| CN111192308B (en) | Image processing method and device, electronic equipment and computer storage medium | |
| Park et al. | Robust estimation of heights of moving people using a single camera | |
| CN112287901A (en) | Target object detection method and device, electronic equipment and storage medium | |
| Kelly et al. | Pedestrian detection using stereo and biometric information | |
| Herrmann et al. | Feature-based localization refinement of players in soccer using plausibility maps |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |