CN108986151B

CN108986151B - Multi-target tracking processing method and equipment

Info

Publication number: CN108986151B
Application number: CN201710399438.5A
Authority: CN
Inventors: 袁誉乐; 胡学峰; 赵勇; 谭兵
Original assignee: Huawei Technologies Co Ltd
Current assignee: Haisi Technology Co ltd
Priority date: 2017-05-31
Filing date: 2017-05-31
Publication date: 2021-12-03
Anticipated expiration: 2037-05-31
Also published as: CN108986151A

Abstract

The application discloses a multi-target tracking processing method and equipment, relates to the technical field of information processing, and is used for solving the problem that in the prior art, due to the fact that target fusion cannot be well processed, the track of a tracked target is discontinuous, and therefore target tracking fails. The method comprises the following steps: obtaining a candidate matrix according to the distance between each target in the multiple targets and the template associated with each target, and determining whether at least two targets in the multiple targets are fused according to the positions of a first indicator and a second indicator in the candidate matrix; under the condition that fusion of at least two targets exists in a plurality of targets, adding a target template associated with the fused targets, wherein the fused targets are obtained by fusing the at least two targets, and history information of template records associated with each of the at least two targets is at least recorded in the target template; the embodiment of the invention is applied to a target fusion scene by utilizing a target template to track a fusion target.

Description

Multi-target tracking processing method and equipment

Technical Field

The embodiment of the invention relates to the technical field of information processing, in particular to a multi-target tracking processing method and equipment.

Background

The video monitoring scenes are complex many times, and when a plurality of targets appear in the same scene, the target fusion and splitting conditions are easy to occur. For example, multiple objects go together, i.e., object fusion occurs, and then multiple objects go together are separated again, i.e., object fragmentation occurs.

In the conventional technology, on one hand, a mean shift method is usually adopted for target splitting and target fusion, and the method finds a target position through iterative operation to realize target tracking. However, the mean shift method often fails in tracking a small target and a fast moving target, and when the target is completely occluded, the features of the occluded target may be incomplete or disappear, resulting in interruption of the tracking trajectory of the occluded target. On the other hand, modeling is generally carried out on the multi-target real-time tracking problem, and a generative probability model of the multi-target real-time tracking problem is established; performing offline training on a correctly labeled training set aiming at a global conditional probability item in a generative probability model, performing global behavior prediction universally suitable for various scenes, and performing real-time online training on a local behavior prediction suitable for each target by using tracking data of the target before a current frame aiming at a local conditional probability item in the generative probability model; and combining the global behavior prediction and the local behavior prediction to obtain the behavior prediction of the target, and performing multi-target tracking according to the predicted target behavior.

However, in the conventional technology, multi-target tracking is performed by establishing a probability model of a single target, and association among multiple targets is not involved, so that the condition of target fusion cannot be well handled.

Disclosure of Invention

The embodiment of the invention provides a multi-target tracking processing method and equipment, which are used for solving the problem of target tracking failure caused by discontinuous track of a tracked target due to the fact that target fusion cannot be well processed in the prior art.

In order to achieve the purpose, the embodiment of the invention adopts the following technical scheme:

in a first aspect, an embodiment of the present invention provides a multi-target tracking processing method, including: obtaining a candidate matrix according to the distance between each target in the plurality of targets and the template associated with each target, wherein the candidate matrix comprises a plurality of first indicators and a plurality of second indicators, one first indicator indicates that the distance between the template corresponding to the position of the first indicator and the target corresponding to the position of the first indicator is the minimum in the row where the first indicator is located, and one second indicator indicates that the target corresponding to the position of the second indicator is matched with the template corresponding to the position of the second indicator; the template is used for recording historical information of a target associated with the template; determining whether at least two targets in the plurality of targets are fused according to the positions of the first indicator and the second indicator in the candidate matrix; under the condition that fusion of at least two targets exists in a plurality of targets, generating a target template associated with the fused targets, wherein the fused targets are obtained by fusing the at least two targets, and history information of a template record associated with each of the at least two targets is at least recorded in the target template; and tracking the fused target by using the target template.

According to the multi-target tracking processing method, a candidate matrix is constructed according to the distance between each target and each template, and the positions of the first indicator and the second indicator in the candidate matrix are utilized to accurately determine which targets are subjected to target fusion or target splitting at which time point (namely, the specific frame number in the video). And judging a plurality of targets with target fusion, and tracking the fused targets by using the newly added target template in the time period of the target fusion, so that the problem of discontinuous tracking tracks generated after the target fusion can be avoided, and the problem of tracking failure caused by the target fusion can be reduced. In addition, the method provided by the application only relates to matrix operation with small dimensionality, and is lower in calculation complexity compared with a method adopting mean shift in the traditional technology, suitable for being implemented on a chip or an embedded platform and lower in power consumption.

With reference to the first aspect, in a first possible implementation manner of the first aspect, determining whether at least two targets in the multiple targets are fused according to positions of the first indicator and the second indicator in the candidate matrix includes: and determining that the target associated with the template corresponding to the position of each first target indicator in the at least one first target indicator is fused with the target associated with the template corresponding to the position of the second target indicator when at least one first target indicator and at least one second target indicator exist in the candidate matrix, wherein the first target indicator is any one first indicator in the first column, and the second target indicator is any one second indicator in the first column.

With reference to the first aspect or the first possible implementation manner of the first aspect, in a second possible implementation manner of the first aspect, generating an object template associated with a fused object when it is determined that at least two objects exist in a plurality of objects and are fused, includes: acquiring a fusion mark of a template associated with each of at least two targets, wherein the fusion mark of one template is used for indicating whether the targets associated with the template are fused or not; under the condition that the fusion mark of each target-associated template indicates that the targets associated with each template are fused, historical information of the targets recorded by each target-associated template is added to the target template, and one template records the historical information of the targets associated with the template, so that the historical information of the targets recorded by each target-associated template is added to the target template, and the continuous track of the fused targets tracked by the target template can be ensured.

With reference to any one of the first aspect to the second possible implementation manner of the first aspect, in a third possible implementation manner of the first aspect, the method provided by the embodiment of the present invention further includes: determining that at least one second column exists in the candidate matrix, wherein only a second indicator exists in the second column, updating the template corresponding to the position of the second indicator in the second column by using the target corresponding to the position of the second indicator in the second column, and acquiring an updated template, wherein the second column is any one column in the candidate matrix; and determining that at least one first column exists in the candidate matrix, at least one first indicator and at least one second indicator exist in the first column, updating the template corresponding to the position of the second indicator in the first column by using the target corresponding to the position of the second indicator in the first column, and updating the template corresponding to the position of each first indicator in the at least one first indicator, wherein the track of the template can be ensured to be continuous by using the target to update the template in real time.

With reference to any one of the first aspect to any one of the third possible implementation manners of the first aspect, in a fourth possible implementation manner of the first aspect, determining that at least one second column exists in the candidate matrix, where only the second indicator exists in the second column, and updating, by using a target corresponding to a position where the second indicator exists in the second column, a template corresponding to the position where the second indicator exists in the second column, to obtain an updated template includes: adding the coordinates of the center point of the target corresponding to the position of the second indicator in the second column to the track of the template corresponding to the position of the second indicator in the second column; determining that at least one first column exists in the candidate matrix, at least one first indicator and at least one second indicator exist in the first column, updating a template corresponding to a position of the second indicator in the first column by using a target corresponding to a position of the second indicator in the first column, and updating a template corresponding to a position of each first indicator in the at least one first indicator, wherein the method comprises the following steps: the coordinates of the center point of the target corresponding to the position of the second indicator in the first column are added to the track of the template corresponding to the position of each first indicator and the track of the template corresponding to the position of the second indicator, and the coordinates of the center point of the target are added to each template, so that the track continuity of the templates can be guaranteed.

With reference to any one of the first aspect to the fourth possible implementation manner of the first aspect, in a fifth possible implementation manner of the first aspect, the method provided by the embodiment of the present invention further includes: and updating the first confidence degrees of the updated templates, wherein the first confidence degree of one template is used for representing the credibility of the target corresponding to the template tracked by the template, and the accuracy of the target tracking by the template can be provided in the subsequent tracking process by updating the first confidence degrees of the templates.

With reference to any one of the first aspect to any one of the fifth possible implementation manners of the first aspect, in a sixth possible implementation manner of the first aspect, each of the multiple templates has a fusion tag, where the fusion tag is a first tag or a second tag, the first tag indicates that no fusion occurs in targets associated with the templates, and the second tag indicates that the fusion occurs in targets associated with the templates, and the method provided by the embodiment of the present invention further includes: determining one or more fusion templates from the plurality of templates according to the fusion mark of each template in the plurality of templates, wherein the fusion template refers to the template with the fusion mark as the second mark, and determining at least one candidate target in the tracking area of each fusion template in the one or more fusion templates; performing the following steps for at least one candidate target within each fused template tracking area to determine split targets within each fused template tracking area: if it is determined that at least one candidate target in a fusion template tracking area has a target which has no template and is intersected with the fusion template tracking area, determining that the fusion target is split; acquiring image characteristic information of each split target in a plurality of split targets from the image of the fusion target; determining a template with similarity meeting preset requirements with the image characteristic information of each split target as a template corresponding to each split target according to the image characteristic information of each split target and historical information of each associated target recorded by each template before fusion; the template corresponding to each split target is updated by using the information of the current moment of each split target, and the split targets can be conveniently tracked subsequently by determining the point position of the fusion target at which time point the fusion target is split, so that the continuous track of each split target is ensured.

With reference to the first aspect to the sixth possible implementation manner of the first aspect, in a seventh possible implementation manner of the first aspect, in an embodiment of the present invention, each of the multiple templates further includes a second confidence level, and the method provided in the embodiment of the present invention further includes: and if the second confidence coefficient is smaller than or equal to the preset threshold, deleting the template with the second confidence coefficient smaller than or equal to the preset threshold so as to prevent redundant templates from occupying the space of the multi-target tracking processing equipment.

With reference to the first aspect to the seventh possible implementation manner of the first aspect, in an eighth possible implementation manner of the first aspect, the method provided by the embodiment of the present invention further includes: and when the second confidence coefficient of any template is determined to be smaller than or equal to the preset threshold, deleting the template with the second confidence coefficient smaller than or equal to the preset threshold, and deleting the template with the second confidence coefficient smaller than or equal to the preset threshold to use the redundant template to occupy the space of the multi-target tracking processing equipment.

Combining the first aspect with the first aspectAny one of eight possible implementation manners, in a ninth possible implementation manner of the first aspect, the candidate matrix further includes a third indicator, configured to indicate a distance between a target at a position where the third indicator is located and a template corresponding to the position where the third indicator is located, and the candidate matrix is obtained according to a distance between each target of the multiple targets and a template associated with each target, including: establishing a first matrix with M rows and N columns according to the distance between each target and the template associated with each target, wherein any element M in the first matrix_ijRepresenting the euclidean distance between a template identified as i and a target identified as j, where i 1.., M, j, N, M is the number of targets and N is the number of templates; determining the position of the minimum value of each row and the position of the minimum value of each column in the first matrix; according to the position of the minimum value of each row and the position of the minimum value of each column, assigning a position corresponding to the position of the minimum value of each row in the matrix to be selected to a first indicator, and assigning a position corresponding to the position of the minimum value of each column in the matrix to be selected to a third indicator; acquiring a target position according to the position of each row of minimum values and the position of each column of minimum values in the first matrix, wherein the target position is the position where the position of each row of minimum values is the same as the position of each column of minimum values; and assigning a second indicator to a position corresponding to the target position in the matrix to be selected to obtain a candidate matrix, wherein the matrix to be selected is a matrix with M rows and N columns, and each element in the matrix to be selected is an initial value.

In a second aspect, an embodiment of the present invention provides a multi-target tracking processing apparatus, including: the device comprises an obtaining unit, a calculating unit and a processing unit, wherein the obtaining unit is used for obtaining a candidate matrix according to the distance between each target in a plurality of targets and a template associated with each target, the candidate matrix comprises a plurality of first indicators and a plurality of second indicators, one first indicator represents that the distance between the template corresponding to the position of the first indicator and the target corresponding to the position of the first indicator is minimum in the row where the first indicator is located, and one second indicator represents that the target corresponding to the position of the second indicator is matched with the template corresponding to the position of the second indicator; the template is used for recording historical information of a target associated with the template; a determining unit, configured to determine whether at least two targets in the multiple targets are fused according to positions of the first indicator and the second indicator in the candidate matrix; the generating unit is used for generating a target template associated with the fusion target under the condition that at least two targets are determined to be fused in the plurality of targets, the fusion target is obtained by fusing the at least two targets, and historical information of template records associated with each of the at least two targets is at least recorded in the target template; and the tracking unit is used for tracking the fusion target by using the target template.

With reference to the second aspect, in a first possible implementation manner of the second aspect, the determining unit is specifically configured to determine that at least one first column exists in the candidate matrix, and when at least one first target indicator and at least one second target indicator exist in the first column, it is determined that a target associated with a template corresponding to a location where each first target indicator in the at least one first target indicator is located is fused with a target associated with a template corresponding to a location where the second target indicator is located, where the first target indicator is any one first indicator in the first column, and the second target indicator is any one second indicator in the first column.

With reference to the second aspect or the first possible implementation manner of the second aspect, in any one of the second possible implementation manners of the second aspect, the obtaining unit is further configured to obtain a fusion flag of a template associated with each of the at least two targets, where the fusion flag of one template is used to indicate whether the targets associated with the template are fused, and the generating unit is specifically configured to, when the fusion flag of the template associated with each target indicates that the targets associated with each template are fused, add history information of the targets recorded by each target-associated template to the target template.

With reference to any one of the second aspect to the second possible implementation manner of the second aspect, in a third possible implementation manner of the second aspect, the apparatus provided in the embodiment of the present invention further includes: the updating unit is used for determining that at least one second column exists in the candidate matrix, only a second indicator exists in the second column, and updating the template corresponding to the position of the second indicator in the second column by using the target corresponding to the position of the second indicator in the second column to obtain an updated template, wherein the second column is any one column in the candidate matrix; and the template updating module is used for determining that at least one first column exists in the candidate matrix, at least one first indicator and at least one second indicator exist in the first column, updating a template corresponding to the position of the second indicator in the first column by using a target corresponding to the position of the second indicator in the first column, and updating a template corresponding to the position of each first indicator in the at least one first indicator.

With reference to any one of the second aspect to any one of the third possible implementation manners of the second aspect, in a fourth possible implementation manner of the second aspect, the updating unit in the embodiment of the present invention is specifically configured to add the coordinates of the center point of the target corresponding to the position of the second indicator in the second column to the trajectory of the template corresponding to the position of the second indicator in the second column; or, the method is used for adding the coordinates of the center point of the target corresponding to the position of the second indicator in the first column to the track of the template corresponding to the position of each first indicator and to the track of the template corresponding to the position of the second indicator.

With reference to any one of the second aspect to the fourth possible implementation manner of the second aspect, in a fifth possible implementation manner of the second aspect, the updating unit is further configured to update a first confidence of the updated template, where the first confidence of one template is used to represent a confidence of tracking, by using the template, an object corresponding to the template.

With reference to any one of the second aspect to any one of the fifth possible implementation manners of the second aspect, in a sixth possible implementation manner of the second aspect, each of the plurality of templates has a fusion flag, where the fusion flag is a first flag or a second flag, the first flag indicates that no fusion occurs in targets associated with the template, and the second flag indicates that fusion occurs in targets associated with the template, and the determining unit is further configured to: determining one or more fusion templates from the plurality of templates according to the fusion mark of each template in the plurality of templates, wherein the fusion template refers to the template with the fusion mark as the second mark, and at least one candidate target positioned in the tracking area of each fusion template in the one or more fusion templates is determined; and for performing the following steps for at least one candidate target within each fused template tracking area to determine split targets within each fused template tracking area: if it is determined that at least one candidate target in a fusion template tracking area has a target which has no template and is intersected with the fusion template tracking area, determining that the fusion target is split; the acquisition unit is further used for acquiring image characteristic information of each split target in the split targets from the image of the fusion target; the determining unit is further used for determining a template, the similarity of which with the image characteristic information of each splitting target meets preset requirements, as a template corresponding to each splitting target according to the image characteristic information of each splitting target and historical information of respective associated targets recorded by each template before fusion; and the updating unit is also used for updating the template corresponding to each split target by using the information of the current moment of each split target.

With reference to any one of the second aspect to the sixth possible implementation manner of the second aspect, in a seventh possible implementation manner of the second aspect, each of the plurality of templates further includes a second confidence level, and the updating unit is further configured to: and if at least one unprocessed template exists in the plurality of templates, reducing the second confidence coefficient of the at least one unprocessed template by a preset step length, and acquiring the updated second confidence coefficient of each unprocessed template in the at least one unprocessed template.

With reference to any one of the second aspect to the seventh possible implementation manner of the second aspect, in an eighth possible implementation manner of the second aspect, the apparatus provided in the embodiment of the present invention further includes: and the deleting unit is used for deleting the template with the second confidence coefficient smaller than or equal to the preset threshold when the second confidence coefficient of any template is smaller than or equal to the preset threshold.

With reference to the second aspect to the eighth possible implementation manner of the second aspect, in a ninth possible implementation manner of the second aspect, the candidate momentsThe array further includes a third indicator, configured to indicate that a distance between a target at a location of the third indicator and a template corresponding to the location of the third indicator is smallest in a column where the third indicator is located, and the apparatus provided in the embodiment of the present invention further includes: the establishing unit is used for establishing a first matrix with M rows and N columns according to the distance between each target and the template associated with each target, and any element M in the first matrix_ijRepresenting the euclidean distance between a template identified as i and a target identified as j, where i 1.., M, j, N, M is the number of targets and N is the number of templates; the determining unit is further used for determining the position of the minimum value of each row and the position of the minimum value of each column in the first matrix; the apparatus further comprises: the assignment unit is used for assigning a position corresponding to the position of the minimum value of each row in the matrix to be selected to a first indicator and assigning a position corresponding to the position of the minimum value of each column in the matrix to be selected to a third indicator according to the position of the minimum value of each row and the position of the minimum value of each column; the obtaining unit is further specifically configured to obtain a target position according to a position of each row of minimum values and a position of each column of minimum values in the first matrix, where the target position is a position where the position of each row of minimum values and the position of each column of minimum values are the same, and to assign a second indicator to a position in the candidate matrix corresponding to the target position, so as to obtain a candidate matrix, where the candidate matrix is an M-row × N-column matrix, and each element in the candidate matrix is an initial value.

In a third aspect, an embodiment of the present invention provides a multi-target tracking processing apparatus, including: the multi-target tracking processing device comprises a memory, a processor, a bus and a transceiver, wherein the memory stores codes and data, the processor is connected with the memory through the bus, and the processor runs the codes in the memory to enable the device to execute the multi-target tracking processing method described in any one of the possible implementation modes described in the first aspect to the ninth aspect.

In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, which includes instructions that, when executed on a multi-target tracking processing device, cause the multi-target tracking processing device to perform the multi-target tracking processing method described in any one of the possible implementation manners described in the first aspect to the tenth aspect.

In a fifth aspect, an embodiment of the present invention provides a computer program product, where the computer program product includes computer executable instructions, and the computer executable instructions are stored in a computer readable storage medium; the at least one processor of the multi-target tracking processing device may read the computer executable instructions from the computer readable storage medium, and the execution of the computer executable instructions by the at least one processor causes the multi-target tracking processing device to implement the multi-target tracking processing method described in any one of the possible implementations described in the first aspect to the tenth aspect.

It is understood that any one of the multi-target tracking processing method, the multi-target tracking processing device, the computer storage medium or the computer program product provided above is used for executing the corresponding method provided above, and therefore, the beneficial effects achieved by the method can refer to the beneficial effects in the corresponding method provided above, and are not described herein again.

Drawings

FIG. 1 is a schematic view of a scene of object fusion and object splitting;

fig. 2 is a schematic diagram illustrating a server according to an embodiment of the present invention;

fig. 3 is a first schematic flow chart of a multi-target tracking processing method according to an embodiment of the present invention;

fig. 4 is a schematic flow chart of a multi-target tracking processing method according to an embodiment of the present invention;

fig. 5 is a schematic flow chart of a multi-target tracking processing method according to an embodiment of the present invention;

FIG. 6 is a schematic diagram illustrating another multi-target tracking processing apparatus according to an embodiment of the present invention;

fig. 7 is a schematic composition diagram of another multi-target tracking processing device according to an embodiment of the present invention.

Detailed Description

In a video monitoring scene, a target fusion scene and a target splitting scene as shown in fig. 1 usually occur, and a technical scheme adopted in a conventional technical scheme cannot solve the problem of tracking failure or discontinuous tracking track caused by track fracture generated after target fusion. And judging a plurality of targets with target fusion, and tracking the fused targets by using the newly added target template in the time period of the target fusion, so that the problem of discontinuous tracking tracks generated after the target fusion can be avoided, and the problem of tracking failure caused by the target fusion can be reduced. In addition, the method provided by the application only relates to matrix operation with small dimensionality, and is lower in calculation complexity compared with a method adopting mean shift in the traditional technology, suitable for being implemented on a chip or an embedded platform and lower in power consumption.

In practical use, the multi-target tracking processing device may be a server, and thus as shown in fig. 2, fig. 2 is a simplified schematic diagram illustrating a case where the multi-target tracking processing method provided by the embodiment of the present invention may be applied to a server. As shown in fig. 2, the server may include: a processor 101, a transceiver 102, a memory 104, and a bus 103. Wherein, the transceiver 102, the processor 101 and the memory 104 are connected to each other through a bus 103; the bus 103 may be a PCI bus or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown in FIG. 2, but it is not intended that there be only one bus or one type of bus. Wherein the memory 104 is used for storing program codes and data of the server. The communication interface 102 is used to support the server to communicate with other devices, and the processor 101 is used to support the server to execute the program codes and data stored in the memory 104 to implement a multi-target tracking processing method provided by the embodiment of the invention.

As shown in fig. 3, fig. 3 shows a multi-target tracking processing method provided by the present application, which includes:

s101, acquiring a candidate matrix by the multi-target tracking processing equipment according to the distance between each target in the multiple targets and the template associated with each target, wherein the candidate matrix comprises multiple first indicators and multiple second indicators, one first indicator indicates that the distance between the template corresponding to the position of the first indicator and the target corresponding to the position of the first indicator is the minimum in the row where the first indicator is located, and one second indicator indicates that the target corresponding to the position of the second indicator is matched with the template corresponding to the position of the second indicator; the template is used to record historical information of the objects associated with the template.

In this embodiment of the present invention, the step of one second indicator indicating that the target corresponding to the location of the second indicator matches the template corresponding to the location of the second indicator is: the Euclidean distance between the target corresponding to the position of the second indicator and the template corresponding to the position of the second indicator is the minimum, and the Euclidean distance between the template corresponding to the position of the second indicator and the target corresponding to the position of the second indicator is the minimum.

Optionally, the history information of the target in the embodiment of the present invention refers to information such as deformation (for example, increase or decrease or other changes) or position changes of the target in a time sequence.

For example, four objects a, B, C, and D in the scene, which respectively generate a1, a2, A3 … in each frame image in time series; b1, B2, B3 …; c1, C2, C3 …; d1, D2, D3 …, and the like. Because each target may be deformed, enlarged or reduced or otherwise changed in time series, in order to stably track each target, a template is constructed for the same target on different time series.

S102, the multi-target tracking processing equipment determines whether at least two targets are fused in the multiple targets according to the positions of the first indicator and the second indicator in the candidate matrix.

S103, under the condition that the multi-target tracking processing equipment determines that at least two targets are fused in the plurality of targets, a target template associated with the fused targets is generated, the fused targets are obtained by fusing the at least two targets, and at least history information of template records associated with each target in the at least two targets is recorded in the target template.

When the target template is generated, cache information and history information of the fusion target are initially recorded in the target template, and the cache information and the history information are image blocks corresponding to the fusion target. Therefore, the target template finally associated with the fusion target not only records the historical information of each target before fusion of the fusion target, but also records the cache information and the historical information of the fusion target at the current moment, so that the track of the fusion target can be ensured to be continuous.

Optionally, in the embodiment of the present invention, each template records information of a previous Q frame of a target associated with each template in a current frame, and uses the previous Q frame information of the current frame as cache information. In the embodiment of the present invention, Q is an integer greater than or equal to 1, and for example, Q in the present application is 15.

And S104, tracking the fusion target by the multi-target tracking processing equipment by using the target template.

Optionally, the step S101 executed by the multi-target tracking processing device in the present application may be specifically implemented by the following steps:

s1011, the multi-target tracking processing equipment establishes a first matrix with M rows and N columns according to the distance between each target and the template related to each target, and any element M in the first matrix_ijDenotes the euclidean distance between a template identified as i and an object identified as j, where i 1.

It is understood that each object in the present application has an identifier, the identifiers of different objects are different, the identifier of an object is used for uniquely identifying the object associated with the identifier, and may be the name, number, etc. of the object.

Each template has an identification, the identifications of the different templates being different, the identifications of the templates being used to identify the object associated with the template.

And S1012, the multi-target tracking processing equipment determines the position of the minimum value of each row and the position of the minimum value of each column in the first matrix.

And S1013, the multi-target tracking processing device endows the position corresponding to the position of the minimum value of each row in the matrix to be selected with a first indicator and endows the position corresponding to the position of the minimum value of each column in the matrix to be selected with a third indicator according to the position of the minimum value of each row and the position of the minimum value of each column.

And S1014, the multi-target tracking processing equipment acquires a target position according to the position of the minimum value of each row and the position of the minimum value of each column in the first matrix, wherein the target position is the position where the position of the minimum value of each row is the same as the position of the minimum value of each column.

And S1015, the multi-target tracking processing device assigns a second indicator to a position corresponding to the target position in the candidate matrix to obtain a candidate matrix, wherein the candidate matrix is a matrix with M rows and N columns, and each element in the candidate matrix is an initial value.

Optionally, the initial value of each element in the candidate matrix of the present application is the same, and the present application does not limit the specific value of the initial value, and may be set as needed in an actual process, and in order to reduce the complexity of subsequent calculation, for example, the present application may set the initial value to 0.

It is understood that the candidate matrix in step S101 includes not only the first indicator, the second indicator, but also the third indicator and the initial value.

It will be appreciated that the initial values of the elements in the candidate matrix at the positions corresponding to the positions of the minimum values in each row of the first matrix will be replaced by the first indicator, and the initial values of the elements in the candidate matrix at the positions corresponding to the positions of the minimum values in each column of the first matrix will be replaced by the third indicator. Of course, when the initial value is not 0, the initial value of the element at the position corresponding to the position of the minimum value in each row in the first matrix in the candidate matrix may be superimposed with the first indicator, and the superimposed result may be used as the element at the position corresponding to the position of the minimum value in each column in the first matrix in the candidate matrix, and the superimposed result of the initial value of the element at the position corresponding to the position of the minimum value in each column in the first matrix in the candidate matrix and the third indicator may be used as the element at the position corresponding to the position of the minimum value in each column in the first matrix in the candidate matrix, which is not limited in this application.

In addition, if the position of each row minimum value and each column minimum value in the first matrix is the same, for example, the target position is given a second indicator in the candidate matrix.

Optionally, the first indicator, the second indicator, and the third indicator are not limited in this application, and may be letters, numbers, or a combination of letters and numbers. Illustratively, the first indicator is "1", the second indicator is "11", and the third indicator is "10", so that it can be determined whether the distance between the object and the template is the minimum or the distance between the template and the object is the minimum in the candidate matrix according to the first indicator, the second indicator, and the third indicator. Therefore, the elements in the final candidate matrix will contain 11, 10, 1 and 0 at most, and other numbers may not exist in the matching matrix.

Specifically, "11" indicates that the target corresponding to the position of "11" and the template are matched, that is, the euclidean distance between the target corresponding to the position of "11" and the template corresponding to the position of "11" is the smallest in the row or column where "11" is located, and the euclidean distance between the template at the position of "11" and the target corresponding to the position of "11" is the smallest in the row or column where "11" is located. "10" means that the distance between the object corresponding to the position of "10" and the template corresponding to the position of "10" is the smallest in the column of "10". "1" indicates that the distance between the template corresponding to the position of "1" and the target corresponding to the position of "1" is the smallest in the row of "1".

Illustratively, a plurality of objects are an object a, an object B, an object C and an object D, and a plurality of templates are a template 1, a template 2, a template 3 and a template 4, for example, where the template 1 is used to record change information of the object a in time series, that is, the object a is associated with the template 1, the template 2 is used to record change information of the object B in time series, the template 3 is used to record change information of the object C in time series, and the template 4 is used to record change information of the object D in time series.

Establishing a first matrix by taking the number of the templates as the row number of the first matrix and the number of the targets as the column number of the first matrix, wherein the first matrix can be expressed as:

wherein 1A, 2A, 3A and 4A represent euclidean distances between the target a and the template 1, template 2, template 3 and template 4, respectively; 1B, 2B, 3B, and 4B represent euclidean distances between the object B and the template 1, template 2, template 3, and template 4, respectively; 1C, 2C, 3C, and 4C represent euclidean distances between the object C and the template 1, the template 2, the template 3, and the template 4, respectively; 1D, 2D, 3D, and 4D represent euclidean distances between the object D and the template 1, the template 2, the template 3, and the template 4, respectively.

And establishing a candidate matrix according to the minimum value of each row and the minimum value of each column in the first matrix and the matrix to be selected.

Exemplary, the candidate matrix is:

assigning a position corresponding to the minimum value of each row in the first matrix in the matrix to be selected to a first indicator, and assigning the position corresponding to the minimum value of each row in the first matrix in the matrix to be selected to a first indicatorAnd if the minimum value position of each row and the minimum value position of each column in the first matrix are the same position, such as a target position, the position associated with the target position in the candidate matrix is assigned with a second indicator so as to obtain a candidate matrix.

It is understood that the position associated with the position of the first matrix in the candidate matrix means that a certain position is the ith row and the jth column in the first matrix, and then the corresponding position in the candidate matrix is also the ith row and the jth column in the candidate matrix.

For example, the minimum values of each row in the first matrix are: 1A (the corresponding position in the first matrix is 1 st row and 1 st column), 2B (the corresponding position in the first matrix is 2 nd row and 2 nd column), 3C (the corresponding position in the first matrix is 3 rd row and 3 rd column), and 4C (the corresponding position in the first matrix is 4 th row and 3 rd column), the minimum value of each column in the first matrix is 2A (the corresponding position in the first matrix is 1 st column and 2 nd row), 2B (the corresponding position in the first matrix is 2 nd column and 2 nd row), 3C (the corresponding position in the first matrix is 3 rd column and 3 rd row), and 4D (the corresponding position in the first matrix is 4 th row and 4 th column), respectively, and the first indicator is: 1 and the third indicator is 10.

Therefore, the multi-target tracking processing device needs to assign a first indicator to the 1 st row and the 1 st column of the candidate matrix, assign the first indicator to the 2 nd row and the 2 nd column of the candidate matrix, assign the first indicator to the 3 rd row and the 3 rd column of the candidate matrix, assign the first indicator to the 3 rd row and the 4 th column of the candidate matrix, assign the third indicator to the 2 nd row of the 1 st column of the candidate matrix, assign the third indicator to the 2 nd row of the 2 nd column of the candidate matrix, assign the third indicator to the 3 rd row of the 3 rd column of the candidate matrix, and assign the third indicator to the 4 th row of the 4 th column of the candidate matrix. Since the minimum value of the 2 nd row and the minimum value of the 2 nd column in the first matrix are at the same position in the first matrix, and the minimum value of the 3 rd row and the minimum value of the 3 rd column in the first matrix are at the same position in the first matrix, the sum of the first indicator and the third indicator is given to the position of the 2 nd row and the 2 nd column in the matrix to be selected, and the sum of the first indicator and the third indicator, that is, 11, is given to the position of the 3 rd row and the 3 rd column in the matrix to be selected. Therefore, if the position of the minimum value in each row and the position of the minimum value in each column are both target positions, a second indicator is given to a position corresponding to the target position in the candidate matrix, and therefore, the candidate matrix can be expressed as:

wherein 11 indicates that the object and the template are matched, that is, the euclidean distance between the object and the template is the smallest, and the euclidean distance between the template and the object is the smallest, 10 indicates that the euclidean distance of the template corresponding to the position of the object distance 10 corresponding to the position of 10 is the smallest in the column of 10, and 1 indicates that the euclidean distance of the object corresponding to the position of 1 and the template distance corresponding to the position of 1 is the smallest in the row of 1.

It should be noted that, in the embodiment of the present invention, how to construct the first matrix and the candidate matrix is described by taking the target as a column and taking the template as a row, in an actual process, the first matrix and the candidate matrix may also be constructed by taking the target as a row and taking the template as a column, which is not limited in the present application.

It is to be understood that the second indicator of the embodiment of the present invention may also be other values and letters, and is not limited to the sum of the first indicator and the third indicator, and the embodiment of the present invention is only described by taking the sum of the first indicator and the third indicator as the second indicator.

Optionally, step S102 in this application may be specifically implemented by the following method: determining that there is at least one first column in the candidate matrix, and if there is at least one first target indicator and at least one second target indicator in the first column, determining that a target associated with a template corresponding to a location of each first target indicator in the at least one first target indicator is fused with a target associated with a template corresponding to a location of the second target indicator, where the first target indicator is any one first indicator in the first column, and the second target indicator is any one second indicator in the first column.

Specifically, the multi-target tracking processing device counts the number of the first indicator and the second indicator in each column of the candidate matrix, and if the number of the first indicator and the number of the second indicator are not 0 in any column of the candidate matrix, it indicates that target fusion occurs.

Exemplarily, as can be seen from the candidate matrix MTH, the element "1" in row 1, column 1 indicates that the template identified as 1 (i.e., template 1) has the smallest euclidean distance from the target identified as 1 (i.e., target a), the element "10" in row 2, column 1 indicates that the euclidean distance between the target identified as 1 (i.e., template a) and the template identified as 2 (i.e., template 2) is the smallest, the element "11" in row 2, column 2 indicates that the euclidean distance between the target identified as 2 (i.e., target B) and the template identified as 2 (i.e., template 2) is the smallest (i.e., euclidean distance between target B and template 2 is the smallest among the four templates, and template 2 has the smallest euclidean distance between target B and target B among the four targets), and similarly, the element "11" in column 3 of row 3 indicates that the euclidean distances between target C and template 3 are the smallest, and since there is also the element 1 in column 3, row 4 of the candidate matrix, therefore, it can be found that the euclidean distance between the template 4 and the target C is the smallest, and therefore, it can be seen that the euclidean distances between the template 3 and the template 4 and the target C are the smallest, which indicates that target fusion occurs, that is, the target C associated with the template 3 and the target D associated with the template 4 are fused, so that fusion processing needs to be performed on the fusion target F generated by the target C and the target D, and specifically, the target template associated with the fusion target F can be added.

For example, if there are both "11" and "0" in any column, it indicates that object fusion occurs, and the object where the object fusion occurs is an object associated with the template corresponding to the position of "11" in the candidate matrix and an object associated with the template corresponding to the position of "0".

It is understood that each template in the present application has an Identifier (ID) for uniquely identifying one template, of course, the ID of one template may also be the number of the template, and it is understood that the multi-target tracking processing device in the present application assigns a number to each template in a preset order.

Each template also has a display mark, the display mark of one template is used for indicating that the template can be used for normally tracking the target or cannot be used for tracking the target, and the display mark is usually a third mark or a fourth mark, the third mark is used for indicating that the template can be used for normally tracking the target, and the fourth mark is used for indicating that the template cannot be used for tracking the target.

The third mark and the fourth mark may be letters, numbers or a combination of letters and numbers, and the present application is not limited thereto, for example, the third mark may be "TRUE" and the fourth mark may be "FALSE".

Optionally, with reference to fig. 3, as shown in fig. 4, after step S103 in the present application, the following implementation may be specifically included:

s311, acquiring a fusion mark of a template associated with each of at least two targets, wherein the fusion mark of one template is used for indicating whether the targets associated with the template are fused or not.

S312, judging whether the fusion mark of the template associated with each target is used for indicating the fusion of the respective associated targets.

And S313, under the condition that the fusion mark of each target-associated template indicates that the targets associated with each template are fused, adding the historical information of the targets recorded by each target-associated template into the fusion template.

Optionally, the fusion flag in this application is a first flag or a second flag, where the first flag is used to indicate that the targets associated with the template are not fused, and the second flag is used to indicate that the targets associated with the template are fused.

The content of the first mark and the second mark in this application may be a combination of one or more of letters, numbers and symbols, which is not limited in this application. Illustratively, the first label in this application is "FALSE" and the second label is "TRUE".

Specifically, whether the fusion mark of the template associated with each of the at least two targets is TRUE is judged, and in the case that the fusion mark of the template associated with each target is determined to be TRUE, the history information of the target recorded by the template associated with each target, the fusion mark of which is the second mark, is added to the target template.

It should be noted that, the fusion flag of the template associated with each target is TRUE, which indicates that the target associated with the target has been fused, and therefore, in order to track the fused target, the history information of the target associated with the template recorded by the template needs to be migrated to the newly added target template.

After step S313, the embodiment of the present invention further includes:

and S314, after determining that the historical information of the targets recorded by the templates associated with each target marked as the second mark is added into the target templates, the multi-target tracking processing device releases the historical information of the targets recorded by the templates associated with each target marked as the second mark and deletes the templates associated with each target marked as the second mark.

Furthermore, as shown in fig. 4, after step S312 in the present application, the following steps may be specifically implemented:

and S315, under the condition that the fusion mark of the template associated with each target indicates that the targets associated with each template are not fused, the multi-target tracking processing equipment updates the history information of each template by using the cache information of the template record associated with each target which is not fused.

And S316, setting the updated display mark of the template associated with each target without fusion as a fourth mark by the multi-target tracking processing equipment.

It can be understood that the fusion mark of one template is the second mark, which indicates that the target associated with the template has been fused, that is, the template is the fusion template, and therefore, the fusion template is no longer used to track the target in the process of tracking the fusion target, and therefore, it is usually necessary to set the display mark of the fusion template as the fourth mark. The display label of a template is the fourth label indicating that the template will not be used in the subsequent tracking process.

Optionally, in order to perform the subsequent processing on the fusion target split condition, as shown in fig. 4, after step S103, the method further includes:

and S105, the multi-target tracking processing equipment sets the fusion mark of the newly added target template as a second mark.

Specifically, the multi-target tracking processing device sets a fusion flag of the newly added target template to TRUE, which indicates that the target associated with the template is fused.

It can be understood that, when the target template is generated, the multi-target tracking processing device allocates one ID to the newly generated target template, and since the history information recorded in the templates with the fusion marks of TRUE is recorded in the newly generated target template, the multi-target tracking processing device assigns the ID of each template with the fusion marks of TRUE to the ID of the newly generated target template, so that the number of fusion IDs of the templates is added to the original ID of the newly generated target template. That is, the ID of the newly generated target template may be larger than the previous ID number by, for example, 1. If the label is the maximum value of the current template ID, the sum of the number of templates before fusion (namely the number of fusion IDs) is used as the ID number of the newly generated target template.

Optionally, in the tracking process, the track of the target is continuously changed, and therefore each template needs to be updated, and the method provided by the present application further includes:

s106, the multi-target tracking processing equipment determines that at least one second column exists in the candidate matrix, only a second indicator exists in the second column, the template corresponding to the position of the second indicator in the second column is updated by using the target corresponding to the position of the second indicator in the second column, the updated template is obtained, and the second column is any column in the candidate matrix.

For example, if there is only an element "11" in the second column of the candidate matrix MTH described in the above embodiment, it indicates that the target and the template corresponding to the second row and the second column are one-to-one matched, and in this embodiment of the present invention, the one-to-one matching refers to: if the second indicator exists in any column of the candidate matrix, if the elements of the other positions of the column except the position of the second indicator are all 0, the target corresponding to the position of the second indicator and the template are matched one to one.

Specifically, step S106 can be implemented by: and the multi-target tracking processing equipment adds the coordinates of the center point of the target corresponding to the position of the second indicator in the second column to the track of the template corresponding to the position of the second indicator in the second column. For example, since the target corresponding to the position of "11" in the second column of the candidate matrix MTH is the target B and the template is the template 2, the trajectory of the template 2 may be updated by the coordinates of the center point of the target B.

S107, the multi-target tracking processing device determines that at least one first column exists in the candidate matrix, at least one first indicator and a second indicator exist in the first column, the template corresponding to the position of the second indicator in the first column is updated by using the target corresponding to the position of the second indicator in the first column, and the template corresponding to the position of each first indicator in the at least one first indicator is updated.

Specifically, step S107 can be implemented by: and the multi-target tracking processing equipment adds the center point coordinates of the target corresponding to the position of the second indicator in the first column to the track of the template corresponding to the position of each first indicator and to the track of the template corresponding to the position of the second indicator.

For example, not only "11" but also "1" exists in the third column of the candidate matrix MTH as described in the above embodiment, and therefore, it may be determined that the target associated with the template corresponding to the position of "11" in the third column (i.e., the target C associated with the template 3) is fused with the target associated with the template corresponding to the position of "1" in the third column (i.e., the target D associated with the template 4), and therefore, the coordinates of the center point of the fused target generated by fusing the target C and the target D need to be added to the trajectories of the template 3 and the template 4, respectively.

The track of the template is used for recording the historical track of the target associated with the template, and is mainly used for drawing the track of the target.

It is understood that after steps S106 and S107 are performed, the number of track points of each template added with the coordinates of the center point is increased by 1.

Optionally, after step S106 and step S107, the present application further includes:

and S108, the multi-target tracking processing device updates the first confidence degree of the updated template, wherein the first confidence degree of one template is used for representing the confidence degree of tracking the target corresponding to the template by using the template.

It is understood that each template in the present application has a first confidence level before being updated, and the first confidence level is set by system initialization, for example, the first confidence level of each template before being updated is a preset value, which is not limited in the present application and can be set as needed, and for example, the preset value in the present application is 18.

It should be noted that, in the present application, the first confidence of the target template added for fusing the target is an initial value, and the initial value is 1.

The confidence level in the embodiment of the invention is used for maintaining the activity degree of the template, and if one template is inactive for a long time, the template can be considered not to be used, such as a target walking out of a scene. Thus, the first confidence of a template may be used to represent the confidence of the template in the object to which the template corresponds.

Specifically, step S108 in the present application may be specifically implemented by: and the multi-target tracking processing equipment increases the first confidence coefficient of the updated template by a preset step length to obtain the first confidence coefficient of the updated template.

Illustratively, if the preset step size is 1, the first confidence of the updated template becomes 19.

Since the fused objects are separated after a period of time elapses after the fusion of the multiple objects occurs in the tracking process, after step S104, as shown in fig. 5, the present application further includes:

and S109, determining one or more fusion templates from the plurality of templates by the multi-target tracking processing equipment according to the fusion mark of each template in the plurality of templates, wherein the fusion template refers to the template with the fusion mark as the second mark.

S110, the multi-target tracking processing device determines at least one candidate target located in the tracking area of each of the one or more fusion templates.

The multi-target tracking processing device performs the following steps on at least one candidate target in each fusion template tracking area to determine a split target in each fusion template tracking area:

s111, if the multi-target tracking processing equipment determines that at least one candidate target in a fusion template tracking area has a target which has no template and is intersected with the fusion template tracking area, determining that the fusion target is split.

Specifically, in step S111, whether the target intersects with the tracking area of the fusion template may be determined by, for example, intersecting the target with a bounding rectangle of the fusion template, for example, assuming that the target X circumscribes the upper left corner coordinate (X) of the rectangle O of the target X_o1,Y_o1) Lower right corner coordinate (X)_o2,Y_o2) The coordinates (X) of the upper left corner of the circumscribed rectangle T of the fused template_t1,Y_t1) Lower right corner coordinate (X)_t2,Y_t2) Coordinates of the upper left corner of the intersection region R of the target X bounding rectangle O and the fusion template bounding rectangle T (X)_r1,Y_r1) Lower right corner coordinate (X)_r2,Y_r2). Then there is X_r1＝max(X_o1,X_t1),Y_r1＝max(Y_o1,Y_t1),X_r2＝min(X_o2,X_t2),Y_r2＝min(Y_o2,Y_t2) And, therefore,

it should be noted that, if there is an existing template of the target X associated with at least one candidate target in the tracking area of the fused template, it indicates that the target X is normally tracked and does not belong to the target split from the fused target.

And S112, acquiring image characteristic information of each split target in the split targets from the image of the fusion target by the multi-target tracking processing equipment.

Specifically, the image of the fusion target can refer to a color image or a gray image of the fusion template, and in order to reduce the computational complexity of the multi-target tracking processing equipment, each split target is cut out from the gray image of the fusion target according to a preset size proportion, and image characteristic information of each split target is obtained.

In the embodiment of the present invention, a manner of acquiring image feature information of each split target may be a manner in the prior art, and details of the method are not described herein again.

S113, determining, by the multi-target tracking processing device, a template, the similarity of which with the image feature information of each split target meets preset requirements, as a template corresponding to each split target according to the image feature information of each split target and history information of each associated target recorded by each template before fusion.

Specifically, in order to reduce the computational complexity of the multi-target tracking processing device, the embodiment of the invention measures the similarity between the history information of the respectively associated target recorded by each template and each split gray image by using a histogram comparison method.

Illustratively, the histogram comparison method includes any one of a correlation coefficient, a chi-square, a histogram intersection and a babbit distance, and the embodiment of the invention may adopt the correlation coefficient method according to a formula

To calculate the similarity between the history information of the respectively associated targets of each template record before fusion and the image feature information of each split target. Wherein x is_i，y_iThe values of the ith Bin (Bin) in the histogram of the template and the histogram of the split target are respectively represented. n represents the number of bins.

In the embodiment of the invention, the gray level histogram divides the gray level 0-255 into n equal parts, each equal part is a Bin, and n is the number of bins.

And S114, updating the template corresponding to each split target by the multi-target tracking processing equipment according to the current moment information of each split target.

Specifically, step S114 may be implemented in the following manner: and updating the template corresponding to each split target according to the information of the current moment of each split target, the weight corresponding to the information of the current moment of each split target, the historical information recorded by the template corresponding to each split target before and the weight of the historical information recorded by the template.

In addition, after updating the template corresponding to each split target, the multi-target tracking processing device sets the display flag of the updated template corresponding to each split target to TRUE, and deletes the template corresponding to the fusion target, for example, the target template.

The previous targets are fused, the fused targets correspond to a fused template, after the fused targets are split, a template before the split targets cannot be matched possibly exists, at this time, the multi-target tracking processing device also records the state after the split targets, and if the template cannot be found after a certain image frame, for example, 15 frames, the multi-target tracking processing device constructs the template according to the recorded 15 frames of information.

Optionally, after step S114, the method provided by the present application further includes: and S115, the multi-target tracking processing equipment tracks each split target by using the updated template corresponding to each split target.

Illustratively, the split target a corresponds to the template 1, and the split target 2 corresponds to the template 2, then the split target a is tracked by using the template 1, and the split target 2 is tracked by using the template 2.

In order to improve the processing efficiency of the multi-target tracking processing device and avoid storing excessive redundant information in the multi-target tracking processing device, each template in the application further includes a second confidence level, and the method provided by the embodiment of the invention further includes:

s116, the multi-target tracking processing device determines that at least one unprocessed template exists in the plurality of templates, and then the second confidence of the at least one unprocessed template is reduced by a preset step length, so that the updated second confidence of each unprocessed template in the at least one unprocessed template is obtained.

In the multi-target tracking processing method provided by the embodiment of the invention, a template needs to be searched for each target, and when one target does not correspond to the previous template at the current time, the previous template cannot be used for tracking the target, so that the previous template can be regarded as unprocessed after the current time, namely the template is an unprocessed template. Therefore, the multi-target tracking processing method maintains the template which is not corresponding to the target before the current moment for a period of time, if the template is not processed for a longer time, the template is considered invalid, and the multi-target tracking processing equipment can delete the template, so that the memory space of the multi-target tracking processing equipment is saved.

It can be understood that, in the initialization process of the multi-target tracking processing device, the second confidence of each template is initialized to a threshold, and in the embodiment of the present invention, the threshold may be 10.

Optionally, after step S116, the present application further includes: and S117, when the multi-target tracking processing device determines that the second confidence coefficient of any template is smaller than or equal to the preset threshold, deleting the template with the second confidence coefficient smaller than or equal to the preset threshold. By deleting the template with the second confidence coefficient smaller than or equal to the preset threshold value, the storage of excessive redundant information in the multi-target tracking processing equipment can be avoided.

The preset threshold in the embodiment of the present invention may be set as needed, and it may be understood that the preset threshold is smaller than the second confidence, for example, the preset threshold in the present application may be set as 0. That is, when determining that the second confidence of one template becomes 0, the multi-target tracking processing device deletes the template with the second confidence of 0.

The above-mentioned scheme provided by the embodiment of the present invention is introduced mainly from the perspective of interaction between network elements. It is understood that each network element, for example, the multi-target tracking processing device, includes a hardware structure and/or a software module for performing each function in order to realize the functions. Those of skill in the art will readily appreciate that the present invention can be implemented in hardware or a combination of hardware and computer software, in conjunction with the exemplary algorithm steps described in connection with the embodiments disclosed herein. Whether a function is performed as hardware or computer software drives hardware depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.

According to the embodiment of the invention, the multi-target tracking processing equipment can be divided into the functional modules according to the method example, for example, each functional module can be divided corresponding to each function, or two or more functions can be integrated into one processing module. The integrated module can be realized in a hardware mode, and can also be realized in a software functional module mode. It should be noted that, the division of the modules in the embodiment of the present invention is schematic, and is only a logic function division, and there may be another division manner in actual implementation.

In the case of dividing the function modules according to the respective functions, fig. 6 shows a schematic diagram of a possible composition of the multi-target tracking processing device according to the foregoing embodiment, and as shown in fig. 6, the server may include: an acquisition unit 301, a determination unit 302, a generation unit 303, and a tracking unit 304. The acquisition unit 301 is configured to support the multi-target tracking processing device to execute steps S101, S1014, S1015, S311, S112, and S116 in the foregoing embodiments; the determination unit 302 is configured to support the multi-target tracking processing device to execute steps S102, S1012, S312, and S316, S105, S109, S110, S111, S113 in the above-described embodiment, the generation unit 303 is configured to support the multi-target tracking processing device to execute steps S103 and S313 in the above-described embodiment, and the tracking unit 304 is configured to support the multi-target tracking processing device to execute steps S104 and S115 in the above-described embodiment. In addition, in order to implement the functions provided by the above method, the multi-target tracking processing device in the present application further includes: an update unit 305, a delete unit 306, a build unit 307, and an assign unit 308. The updating unit 305 is configured to support the multi-target tracking processing device to execute steps S315, S106, S107, S108, and S114 in the foregoing embodiment, the deleting unit 306 is configured to support the multi-target tracking processing device to execute steps S314 and S117 in the foregoing embodiment, the establishing unit 307 is configured to support the multi-target tracking processing device to execute step S1011 in the foregoing embodiment, and the assigning unit 308 is configured to support the multi-target tracking processing device to execute step S1013 in the foregoing embodiment. It should be noted that all relevant contents of each step related to the above method embodiment may be referred to the functional description of the corresponding functional module, and are not described herein again.

The multi-target tracking processing device provided by the embodiment of the invention is used for executing the multi-target tracking processing method, so that the same effect as the multi-target tracking processing method can be achieved.

In the case of a hardware implementation, the acquiring unit 301, the determining unit 302, the generating unit 303, the tracking unit 304, the updating unit 305, the deleting unit 306, the establishing unit 307, and the assigning unit 308 in this application are all integrated on a processor of the apparatus shown in fig. 2.

In the case of an integrated unit, fig. 7 shows another possible composition diagram of the multi-target tracking processing apparatus involved in the above-described embodiment. As shown in fig. 7, the multi-target tracking processing apparatus includes: a processing module 311 and a communication module 312.

The processing module 311 is used to control and manage the actions of the multi-target tracking processing device, for example, the processing module 311 is used to support the multi-target tracking processing device to execute steps S101, S1014, S1015, S311, S112, S116, S102, S1012, S312, S316, S105, S109, S110, S111, S113, S103, S313, S104, S115, S315, S106, S107, S108, S114, S314, and S117, S1011, S1013 in the above-described embodiments. And/or other processes for the techniques described herein. It should be noted that, the specific execution sequence of the above steps may refer to the description in the above embodiments, and is not described herein again. The communication module 312 is used to support communication between the multi-target tracking processing device and other network entities. The multi-target tracking processing device may further include a storage module 313 for storing program codes and data of the multi-target tracking processing device.

The processing module 311 may be a processor or a controller, among others. Which may implement or perform the various illustrative logical blocks, modules, and circuits described in connection with the disclosure. A processor may also be a combination of computing functions, e.g., comprising one or more microprocessors, a DSP and a microprocessor, or the like. The communication module 312 may be a communication interface, etc. The storage module 313 may be a memory.

When the processing module 311 is a processor, the communication module 312 is a communication interface, and the storage module 313 is a memory, the multi-target tracking processing device according to the embodiment of the present invention may be a server as shown in fig. 2.

In one aspect, an embodiment of the present invention provides a computer-readable storage medium including instructions that, when executed on a multi-target tracking processing device, cause the multi-target tracking processing device to perform steps S101, S1014, S1015, S311, S112, S116, S102, S1012, S312, S316, S105, S109, S110, S111, S113, S103, S313, S104, S115, S315, S106, S107, S108, and S114, S314, and S117, S1011, S1013 in the above-described embodiments. And/or other processes for the techniques described herein. It should be noted that, the specific execution sequence of the above steps may refer to the description in the above embodiments, and is not described herein again.

In another aspect, an embodiment of the present invention provides a computer program product, which includes computer executable instructions stored in a computer readable storage medium; the at least one processor of the multi-target tracking processing device may read the computer-executable instructions from the computer-readable storage medium, and the at least one processor executes the computer-executable instructions to cause the multi-target tracking processing device to implement steps S101, S1014, S1015, S311, S112, S116, S102, S1012, S312, S316, S105, S109, S110, S111, S113, S103, S313, S104, S115, S315, S106, S107, S108, and S114, S314 and S117, S1011, S1013 in the above-described embodiments. And/or other processes for the techniques described herein. It should be noted that, the specific execution sequence of the above steps may refer to the description in the above embodiments, and is not described herein again.

Through the above description of the embodiments, it is clear to those skilled in the art that, for convenience and simplicity of description, the foregoing division of the functional modules is merely used as an example, and in practical applications, the above function distribution may be completed by different functional modules according to needs, that is, the internal structure of the device may be divided into different functional modules to complete all or part of the above described functions.

In the several embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other ways. For example, the above-described device embodiments are merely illustrative, and for example, the division of the modules or units is only one logical functional division, and there may be other divisions when actually implemented, for example, a plurality of units or components may be combined or may be integrated into another device, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.

The units described as separate parts may or may not be physically separate, and parts displayed as units may be one physical unit or a plurality of physical units, that is, may be located in one place, or may be distributed in a plurality of different places. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.

The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a readable storage medium. Based on such understanding, the technical solution of the embodiments of the present invention may be essentially or partially contributed to by the prior art, or all or part of the technical solution may be embodied in the form of a software product, where the software product is stored in a storage medium and includes several instructions to enable a device (which may be a single chip, a chip, or the like) or a processor (processor) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.

The above description is only an embodiment of the present invention, but the scope of the present invention is not limited thereto, and any changes or substitutions within the technical scope of the present invention are intended to be covered by the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the appended claims.

Claims

1. A multi-target tracking processing method is characterized by comprising the following steps:

obtaining a candidate matrix according to the distance between each target in a plurality of targets and a template associated with each target, wherein the candidate matrix comprises a plurality of first indicators and a plurality of second indicators, one first indicator indicates that the distance between the template corresponding to the position of the first indicator and the target corresponding to the position of the first indicator is the minimum in the row where the first indicator is located, and one second indicator indicates that the target corresponding to the position of the second indicator is matched with the template corresponding to the position of the second indicator; the template is used for recording historical information of a target associated with the template;

determining whether at least two targets of the plurality of targets are fused according to the positions of the first indicator and the second indicator in the candidate matrix;

under the condition that fusion of at least two targets exists in the plurality of targets, generating a target template associated with the fused target, wherein the fused target is obtained by fusing the at least two targets, and history information of a template record associated with each target in the at least two targets is at least recorded in the target template;

tracking the fused target by using the target template;

the candidate matrix further includes a third indicator, configured to indicate that a distance between a target at a location of the third indicator and a template corresponding to the location of the third indicator is the smallest in a column where the third indicator is located, and the obtaining the candidate matrix according to a distance between each target of the multiple targets and a template associated with each target includes:

establishing a first matrix with M rows and N columns according to the distance between each target and the template associated with each target, wherein any element M in the first matrix_ijRepresenting the euclidean distance between a template identified as i and a target identified as j, where i 1.., M, j, N, M is the number of templates and N is the number of targets;

determining the position of the minimum value of each row and the position of the minimum value of each column in the first matrix;

according to the position of the minimum value of each row and the position of the minimum value of each column, assigning a position corresponding to the position of the minimum value of each row in a matrix to be selected as a first indicator, and assigning a position corresponding to the position of the minimum value of each column in the matrix to be selected as a third indicator;

acquiring a target position according to the position of each row of minimum values and the position of each column of minimum values in the first matrix, wherein the target position is the position where the position of each row of minimum values and the position of each column of minimum values are the same;

and assigning a second indicator to a position corresponding to the target position in the candidate matrix to obtain the candidate matrix, wherein the candidate matrix is a matrix with M rows and N columns, and each element in the candidate matrix is an initial value.

2. The method of claim 1, wherein determining whether at least two objects of the plurality of objects are merged based on the positions of the first indicator and the second indicator in the candidate matrix comprises:

determining that at least one first column exists in the candidate matrix, and if at least one first target indicator and at least one second target indicator exist in the first column, determining that a target associated with a template corresponding to a position of each first target indicator in the at least one first target indicator is fused with a target associated with a template corresponding to a position of the second target indicator, where the first target indicator is any one first indicator in the first column, and the second target indicator is any one second indicator in the first column.

3. The method of claim 1, wherein generating the object template associated with the fused object in the case that it is determined that at least two objects of the plurality of objects are fused comprises:

acquiring a fusion mark of a template associated with each of the at least two targets, wherein the fusion mark of one template is used for indicating whether the targets associated with the template are fused or not;

and in the case that the fusion mark of the template associated with each target indicates that the target associated with each template is fused, adding the historical information of the target recorded by the template associated with each target into the target template.

4. The method according to any one of claims 1-3, further comprising:

determining that at least one second column exists in the candidate matrix, wherein only the second indicator exists in the second column, updating the template corresponding to the position of the second indicator in the second column by using the target corresponding to the position of the second indicator in the second column, and acquiring an updated template, wherein the second column is any one column in the candidate matrix;

determining that at least one first column exists in the candidate matrix, wherein at least one first indicator and the second indicator exist in the first column, updating a template corresponding to the position of the second indicator in the first column by using a target corresponding to the position of the second indicator in the first column, and updating a template corresponding to the position of each first indicator in the at least one first indicator.

5. The method according to claim 4, wherein the determining that there is at least one second column in the candidate matrix, and only the second indicator exists in the second column, and using the target corresponding to the position of the second indicator in the second column to update the template corresponding to the position of the second indicator in the second column, obtaining an updated template includes:

adding the coordinates of the center point of the target corresponding to the position of the second indicator in the second column to the track of the template corresponding to the position of the second indicator in the second column;

the determining that at least one first column exists in the candidate matrix, at least one first indicator and the second indicator exist in the first column, and updating a template corresponding to a position of the second indicator in the first column and a template corresponding to a position of each first indicator in the at least one first indicator by using a target corresponding to a position of the second indicator in the first column includes:

and adding the coordinates of the center point of the target corresponding to the position of the second indicator in the first column to the track of the template corresponding to the position of each first indicator and to the track of the template corresponding to the position of the second indicator.

6. The method of claim 4, further comprising:

and updating the first confidence degrees of the updated templates, wherein the first confidence degree of one template is used for representing the confidence degree of tracking the target corresponding to the template by using the template.

7. The method of any of claims 1-3, wherein each template in the plurality of templates has a fusion token, the fusion token being a first token indicating that no fusion of template-associated objects has occurred or a second token indicating that fusion of template-associated objects has occurred, the method further comprising:

determining one or more fused templates from the plurality of templates according to the fused label of each template in the plurality of templates, wherein the fused template refers to the template with the fused label as the second label,

determining at least one candidate target located within each of the one or more fused templates tracking region;

performing the following steps for at least one candidate target within each fused template tracking area to determine split targets within each fused template tracking area:

if it is determined that at least one candidate target in a fusion template tracking area has a target which has no template and is intersected with the fusion template tracking area, determining that the fusion target is split;

acquiring image characteristic information of each split target in a plurality of split targets from the image of the fusion target;

determining a template with similarity meeting preset requirements with the image characteristic information of each split target as a template corresponding to each split target according to the image characteristic information of each split target and history information of each associated target recorded by each template before fusion;

and updating the template corresponding to each split target by using the information of the current moment of each split target.

8. The method of any of claims 1-3, wherein each of the plurality of templates further comprises a second confidence level, the method further comprising:

and if at least one unprocessed template exists in the plurality of templates, reducing the second confidence coefficient of the at least one unprocessed template by a preset step length, and acquiring the updated second confidence coefficient of each unprocessed template in the at least one unprocessed template.

9. The method of claim 8, further comprising:

and when the second confidence coefficient of any template is determined to be smaller than or equal to the preset threshold, deleting the template with the second confidence coefficient smaller than or equal to the preset threshold.

10. A multi-target tracking processing apparatus, comprising:

the device comprises an obtaining unit, a calculating unit and a processing unit, wherein the obtaining unit is used for obtaining a candidate matrix according to the distance between each target in a plurality of targets and a template associated with each target, the candidate matrix comprises a plurality of first indicators and a plurality of second indicators, one first indicator represents that the distance between the template corresponding to the position of the first indicator and the target corresponding to the position of the first indicator is minimum in the row where the first indicator is located, and one second indicator represents that the target corresponding to the position of the second indicator is matched with the template corresponding to the position of the second indicator; the template is used for recording historical information of a target associated with the template;

a determining unit, configured to determine whether at least two targets in the plurality of targets are fused according to positions of the first indicator and the second indicator in the candidate matrix;

the generating unit is used for generating a target template associated with a fusion target under the condition that at least two targets are determined to be fused in the plurality of targets, wherein the fusion target is obtained by fusing the at least two targets, and history information of a template record associated with each of the at least two targets is at least recorded in the target template;

a tracking unit for tracking the fusion target by using the target template;

the candidate matrix further comprises: a third indicator for indicating that a distance between a target at a location of the third indicator and a template corresponding to the location of the third indicator is smallest in a column of the third indicator, the apparatus further comprising:

an establishing unit, configured to establish a first matrix with M rows by N columns according to a distance between each target and a template associated with each target, where any element M in the first matrix is_ijRepresenting the euclidean distance between a template identified as i and a target identified as j, where i 1.., M, j, N, M is the number of templates and N is the number of targets;

the determining unit is further configured to determine a position of a minimum value in each row and a position of a minimum value in each column in the first matrix;

the apparatus further comprises: the assignment unit is used for assigning a position corresponding to the position of the minimum value of each row in a matrix to be selected to a first indicator and assigning a position corresponding to the position of the minimum value of each column in the matrix to be selected to a third indicator according to the position of the minimum value of each row and the position of the minimum value of each column;

the obtaining unit is further specifically configured to obtain a target position according to a position of each row of minimum values and a position of each column of minimum values in the first matrix, where the target position is a position where the position of each row of minimum values and the position of each column of minimum values are the same; and the candidate matrix is obtained by assigning a second indicator to a position corresponding to the target position in the candidate matrix, wherein the candidate matrix is a matrix with M rows and N columns, and each element in the candidate matrix is an initial value.

11. The apparatus according to claim 10, wherein the determining unit is specifically configured to determine that at least one first column exists in the candidate matrix, and if at least one first target indicator and at least one second target indicator exist in the first column, determine that a target associated with a template corresponding to a location where each first target indicator in the at least one first target indicator is located is merged with a target associated with a template corresponding to a location where the second target indicator is located, where the first target indicator is any one first indicator in the first column, and the second target indicator is any one second indicator in the first column.

12. The apparatus according to claim 10, wherein the obtaining unit is further configured to obtain a fusion flag of a template associated with each of the at least two objects, the fusion flag of one template being used to indicate whether fusion of the object associated with the template occurs;

the generating unit is specifically configured to, when the fusion flag of the template associated with each object indicates that the objects associated with each template are fused, add history information of the objects recorded in the template associated with each object to the object template.

13. The apparatus according to any one of claims 10-12, characterized in that the apparatus further comprises:

an updating unit, configured to determine that at least one second column exists in the candidate matrix, where only the second indicator exists in the second column, and update a template corresponding to a location of the second indicator in the second column by using a target corresponding to the location of the second indicator in the second column to obtain an updated template, where the second column is any one column in the candidate matrix; and the template updating module is used for determining at least one first column in the candidate matrix, wherein at least one first indicator and the second indicator exist in the first column, updating a template corresponding to the position of the second indicator in the first column by using a target corresponding to the position of the second indicator in the first column, and updating a template corresponding to the position of each first indicator in the at least one first indicator.

14. The apparatus according to claim 13, wherein the updating unit is specifically configured to add the coordinates of the center point of the object corresponding to the position of the second indicator in the second column to the trajectory of the template corresponding to the position of the second indicator in the second column; or, the processing unit is configured to add the coordinates of the center point of the target corresponding to the position of the second indicator in the first column to the trajectory of the template corresponding to the position of each first indicator and to the trajectory of the template corresponding to the position of the second indicator.

15. The apparatus of claim 13, wherein the updating unit is further configured to update the first confidence degrees of the updated templates, and the first confidence degree of one template is used to indicate the confidence level of tracking the target corresponding to the template with the template.

16. The apparatus according to any of claims 10-12, wherein each template of the plurality of templates has a fusion token, the fusion token being a first token indicating that no fusion of template-associated objects has occurred or a second token indicating that fusion of template-associated objects has occurred,

the determining unit is further configured to: determining one or more fusion templates from the plurality of templates according to the fusion marks of each template in the plurality of templates, wherein the fusion template refers to the template with the fusion mark as the second mark, and at least one candidate target located in the tracking area of each fusion template in the one or more fusion templates is determined; and for performing the following steps for at least one candidate target within each fused template tracking area to determine split targets within each fused template tracking area: if it is determined that at least one candidate target in a fusion template tracking area has a target which has no template and is intersected with the fusion template tracking area, determining that the fusion target is split;

the acquiring unit is further configured to acquire image feature information of each split target in a plurality of split targets from the image of the fusion target;

the determining unit is further configured to determine, as the template corresponding to each split target, a template whose similarity with the image feature information of each split target meets a preset requirement, according to the image feature information of each split target and history information of each associated target recorded in each template before fusion;

and the updating unit is also used for updating the template corresponding to each splitting target by using the information of the current moment of each splitting target.

17. The apparatus according to any of claims 10-12, wherein each of the plurality of templates further comprises a second confidence level, the updating unit further configured to: and if at least one unprocessed template exists in the plurality of templates, reducing the second confidence coefficient of the at least one unprocessed template by a preset step length, and acquiring the updated second confidence coefficient of each unprocessed template in the at least one unprocessed template.

18. The apparatus of claim 17, further comprising:

and the deleting unit is used for deleting the template with the second confidence coefficient smaller than or equal to the preset threshold when the second confidence coefficient of any template is smaller than or equal to the preset threshold.

19. A multi-target tracking processing apparatus, comprising: a memory storing code and data, a processor coupled to the memory via a bus, and a transceiver, the processor executing the code in the memory to cause the apparatus to perform the multi-target tracking processing method as claimed in any one of claims 1 to 9.

20. A computer-readable storage medium storing instructions that, when executed on a multi-target tracking processing apparatus, cause the multi-target tracking processing apparatus to perform the multi-target tracking processing method according to any one of claims 1 to 9.