CN113177920A - Target re-identification method and system of model biological tracking system - Google Patents

Target re-identification method and system of model biological tracking system Download PDF

Info

Publication number
CN113177920A
CN113177920A CN202110472329.8A CN202110472329A CN113177920A CN 113177920 A CN113177920 A CN 113177920A CN 202110472329 A CN202110472329 A CN 202110472329A CN 113177920 A CN113177920 A CN 113177920A
Authority
CN
China
Prior art keywords
target
targets
neural network
under
determining
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202110472329.8A
Other languages
Chinese (zh)
Other versions
CN113177920B (en
Inventor
于兴虎
王春翔
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Ningbo Intelligent Equipment Research Institute Co ltd
Original Assignee
Ningbo Intelligent Equipment Research Institute Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Ningbo Intelligent Equipment Research Institute Co ltd filed Critical Ningbo Intelligent Equipment Research Institute Co ltd
Priority to CN202110472329.8A priority Critical patent/CN113177920B/en
Publication of CN113177920A publication Critical patent/CN113177920A/en
Application granted granted Critical
Publication of CN113177920B publication Critical patent/CN113177920B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0012—Biomedical image inspection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/136—Segmentation; Edge detection involving thresholding
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/187—Segmentation; Edge detection involving region growing; involving region merging; involving connected component labelling
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/60—Analysis of geometric attributes
    • G06T7/62—Analysis of geometric attributes of area, perimeter, diameter or volume
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10056—Microscopic image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30242—Counting objects in image
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02A—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
    • Y02A90/00—Technologies having an indirect contribution to adaptation to climate change
    • Y02A90/10—Information and communication technologies [ICT] supporting adaptation to climate change, e.g. for weather forecasting or climate simulation

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Health & Medical Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Biophysics (AREA)
  • Software Systems (AREA)
  • Evolutionary Computation (AREA)
  • Computational Linguistics (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Biomedical Technology (AREA)
  • Mathematical Physics (AREA)
  • Data Mining & Analysis (AREA)
  • Artificial Intelligence (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Geometry (AREA)
  • Medical Informatics (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Radiology & Medical Imaging (AREA)
  • Quality & Reliability (AREA)
  • Image Analysis (AREA)

Abstract

The invention relates to a target re-identification method and a target re-identification system of a pattern biological tracking system. The method comprises the following steps: acquiring multiple frames of continuous images under a first experiment condition, and extracting a target connected domain in each frame of image; extracting each isolated connected domain in the target connected domain, and extracting the target position of the target according to each isolated connected domain; extracting the intersection region of the isolated connected domains of two adjacent frames; determining a primary associated target according to the intersection area; processing the remaining targets by using a Hungarian algorithm to determine secondary associated targets; generating a continuous image sequence aiming at the same single target in different frame images according to the primary associated target and the secondary associated target, and training a convolutional neural network; and enabling the target in the first frame image to correspond to the same target under the first experimental condition by using the trained convolutional neural network. The invention can identify the target under different experimental conditions and improve the tracking efficiency.

Description

Target re-identification method and system of model biological tracking system
Technical Field
The invention relates to the field of model biological tracking systems, in particular to a target re-identification method and a target re-identification system of a model biological tracking system.
Background
The accurate tracking mode biological motion is the basis of the operation of a micro-operation system and is an important means for verifying the effect of the micro-operation. In biology, drug administration experiments and stress response experiments are often performed, drugs are put in a culture dish containing model organisms or external stimuli are applied, the movement and the response of the model organisms to the stimuli are tracked, and the drug effects and the physiological functions of the model organisms are evaluated by analyzing tracks.
In the field of mode biological tracking, a two-stage tracking algorithm is commonly used for a single video, a first step of detecting each target (for example, obtaining information such as the position, head orientation and the like of each target) is generally to firstly obtain foreground information by using a gaussian mixture model, then perform further detection, and a second step of associating the same target of adjacent frames with target detection information of adjacent two frames. The second step is traditionally done using the Hungarian algorithm for optimal assignment, but the temporal complexity of the algorithm is O (n)3) That is, the running time of the algorithm increases with the number n of targets to the third power, and when the number of targets is large, the running efficiency of the algorithm is sharply reduced.
In the actual operation process, a plurality of tracking videos are generated by performing experiments under different conditions, and it is expected that the same target can be corresponded under the experiment conditions of different tracking videos, that is, the target a is subjected to an experiment under the first experiment condition, and the target a is still expected to be recognized under the second experiment condition, so that the behavior patterns of the same target under different experiment conditions are compared. The two-stage tracking algorithm can only track the target of a single video, cannot identify the target of different videos, and cannot compare behavior patterns of the same target under different experimental conditions.
Disclosure of Invention
The invention aims to provide a target re-identification method and a target re-identification system of a pattern biological tracking system, which aim to solve the problems that when the number of targets is large, the operation efficiency of an algorithm is sharply reduced, only the target tracking of a single video can be realized, the targets of different videos cannot be identified, and further the behavior patterns of the same target under different experimental conditions cannot be compared.
In order to achieve the purpose, the invention provides the following scheme:
a method of target re-identification for a pattern bio-tracking system, comprising:
acquiring a plurality of frames of continuous images under a first experiment condition, and extracting a target connected domain in each frame of image;
extracting each isolated connected domain in the target connected domain, and extracting the target position of the target according to each isolated connected domain;
extracting the intersection region of the isolated connected domains at the same target position of two adjacent frames;
determining a primary associated target according to the intersection region;
processing the remaining targets by using a Hungarian algorithm to determine secondary associated targets;
generating a continuous image sequence aiming at the same single target in different frame images according to the primary associated target and the secondary associated target, and training a convolutional neural network according to the continuous image sequence;
and acquiring a first frame image under a second experimental condition, and enabling a target in the first frame image to correspond to the same target under the first experimental condition by using the trained convolutional neural network.
Optionally, the determining a primary association target according to the intersection region specifically includes:
judging whether the area of the intersection area is larger than an area threshold value or not to obtain a first judgment result;
if the first judgment result shows that the area of the intersection region is larger than the area threshold, determining that two targets corresponding to the intersection region are the same target, associating the two targets, and determining a primary associated target;
and if the first judgment result shows that the area of the intersection region is not larger than the area threshold, determining that the two targets corresponding to the intersection region are not the same target and do not relate to the two targets.
Optionally, the processing the remaining targets by using the hungarian algorithm to determine the secondary associated target specifically includes:
acquiring the target position and the number of the residual targets which are not associated in the kth frame image and the kth-1 frame image;
constructing an unassociated matrix according to the target position and the number of the remaining targets of each remaining target;
and solving the unassociated matrix by using a Hungarian algorithm, associating the remaining targets according to the solved result, and determining a secondary associated target.
Optionally, the convolutional neural network specifically includes: the device comprises an input layer, three convolution layers, a pooling layer arranged between every two adjacent convolution layers, three continuous full-connection layers and an output layer;
the three convolutional layers comprise a first convolutional layer, a second convolutional layer and a third convolutional layer; the first convolutional layer comprises 256 3 × 3 convolution kernels, the second convolutional layer comprises 512 3 × 3 convolution kernels, and the third convolutional layer comprises 1024 3 × 3 convolution kernels;
the input matrix of the pooling layer is half of the input matrix;
the activation function of the full connection layer is ReLu; the activation function of the output layer is softmax.
Optionally, the acquiring a first frame image under a second experimental condition, and using the trained convolutional neural network to make a target in the first frame image correspond to the same target under the first experimental condition specifically includes:
inputting the first frame image under the second experimental condition into the trained convolutional neural network, and determining probability values of any target in the first frame image and each target under the first experimental condition;
and outputting the target under the first experiment condition corresponding to the maximum probability value as the same target of the target in the first frame image.
A target re-identification system for a pattern bio-tracking system, comprising:
the target connected domain extraction module is used for acquiring multiple frames of continuous images under a first experiment condition and extracting a target connected domain in each frame of image;
the target position detection module is used for extracting each isolated connected domain in the target connected domain and extracting the target position of the target according to each isolated connected domain;
the intersection region determining module is used for extracting intersection regions of the isolated connected domains at the same target positions of two adjacent frames;
the target primary association module is used for determining a primary association target according to the intersection region;
the target secondary association module is used for processing the remaining targets by using a Hungarian algorithm and determining secondary association targets;
the convolutional neural network training module is used for generating a continuous image sequence aiming at the same single target in different frame images according to the primary associated target and the secondary associated target and training a convolutional neural network according to the continuous image sequence;
and the re-identification module is used for acquiring a first frame image under a second experimental condition and enabling a target in the first frame image to correspond to the same target under the first experimental condition by using the trained convolutional neural network.
Optionally, the target primary association module specifically includes:
the first judgment unit is used for judging whether the area of the intersection area is larger than an area threshold value or not to obtain a first judgment result;
the target primary association unit is used for determining that two targets corresponding to the intersection area are the same target and associating the two targets to determine a primary association target if the first judgment result indicates that the area of the intersection area is larger than an area threshold;
and the target unassociated unit is used for determining that the two targets corresponding to the intersection area are not the same target and are not associated with the two targets if the first judgment result shows that the area of the intersection area is not larger than the area threshold.
Optionally, the target secondary association module specifically includes:
a residual target position and residual target number acquiring unit, configured to acquire a target position and a residual target number of each of the residual targets that are not associated in the k-th frame image and the k-1-th frame image;
the unassociated matrix construction unit is used for constructing an unassociated matrix according to the target position of each residual target and the number of the residual targets;
and the target secondary association unit is used for solving the unassociated matrix by using a Hungarian algorithm, associating the remaining targets according to the solved result and determining a secondary association target.
Optionally, the convolutional neural network specifically includes: the device comprises an input layer, three convolution layers, a pooling layer arranged between every two adjacent convolution layers, three continuous full-connection layers and an output layer;
the three convolutional layers comprise a first convolutional layer, a second convolutional layer and a third convolutional layer; the first convolutional layer comprises 256 3 × 3 convolution kernels, the second convolutional layer comprises 512 3 × 3 convolution kernels, and the third convolutional layer comprises 1024 3 × 3 convolution kernels;
the input matrix of the pooling layer is half of the input matrix;
the activation function of the full connection layer is ReLu; the activation function of the output layer is softmax.
Optionally, the re-identification module specifically includes:
a probability value calculation unit, configured to input the first frame image under the second experimental condition to the trained convolutional neural network, and determine a probability value between any one of the targets in the first frame image and each target under the first experimental condition;
and the re-identification unit outputs the target under the first experiment condition corresponding to the maximum probability value as the same target of the target in the first frame image.
According to the specific embodiment provided by the invention, the invention discloses the following technical effects: the invention provides a target re-identification method and a target re-identification system of a pattern biological tracking system, which utilize a connected domain to carry out primary association on a target, greatly improve the efficiency of target association, have processing speed not limited by the number of the target, and generate abundant data sets in a short time for the training of a convolutional neural network; secondly, performing secondary association on the remaining targets by using a Hungarian algorithm, wherein the operation time of the secondary association is shorter because most targets are associated in the previous step, and the operation efficiency is further improved; and finally, training a convolutional neural network and identifying a target according to the convolutional neural network, so that the tracking efficiency is improved while the target is identified under different experimental conditions.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings needed in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings without creative efforts.
FIG. 1 is a flow chart of a target re-identification method of the model biometric tracking system of the present invention;
FIG. 2 is a schematic diagram of an input image provided by the present invention;
FIG. 3 is a schematic diagram of the extracted target connected domain provided by the present invention;
FIG. 4 is a schematic diagram of isolated connected domains extracted according to the present invention;
FIG. 5 is a schematic diagram of the detected target location provided by the present invention;
FIG. 6 is a schematic diagram of isolated connected domains of each target of the k-1 frame provided by the present invention;
FIG. 7 is a schematic diagram of isolated connected domains of each target in the kth frame according to the present invention;
FIG. 8 is a schematic diagram of an intersection region after AND operation on isolated connected components of a k-1 frame and a k-th frame according to the present invention;
FIG. 9 is a schematic diagram of the primary association objective provided by the present invention;
FIG. 10 is a schematic diagram of a sequence of successive images of the same object provided by the present invention for 4 frames of images;
FIG. 11 is a schematic diagram of a convolutional neural network structure provided in the present invention;
FIG. 12 is a block diagram of a target re-identification system of the model biometric tracking system of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The invention aims to provide a target re-identification method and a target re-identification system of a model biological tracking system, which can identify targets under different experimental conditions and improve the tracking efficiency.
In order to make the aforementioned objects, features and advantages of the present invention comprehensible, embodiments accompanied with figures are described in further detail below.
Fig. 1 is a flowchart of a target re-identification method of a pattern biometric tracking system according to the present invention, and as shown in fig. 1, a target re-identification method of a pattern biometric tracking system includes:
step 101: acquiring a plurality of frames of continuous images under a first experiment condition, and extracting a target connected domain in each frame of the images.
The foreground (namely the target) of the image is extracted by using the cv of opencv, wherein the input of the function is continuous historical images and a learning rate, the internal mechanism of the function is to input a series of historical images (such as 100 frames of images), each pixel point is modeled by using a plurality of Gaussian functions, the pixel value with high probability density is considered as a background, and the learning rate is updated for each frame of input images. The final effect of the function is that for the input image, taking fig. 2 as an example, there are many patterns of biological zebra fish in the visual field, after a series of continuous history images are input, the function will extract the target foreground, as shown in fig. 3, the foreground is represented by white pixels, and the background is represented by black pixels.
Step 102: and extracting each isolated connected domain in the target connected domain, and extracting the target position of the target according to each isolated connected domain.
findContours are used to extract each isolated connected domain, which can be represented in a different color in practical applications, from the extracted connected domain information, as shown in FIG. 4, and then further extract positional information of the target from each isolated connected domain information (which is designed by the user for different model creatures from the further detection step of the connected domain), as shown in FIG. 5, the circle represents the extracted head position.
Step 103: and extracting the intersection areas of the isolated connected domains at the same target position in two adjacent frames.
For the connected components of two adjacent frames, as shown in fig. 6 and 7, the circles in fig. 6 represent the connected components of each target of the k-1 th frame, the circles in fig. 7 represent the connected components of each target of the k-1 th frame, and the connected components of the k-1 th frame and the k-1 th frame are subjected to an and operation (i.e., an intersection region is extracted; as shown in fig. 8, the intersection region is a region where two isolated connected components intersect).
Step 104: and determining a primary associated target according to the intersection area.
If the area of the intersection region is larger than a certain threshold, the two corresponding frame targets where the red region is located are considered to be the same target, as shown in fig. 9, the intersection area of the two connected regions in the frame is larger than a certain threshold, and the two corresponding targets are considered to be the same target. The step can correlate most targets, particularly for the target which moves slowly, and has the advantages that the running efficiency is not limited by the number of the targets, and only the image and operation and threshold judgment are needed, so that the correlation efficiency is greatly improved.
Step 105: and processing the remaining targets by using a Hungarian algorithm, and determining a secondary associated target.
And processing the remaining targets associated in the last step by using a Hungarian algorithm.
For the image of the k-th frame, there is currently detection information of each target of the k-1-th frame that is not related
Figure BDA0003045952940000071
And detection information of the k-th frame
Figure BDA0003045952940000072
In practical application, the detected information is a target position, Mk-1Is the remaining unassociated target number, N, of the (k-1) th framekFor the remaining unassociated target number of the kth frame,
Figure BDA0003045952940000073
wherein
Figure BDA0003045952940000074
The x-position coordinate and the y-position coordinate of the jth target detected in the kth frame are respectively. The following unassociated matrices are established:
Figure BDA0003045952940000081
wherein,
Figure BDA0003045952940000082
is the distance between the detection value of the k-1 th frame and the detection value of the k-th frame.
Figure BDA0003045952940000083
The objective association can be carried out by solving the matrix by using Hungarian algorithm (judging which detection of the k-1 frame and which detection of the k frame come from the same objective), the association matrix is solved by using a linear _ sum _ assignment function of the scinit-left function library, the input of the function is the unassociated matrix, and the output is the cost matrix D (Z)k-1,Zk) The matrix A with the same size outputs each element of the matrix A to be 0 or 1, if A (i, j) is 1, the ith correlation of the residual target of the k-1 th frame and the jth correlation of the residual target of the k-1 th frame is represented, the target correlation problem is equivalent to solving the matrix A, so that the following target correlation cost energy function is minimized,
Figure BDA0003045952940000084
constraint conditions are as follows:
if it is not
Figure BDA0003045952940000085
If it is not
Figure BDA0003045952940000086
Two constraints ensure that each row or column has only one association, that is, one object of the k-1 frame can only be associated with one object of the k-th frame, but not a plurality of objects. The time complexity of this step is O (n)3) I.e. the running time of the algorithm increases with the number n of targets to the third power, but since the last step is associated with most targets, the running time of this step is very short, thus increasing the efficiency. Meanwhile, because the step is based on a greedy algorithm to obtain an optimal solution instead of the optimal solution, if the targets are dense, namely the relative distances among a plurality of targets are small, the algorithm may be mismatched, but because the previous step completes the association of a part of dense targets, the number of targets processed by the step is reduced sharply, and the optimal solution is obtained more easily.
Step 106: generating a continuous image sequence aiming at the same single target in different frame images according to the primary associated target and the secondary associated target, and training a convolutional neural network according to the continuous image sequence; the convolutional neural network is used for determining probability values of the targets.
For the primary associated target and the secondary associated target obtained through association in steps 101 to 105, performing an and operation on the target images in consecutive time frames and the corresponding connected domains (i.e., the foreground pixel portion is preserved and the background pixels are removed), and generating a consecutive image sequence for a single target, as shown in fig. 10. Due to the high efficiency of target association from steps 101 to 105, a rich data set can be generated within a defined time for training of the neural network. For each object, a series of image data is generated, and if there are 1000 frames of images and 20 objects, 20 data packets of 1000 pictures are generated, and 1000 data in each data packet correspond to a uniform object.
The structure of the convolutional neural network of the present invention is shown in fig. 11, and the convolutional neural network is used for determining the probability value of each target, so as to realize target re-identification under different experimental conditions.
In fig. 11, Input represents a gray scale map (the size is set by the user) Input to a fixed size, that is, a single gray scale map generated in step 5.
C represents convolution layer, s x s @ f represents f convolution kernels with the size of s x s, and the calculation output of each convolution kernel is calculated by using the following formula
Figure BDA0003045952940000091
Wherein,
Figure BDA0003045952940000092
represents the k-thoutThe value of the ith row and the jth column of the matrix output by the convolution kernel,
Figure BDA0003045952940000093
represents the k-thinThe value of the (i + m) th row and (j + n) th column of the input matrix,
Figure BDA0003045952940000094
represents to the k-thinInput of (2) and (k)outThe output convolution kernel m row and n column numerical value, KinThe number of input feature matrices of the previous layer is represented, and m and n are position parameters for convolution calculation corresponding to the size of a convolution kernel. The activation function is ReLu and the formula is
Figure BDA0003045952940000095
Wherein x is the value of the output matrix, and y is the value of the activated output matrix.
S represents the pooling layer and the output is calculated using the following formula:
yi,j＝max(x2i,2j,x2i+1,2j,x2i,2j+1,x2i+1,2j+1)
wherein, yi,jRepresenting the value, x, of the output matrix at row i and column j2i,2jFor the values input into the matrix at row 2i and column 2j, max represents the operation taking the maximum value. The output matrix of the pooling layer has a size half that of the input matrix.
F represents the fully connected layer, the upper number represents the number of output cells, and is calculated using the following formula:
Figure BDA0003045952940000101
wherein, yjJ-th cell representing the output layer, wi,jRepresenting the weight of the jth cell of the output layer relative to the ith cell of the input, xiA value representing the ith element of the input, vinRepresenting the length of the input vector, bjA bias term. The activation function of the fully connected layer still uses ReLu.
Output represents the Output layer, the calculation formula is the same as the full connection layer, but the activation function uses softmax, and the formula is:
Figure BDA0003045952940000102
wherein
Figure BDA0003045952940000103
Representing the ith output, N representing the length of the input vector, equal to the number of objects desired to be identified, xjRepresenting the jth value of the input vector. The physical significance of the output is, yiThe probability that the image is the ith target is shown (for example, 20 targets exist, the image of one target is input, the probabilities that the 20 targets are respectively output are shown, for example, the probability corresponding to the 1 st target is 0.05, the probability corresponding to the 1 st target is 0.01 … …, the probability corresponding to the 5 th target is 0.9 … …, the probability corresponding to the 20 th target is 0.005, and the output with the largest probability value is finally selected as the ID of the target).
The loss function in the neural network training is calculated by using the following formula and using the cross entropy:
Figure BDA0003045952940000104
wherein liLabels representing the ith cell of the network, if the ith output is the ith target, then liIs 1, otherwise is 0. And (3) putting the data generated in the step (5) into a neural network, and training by using a back propagation algorithm until the recognition accuracy of the network (namely the proportion of the correctly recognized target relative to the whole data set) reaches more than 95%.
Step 107: and acquiring a first frame image under a second experimental condition, and enabling a target in the first frame image to correspond to the same target under the first experimental condition by using the trained convolutional neural network.
For a first frame of video under a new experimental condition, generating images of each target in the first frame of video according to steps 101 to 105, as shown in fig. 10, inputting the images of each target into a trained convolutional neural network to obtain a probability value between any target image and each target under the first experimental condition, and outputting the neural network with the maximum probability value for each target as an ID of the target, thereby realizing correspondence, namely re-identification, of the same target under different experimental conditions. Repeating steps 101-105 for video subsequent to the first frame of video under the second experimental condition continues to track the target under the current experimental condition.
For example, the 2 nd target under the second experimental condition corresponds to the 5 th target under the experimental condition 1 (in the first video, the ID of one target is 1, in the new video, the ID assigned to the target by the detection algorithm is 5, the neural network determines that the 5 th target of the new video and the 1 st target of the first video are the same target, so that each motion parameter of the 5 th target of the new video is added into the motion parameter sequence of the first target of the first video, in the second experimental condition (i.e. in the new video), the image sampled from the first frame is input into the neural network, the ID corresponding to the unit with the maximum output probability in the neural network is the target under the first experimental condition corresponding to the second experimental condition, the target with the ID of 5 detected by the first frame detection algorithm under the second experimental condition is input into the neural network, the first unit value output by the network is the maximum, the neural network determines that the 5 th object of the new video is the same as the 1 st object of the first video.
The invention uses the connected domain information to carry out the initial association, thereby greatly improving the efficiency of target association, and simultaneously, the processing speed is not limited by the number of targets, because only the operation and the threshold judgment of the connected domain image are needed. Because the primary association step is carried out, the time complexity is O (n) when the secondary association is carried out by using the Hungarian algorithm3) I.e. the running time of the algorithm increases with the number n of targets to the third power, but since the last step is associated with most targets, the running time of this step is very short, thus increasing the efficiency.
In addition, the Hungarian algorithm is based on a greedy algorithm to obtain an optimized solution instead of the optimal solution, if the targets are dense, namely the relative distances among a plurality of targets are small, mismatching may occur in the algorithm, but as the previous step completes the association of a part of dense targets, the number of the targets processed in the step is reduced sharply, so that the optimal solution is obtained more easily, the accuracy and robustness of the target association are improved, and the target re-identification under different experimental conditions is realized.
The convolutional neural network has long running time, and the tracking is carried out by continuously using the steps 101 to 105 after the target identification is finished, so that the tracking efficiency is improved while the target identification under different experimental conditions is realized.
Since the appearance characteristics of model creatures are very similar, it is almost impossible to manually identify the same target when experiments are performed under different experimental conditions. And the deep neural network can realize the special characteristics of each target by virtue of strong characteristic extraction capability so as to realize the identification of different targets. According to the method, the data set generation of the target is realized by extracting the primary associated target and the secondary associated target, the convolutional neural network is trained automatically, and the target re-identification under different experimental conditions (namely the correspondence of the same target in different images) is realized.
Fig. 12 is a structural diagram of a target re-identification system of a pattern biometric tracking system according to the present invention, wherein the target re-identification system of the pattern biometric tracking system comprises:
a target connected component extracting module 1201, configured to acquire multiple frames of continuous images under the first experimental condition, and extract a target connected component in each frame of the images.
A target position detection module 1202, configured to extract each isolated connected component in the target connected component, and extract a target position of the target according to each isolated connected component.
An intersection region determining module 1203, configured to extract intersection regions of the isolated connected component at two adjacent frames and the same target position.
And a target primary association module 1204, configured to determine a primary association target according to the intersection region.
The target primary association module 1204 specifically includes: the first judgment unit is used for judging whether the area of the intersection area is larger than an area threshold value or not to obtain a first judgment result; the target primary association unit is used for determining that two targets corresponding to the intersection area are the same target and associating the two targets to determine a primary association target if the first judgment result indicates that the area of the intersection area is larger than an area threshold; and the target unassociated unit is used for determining that the two targets corresponding to the intersection area are not the same target and are not associated with the two targets if the first judgment result shows that the area of the intersection area is not larger than the area threshold.
And the target secondary association module 1205 is configured to process the remaining targets by using the hungarian algorithm, and determine a secondary association target.
The target secondary association module 1205 specifically includes: a residual target position and residual target number acquiring unit, configured to acquire a target position and a residual target number of each of the residual targets that are not associated in the k-th frame image and the k-1-th frame image; the unassociated matrix construction unit is used for constructing an unassociated matrix according to the target position of each residual target and the number of the residual targets; and the target secondary association unit is used for solving the unassociated matrix by using a Hungarian algorithm, associating the remaining targets according to the solved result and determining a secondary association target.
A convolutional neural network training module 1206, configured to generate a continuous image sequence for the same single target in different frame images according to the primary associated target and the secondary associated target, and train a convolutional neural network according to the continuous image sequence; the convolutional neural network is used for determining probability values of the targets.
The convolutional neural network specifically includes: the device comprises an input layer, three convolution layers, a pooling layer arranged between every two adjacent convolution layers, three continuous full-connection layers and an output layer; the three convolutional layers comprise a first convolutional layer, a second convolutional layer and a third convolutional layer; the first convolutional layer comprises 256 3 × 3 convolution kernels, the second convolutional layer comprises 512 3 × 3 convolution kernels, and the third convolutional layer comprises 1024 3 × 3 convolution kernels; the input matrix of the pooling layer is half of the input matrix; the activation function of the full connection layer is ReLu; the activation function of the output layer is softmax.
The re-recognition module 1207 is configured to acquire a first frame image under a second experimental condition, and make a target in the first frame image correspond to a same target under the first experimental condition by using the trained convolutional neural network.
The re-identification module 1207 specifically includes: a probability value calculation unit, configured to input the first frame image under the second experimental condition to the trained convolutional neural network, and determine a probability value between any one of the targets in the first frame image and each target under the first experimental condition; and the re-identification unit outputs the target under the first experiment condition corresponding to the maximum probability value as the same target of the target in the first frame image.
The target association method combined with the connected domain information can improve the operation efficiency, and can generate abundant data sets in a short time for training the convolutional neural network, so that the convolutional neural network for target identification is trained, and the same target in different images or videos is identified.
The embodiments in the present description are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other. For the system disclosed by the embodiment, the description is relatively simple because the system corresponds to the method disclosed by the embodiment, and the relevant points can be referred to the method part for description.
The principles and embodiments of the present invention have been described herein using specific examples, which are provided only to help understand the method and the core concept of the present invention; meanwhile, for a person skilled in the art, according to the idea of the present invention, the specific embodiments and the application range may be changed. In view of the above, the present disclosure should not be construed as limiting the invention.

Claims (10)

1. A method for re-identifying a target in a pattern biometric tracking system, comprising:
acquiring a plurality of frames of continuous images under a first experiment condition, and extracting a target connected domain in each frame of image;
extracting each isolated connected domain in the target connected domain, and extracting the target position of the target according to each isolated connected domain;
extracting the intersection region of the isolated connected domains at the same target position of two adjacent frames;
determining a primary associated target according to the intersection region;
processing the remaining targets by using a Hungarian algorithm to determine secondary associated targets;
generating a continuous image sequence aiming at the same single target in different frame images according to the primary associated target and the secondary associated target, and training a convolutional neural network according to the continuous image sequence;
and acquiring a first frame image under a second experimental condition, and enabling a target in the first frame image to correspond to the same target under the first experimental condition by using the trained convolutional neural network.
2. The method of claim 1, wherein determining the primary associated target according to the intersection region comprises:
judging whether the area of the intersection area is larger than an area threshold value or not to obtain a first judgment result;
if the first judgment result shows that the area of the intersection region is larger than the area threshold, determining that two targets corresponding to the intersection region are the same target, associating the two targets, and determining a primary associated target;
and if the first judgment result shows that the area of the intersection region is not larger than the area threshold, determining that the two targets corresponding to the intersection region are not the same target and do not relate to the two targets.
3. The method for re-identifying the target of the pattern biometric tracking system according to claim 1, wherein the processing of the remaining targets by the Hungarian algorithm to determine the secondary associated target specifically comprises:
acquiring the target position and the number of the residual targets which are not associated in the kth frame image and the kth-1 frame image;
constructing an unassociated matrix according to the target position and the number of the remaining targets of each remaining target;
and solving the unassociated matrix by using a Hungarian algorithm, associating the remaining targets according to the solved result, and determining a secondary associated target.
4. The method of claim 1, wherein the convolutional neural network comprises: the device comprises an input layer, three convolution layers, a pooling layer arranged between every two adjacent convolution layers, three continuous full-connection layers and an output layer;
the three convolutional layers comprise a first convolutional layer, a second convolutional layer and a third convolutional layer; the first convolutional layer comprises 256 3 × 3 convolution kernels, the second convolutional layer comprises 512 3 × 3 convolution kernels, and the third convolutional layer comprises 1024 3 × 3 convolution kernels;
the input matrix of the pooling layer is half of the input matrix;
the activation function of the full connection layer is ReLu; the activation function of the output layer is softmax.
5. The method of claim 1, wherein the acquiring a first frame of image under a second experimental condition and using the trained convolutional neural network to make the target in the first frame of image correspond to the same target under the first experimental condition comprises:
inputting the first frame image under the second experimental condition into the trained convolutional neural network, and determining probability values of any target in the first frame image and each target under the first experimental condition;
and outputting the target under the first experiment condition corresponding to the maximum probability value as the same target of the target in the first frame image.
6. A target re-identification system for a patterned biometric tracking system, comprising:
the target connected domain extraction module is used for acquiring multiple frames of continuous images under a first experiment condition and extracting a target connected domain in each frame of image;
the target position detection module is used for extracting each isolated connected domain in the target connected domain and extracting the target position of the target according to each isolated connected domain;
the intersection region determining module is used for extracting intersection regions of the isolated connected domains at the same target positions of two adjacent frames;
the target primary association module is used for determining a primary association target according to the intersection region;
the target secondary association module is used for processing the remaining targets by using a Hungarian algorithm and determining secondary association targets;
the convolutional neural network training module is used for generating a continuous image sequence aiming at the same single target in different frame images according to the primary associated target and the secondary associated target and training a convolutional neural network according to the continuous image sequence;
and the re-identification module is used for acquiring a first frame image under a second experimental condition and enabling a target in the first frame image to correspond to the same target under the first experimental condition by using the trained convolutional neural network.
7. The system for re-identifying the target of the pattern biometric tracking system as claimed in claim 6, wherein the primary target association module comprises:
the first judgment unit is used for judging whether the area of the intersection area is larger than an area threshold value or not to obtain a first judgment result;
the target primary association unit is used for determining that two targets corresponding to the intersection area are the same target and associating the two targets to determine a primary association target if the first judgment result indicates that the area of the intersection area is larger than an area threshold;
and the target unassociated unit is used for determining that the two targets corresponding to the intersection area are not the same target and are not associated with the two targets if the first judgment result shows that the area of the intersection area is not larger than the area threshold.
8. The system for re-identifying the target of the pattern biometric tracking system as claimed in claim 6, wherein the target secondary correlation module comprises:
a residual target position and residual target number acquiring unit, configured to acquire a target position and a residual target number of each of the residual targets that are not associated in the k-th frame image and the k-1-th frame image;
the unassociated matrix construction unit is used for constructing an unassociated matrix according to the target position of each residual target and the number of the residual targets;
and the target secondary association unit is used for solving the unassociated matrix by using a Hungarian algorithm, associating the remaining targets according to the solved result and determining a secondary association target.
9. The system for object re-recognition of a patterned biometric tracking system as in claim 6, wherein the convolutional neural network comprises in particular: the device comprises an input layer, three convolution layers, a pooling layer arranged between every two adjacent convolution layers, three continuous full-connection layers and an output layer;
the three convolutional layers comprise a first convolutional layer, a second convolutional layer and a third convolutional layer; the first convolutional layer comprises 256 3 × 3 convolution kernels, the second convolutional layer comprises 512 3 × 3 convolution kernels, and the third convolutional layer comprises 1024 3 × 3 convolution kernels;
the input matrix of the pooling layer is half of the input matrix;
the activation function of the full connection layer is ReLu; the activation function of the output layer is softmax.
10. The system for re-identifying a target of a pattern biometric tracking system as claimed in claim 6, wherein the re-identification module comprises:
a probability value calculation unit, configured to input the first frame image under the second experimental condition to the trained convolutional neural network, and determine a probability value between any one of the targets in the first frame image and each target under the first experimental condition;
and the re-identification unit outputs the target under the first experiment condition corresponding to the maximum probability value as the same target of the target in the first frame image.
CN202110472329.8A 2021-04-29 2021-04-29 Target re-identification method and system of model biological tracking system Active CN113177920B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110472329.8A CN113177920B (en) 2021-04-29 2021-04-29 Target re-identification method and system of model biological tracking system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110472329.8A CN113177920B (en) 2021-04-29 2021-04-29 Target re-identification method and system of model biological tracking system

Publications (2)

Publication Number Publication Date
CN113177920A true CN113177920A (en) 2021-07-27
CN113177920B CN113177920B (en) 2022-08-09

Family

ID=76925317

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110472329.8A Active CN113177920B (en) 2021-04-29 2021-04-29 Target re-identification method and system of model biological tracking system

Country Status (1)

Country Link
CN (1) CN113177920B (en)

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111598067A (en) * 2020-07-24 2020-08-28 浙江大华技术股份有限公司 Re-recognition training method, re-recognition method and storage device in video
CN111723611A (en) * 2019-03-20 2020-09-29 北京沃东天骏信息技术有限公司 Pedestrian re-identification method and device and storage medium
CN111914664A (en) * 2020-07-06 2020-11-10 同济大学 Vehicle multi-target detection and track tracking method based on re-identification

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111723611A (en) * 2019-03-20 2020-09-29 北京沃东天骏信息技术有限公司 Pedestrian re-identification method and device and storage medium
CN111914664A (en) * 2020-07-06 2020-11-10 同济大学 Vehicle multi-target detection and track tracking method based on re-identification
CN111598067A (en) * 2020-07-24 2020-08-28 浙江大华技术股份有限公司 Re-recognition training method, re-recognition method and storage device in video

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
CHUNXIANG WANG 等: "DanioSense: Automated High-Throughput Quantification of Zebrafish Larvae Group Movement", 《IEEE》 *

Also Published As

Publication number Publication date
CN113177920B (en) 2022-08-09

Similar Documents

Publication Publication Date Title
Wang et al. Robust object detection under occlusion with context-aware compositionalnets
CN110781843B (en) Classroom behavior detection method and electronic equipment
CN107609460B (en) Human body behavior recognition method integrating space-time dual network flow and attention mechanism
CN109117701B (en) Pedestrian intention identification method based on graph convolution
CN107122736B (en) Human body orientation prediction method and device based on deep learning
CN106951923B (en) Robot three-dimensional shape recognition method based on multi-view information fusion
CN110120064B (en) Depth-related target tracking algorithm based on mutual reinforcement and multi-attention mechanism learning
Wang et al. Anomalous behaviors detection for underwater fish using AI techniques
Kubota et al. Perceptual control based on prediction for natural communication of a partner robot
CN104063872B (en) Method for detecting salient regions in sequence images based on improved visual attention model
CN111626184A (en) Crowd density estimation method and system
Tavanaei et al. A minimal spiking neural network to rapidly train and classify handwritten digits in binary and 10-digit tasks
CN112861718A (en) Lightweight feature fusion crowd counting method and system
CN112434599A (en) Pedestrian re-identification method based on random shielding recovery of noise channel
CN114638408A (en) Pedestrian trajectory prediction method based on spatiotemporal information
CN110969110A (en) Face tracking method and system based on deep learning
CN113608663A (en) Fingertip tracking method based on deep learning and K-curvature method
CN112906520A (en) Gesture coding-based action recognition method and device
CN113034542B (en) Moving target detection tracking method
Chen et al. Tssd: Temporal single-shot detector based on attention and lstm
Wang et al. Single shot multibox detector with deconvolutional region magnification procedure
CN113177920B (en) Target re-identification method and system of model biological tracking system
Mozafari et al. A novel fuzzy hmm approach for human action recognition in video
CN111539390A (en) Small target image identification method, equipment and system based on Yolov3
CN113158870B (en) Antagonistic training method, system and medium of 2D multi-person gesture estimation network

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant