WO2021259055A1 - Human body tracking method and device based on rgb-d image - Google Patents
Human body tracking method and device based on rgb-d image Download PDFInfo
- Publication number
- WO2021259055A1 WO2021259055A1 PCT/CN2021/098724 CN2021098724W WO2021259055A1 WO 2021259055 A1 WO2021259055 A1 WO 2021259055A1 CN 2021098724 W CN2021098724 W CN 2021098724W WO 2021259055 A1 WO2021259055 A1 WO 2021259055A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- trajectory
- pedestrian
- center
- state
- depth image
- Prior art date
Links
- 238000000034 method Methods 0.000 title claims abstract description 61
- 238000012544 monitoring process Methods 0.000 claims abstract description 10
- 230000005484 gravity Effects 0.000 claims description 144
- 238000004422 calculation algorithm Methods 0.000 claims description 27
- 238000001514 detection method Methods 0.000 claims description 19
- 238000004364 calculation method Methods 0.000 claims description 8
- 238000004590 computer program Methods 0.000 claims description 7
- 238000012790 confirmation Methods 0.000 claims description 7
- 239000011159 matrix material Substances 0.000 claims description 7
- 238000003860 storage Methods 0.000 claims description 7
- 238000001914 filtration Methods 0.000 claims description 4
- 230000008859 change Effects 0.000 claims description 3
- 238000005192 partition Methods 0.000 claims description 3
- 238000005516 engineering process Methods 0.000 description 9
- 230000009286 beneficial effect Effects 0.000 description 5
- 230000008569 process Effects 0.000 description 4
- 238000013135 deep learning Methods 0.000 description 3
- 238000012545 processing Methods 0.000 description 3
- 238000011160 research Methods 0.000 description 2
- 230000006399 behavior Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000011840 criminal investigation Methods 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- VOEYXMAFNDNNED-UHFFFAOYSA-N metolcarb Chemical compound CNC(=O)OC1=CC=CC(C)=C1 VOEYXMAFNDNNED-UHFFFAOYSA-N 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 230000003252 repetitive effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
- G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
- G06T7/277—Analysis of motion involving stochastic approaches, e.g. using Kalman filters
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
- G06T7/292—Multi-camera tracking
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10024—Color image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20024—Filtering details
- G06T2207/20032—Median filtering
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30196—Human being; Person
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30241—Trajectory
Definitions
- the invention relates to the technical field of human body tracking, in particular to a method and device for human body tracking based on RGB-D images.
- Multi-target Multi-camera tracking is a very important research topic. This technology can be widely used in crime and criminal investigation, warehouse management, unmanned shopping, unmanned driving and other scenarios, and has high practical value.
- the cross-camera multi-target tracking technology is mainly to solve the problem of continuous positioning and tracking confirmation of pedestrians between different cameras.
- the more mature cross-camera multi-target tracking technology mostly tracks targets close to parallel perspective in open scenes, but the actual monitoring scenes are mostly limited by environmental factors.
- the camera is usually installed as a The angle of oblique shooting, so that the problem of pedestrian occlusion will follow, which will cause large differences in the posture of pedestrians from multiple viewing angles.
- These problems will directly affect the tracking quality of pedestrian trajectories. Solve these problems for cross-camera multi-target tracking technology. It is of great significance to move from academic research to actual production.
- the object of the present invention is to provide a human body tracking method and device based on RGB-D images, which adopts multiple RGB-D depth cameras to shoot the monitored area from the top, and by tracking the 3D center of gravity of the pedestrian’s head, it is possible to avoid the appearance of the larger and larger shape of the human body frame tracking.
- the problem of easy occlusion improves the accuracy of pedestrian trajectory tracking.
- the first aspect of the present invention provides a human body tracking method based on RGB-D images, including:
- the predicted position of the 3D center of gravity of each pedestrian trajectory is matched with the actual position of the 3D center of gravity of the corresponding head frame of each depth image, and the trajectory tracking status of each pedestrian trajectory is updated according to the tracking matching result.
- the updated status includes new status and normal Status, lost status and deleted status;
- the updated status includes the initial state, the entry state, the registration state, and the exit state;
- the ReID method is used to match and retrieve the lost pedestrian trajectory and update accordingly, otherwise according to the depth
- the position coordinates of the 3D center of gravity of the head frame matched by the tracking in the image are updated correspondingly to the pedestrian trajectory, where x>0, and x is an integer.
- detecting the human body frame, the human head frame, and the location of the area of the pedestrian in each depth image respectively, and the method of binding the human body frame and the human head frame of the same pedestrian in the depth image to each other includes:
- the bipartite graph maximum matching algorithm is used to screen out the human body frame and head frame belonging to the same pedestrian in each depth image to bind each other.
- the method of using the bipartite graph maximum matching algorithm to filter out the human body frame and the human head frame belonging to the same pedestrian in each depth image based on the inclusion degree corresponding to each depth image includes:
- the method for calculating the predicted position of the 3D center of gravity of the pedestrian trajectory includes:
- Multi-dimensional modeling is performed on the spatial position of the 3D center of gravity of the pedestrian trajectory.
- the dimensional vector of the model includes ( x, y, z, h, V x , V y , V z ), where x, y, z correspond to 3D
- the three-dimensional coordinates of the center of gravity point, V x , V y , and V z correspondingly represent the movement speed of the 3D center of gravity point in the coordinate direction of the corresponding dimension, and h represents the height of the pedestrian to which the 3D center of gravity point belongs;
- the method of tracking and matching the predicted position of the 3D center of gravity of each pedestrian trajectory with the actual position of the 3D center of gravity of the corresponding head frame of each depth image, and updating the trajectory tracking status of each pedestrian trajectory according to the tracking matching result includes:
- the Kalman filter tracking algorithm is used to track the actual position of the 3D center of gravity of the corresponding head frame in each depth image of the current frame to obtain the actual position of the 3D center of gravity;
- the bipartite graph maximum matching algorithm is used based on the cost matrix to filter out the initial pairing of each pedestrian trajectory with the actual position of the 3D center of gravity in each depth image in the current frame;
- the primaries whose cost metric is less than or equal to the cost threshold are screened out and considered as successful, and the primaries whose cost metric is greater than the cost threshold are screened out as unmatched;
- the unpaired primary selection pairing includes the remaining unpaired head frame 3D center of gravity points and the remaining unpaired pedestrian trajectories.
- the trajectory tracking status of is updated to the lost status;
- the trajectory tracking state is the initial state
- continuous m frames of trajectory tracking state are pedestrian trajectories in the lost state
- the trajectory tracking state of the pedestrian trajectory is updated to the deleted state, where n>0, m>0, and n and m are both integers.
- it also includes:
- the method of updating the trajectory area status of each pedestrian trajectory based on the regional position of the 3D center of gravity in each depth image includes:
- the method for judging that the track tracking state of the pedestrian track is the lost state includes:
- the trajectory tracking state of the pedestrian trajectory is considered to be a lost state.
- the second aspect of the present invention provides a human body tracking device based on RGB-D images, which is applied to the RGB-D image-based human body tracking method described in the above technical solution, and the device includes:
- the partition setting unit is used to divide the monitoring area into the target area, the registration area, and the target area in sequence according to the route of travel, and use multiple distributed depth cameras to capture real-time depth images;
- the detection frame binding unit is used to detect the human body frame, human head frame and the location of the area of the pedestrian in each depth image, and bind the human body frame and human head frame of the same pedestrian in the depth image;
- Trajectory tracking state detection unit is used to track and match the predicted position of the 3D center of gravity of each pedestrian trajectory with the actual position of the 3D center of gravity of the corresponding head frame of each depth image, and update the trajectory tracking status of each pedestrian trajectory according to the tracking matching result,
- the updated status includes new status, normal status, lost status and deleted status;
- the trajectory area status detection unit updates the trajectory area status of each pedestrian trajectory based on the location of the 3D center of gravity in each depth image.
- the updated status includes the initial status, the entry status, the registration status, and the exit status;
- the trajectory tracking unit when the corresponding trajectory tracking state of any pedestrian trajectory in the continuous x-frame depth image is the lost state and the trajectory area state is the registered state or the entered state, the ReID method is used to match and retrieve the lost pedestrian trajectory and update it accordingly , Otherwise, update the pedestrian trajectory correspondingly according to the coordinates of the 3D center of gravity of the human head frame matched by the tracking in the depth image, where x>0, and x is an integer.
- a third aspect of the present invention provides a computer-readable storage medium on which a computer program is stored.
- the computer program is run by a processor, the steps of the human body tracking method based on RGB-D images are executed.
- the present invention has the following beneficial effects:
- the monitoring area is divided into the target outside area, the registration area and the target area in sequence according to the travel route, that is to say, the monitoring area that pedestrians first enter is the target outside area. Then enter the target area through the registered area through the outside area of the target.
- the pedestrian leaving the surveillance area is the opposite of the above entry route.
- Detect the human body frame, head frame, and location of the pedestrian in each depth image and bind the human body frame and head frame of the same pedestrian in the depth image to each other, and then according to the actual 3D center of gravity of the head frame in each depth image
- the position is matched with the predicted position of the 3D center of gravity of the pedestrian trajectory, and the tracking status of each pedestrian trajectory is updated according to the obtained tracking matching result, and according to the 3D center of gravity of the head frame in each depth image of the current frame.
- the location of the area updates the status of the trajectory area of the pedestrian trajectory, so that when the tracking and matching of the 3D center of gravity of the head frame is normal, the position information method (the position coordinate of the 3D center of gravity of the head frame) is used to update the pedestrian trajectory.
- the ReID strategy is used to match the lost pedestrian trajectory Retrieve and update.
- the 3D center of gravity point of the head frame is effectively tracked and matched by the way of taking the depth picture overhead and polling calculation, and solves the problem of tracking failure caused by occlusion in the process of cross-camera pedestrian tracking.
- the registration area it is possible to automatically register the base library feature data table for the pedestrian when the pedestrian enters the registration area, so that when the pedestrian trajectory matching based on the location coordinate fails, it will automatically switch to the ReID strategy using deep learning to match the pedestrian trajectory. Retrieval improves the accuracy and reliability of trajectory tracking results.
- FIG. 1 is a schematic flowchart of a human body tracking method based on RGB-D images in Embodiment 1 of the present invention
- Fig. 2 is a schematic flow chart of the mutual binding processing of the human body frame and the human head frame of the same pedestrian in each depth image of the current frame in Fig. 1.
- This embodiment provides a human body tracking method based on RGB-D images, including:
- the predicted position of the 3D center of gravity of each pedestrian trajectory is matched with the actual position of the 3D center of gravity of the corresponding head frame of each depth image, and the trajectory tracking status of each pedestrian trajectory is updated according to the tracking matching result.
- the updated status includes new status and normal Status, lost status and deleted status;
- the updated status includes the initial state, the entry state, the registration state, and the exit state;
- the lost pedestrian trajectory is retrieved and updated according to the ReID method. Otherwise, the 3D center of gravity of the matched head frame is tracked in the depth image. The point position coordinates update the pedestrian trajectory correspondingly.
- the monitoring area is divided into the target outside area, the registration area, and the target area in sequence according to the travel route, that is to say, the monitored area that pedestrians first enter is the target outside area. , And then enter the target inner area through the registered area through the outside area of the target.
- the route for pedestrians leaving the surveillance area is opposite to the above entry route.
- Polling detects the human body frame, head frame and the location of the area of the pedestrian in each depth image, and binds the human body frame and head frame of the same pedestrian in the depth image, and then according to the 3D center of gravity of the head frame in each depth image
- the actual position and the predicted position of the 3D center of gravity of the pedestrian trajectory are tracked and matched, and the tracking status of each pedestrian trajectory is updated according to the obtained tracking matching results, and the 3D center of gravity of the head frame in each depth image of the current frame is located
- Update the status of the trajectory area of the pedestrian trajectory so that when the tracking and matching of the 3D center of gravity of the head frame is normal, the position information method (the position coordinate of the 3D center of gravity of the head frame) is used to update the pedestrian trajectory, and the pedestrian trajectory is updated in the head frame.
- the ReID strategy is used to perform the lost pedestrian trajectory The match is retrieved and updated.
- the 3D center of gravity point of the head frame is effectively tracked and matched by the way of taking the depth picture overhead and polling calculation, which solves the problem of tracking failure caused by occlusion in the process of cross-camera pedestrian tracking.
- the registration area it is possible to automatically register the base library feature data table for the pedestrian when the pedestrian enters the registration area, so that when the pedestrian trajectory matching based on the location coordinate fails, it will automatically switch to the ReID strategy using deep learning to match the pedestrian trajectory. Retrieval improves the accuracy and reliability of trajectory tracking results.
- the above-mentioned target area, registration area, and target area are divided by setting a 3D coordinate range boundary line on the monitoring area, and the registration area is a part of the target area.
- At least one depth camera is set above the area outside the target, the registration area and the area inside the target. It is used to collect the depth image of the captured area in real time. Through the setting of multiple depth cameras, it can collect the depth from multiple angles in real time. image.
- the human body frame, human head frame, and the location of the area of the pedestrian in each depth image are respectively detected, and the method of binding the human body frame and human head frame of the same pedestrian in the depth image includes:
- the bipartite graph maximum matching algorithm is used to screen out the human body frame and head frame belonging to the same pedestrian in each depth image to bind each other.
- each depth image of the current frame corresponds to a total of k depth images collected in the current frame, and the area of the human body frame and the area of the human head frame appearing in each depth image are sequentially calculated.
- the two pairs here include all the paired combinations of the human body frame and the human head frame in the depth image, and then for each
- the inclusion degree corresponding to the depth image uses the bipartite graph maximum matching algorithm to filter out the human body frame and the head frame belonging to the same pedestrian in each depth image to bind each other.
- the method for calculating the inclusion degree is: dividing the overlapping area of a pair of human body frame and human head frame by the area of the human head frame therein to obtain the inclusion degree of the pair of human body frame and human head frame.
- the method of using the bipartite graph maximum matching algorithm to filter out the human body frame and the human head frame belonging to the same pedestrian in each depth image based on the inclusion degree corresponding to each depth image includes:
- the RGB-D target detection technology is used to poll and detect the human body frame, head frame, and location of the pedestrian in the depth image corresponding to each depth camera, and polling to calculate the binding of the human frame and head frame in each depth image result.
- the human head is a part of the human body. Therefore, the human body frame and the human head frame of each instance of the target detection output will have a greater degree of overlap, but the prior art cannot directly output the binding of the human head frame and the human body frame. Therefore, the matching problem of the human body frame and the human head frame in the same depth image can be modeled as an assignment problem for binding.
- This embodiment uses the bipartite graph maximum matching algorithm (KM algorithm) as the modeling solution method for the assignment problem, and A new cost measurement method-inclusion degree is designed to solve the problem of calculation of assignment cost.
- the calculation of the cost metric can be expressed by the following formula:
- S h ⁇ b represent detection frames and frame head detection area of overlap
- S h is the area of the detection head frame
- D inclusion contains values represents.
- the inclusion degree set is ⁇ D body1head1 , D body1head2 , D body2head1 , D body2head2 ⁇ .
- the optimization goal of the KM algorithm is to match as many human body frames and human head frames as possible. At the same time, the sum of the inclusion degrees of the matching results should be as large as possible.
- the result assigned by the KM algorithm is that B body1 and B head2 are the body frame sum of the same pedestrian Human head frame, B body2 and B head1 are the human body frame and head frame of another pedestrian, the total cost value is 1.4.
- the coincidence degree threshold is set to filter the assignment result, which can be expressed by the following formula:
- D matchedBodyN_HeadM represents the inclusion degree of the human body frame B bodyN and the human head frame B HeadM that are paired after being assigned by the KM algorithm
- Filter_Thresh is the inclusion threshold
- the matching result M ( D matchedBodyN_HeadM ) below the threshold is judged to be 0, and the correspondence is cancelled
- the matching result M ( D matchedBodyN_HeadM ) that is higher than the threshold is judged to be 1, and the pairing relationship is maintained as a legal output.
- the human head frame and the human body frame of the same pedestrian instance in each depth image of the current frame can be bound to each other, providing reliable preprocessing input for subsequent tracking and matching.
- the method for calculating the predicted position of the 3D center of gravity of the pedestrian trajectory includes:
- Multi-dimensional modeling is performed on the spatial position of the 3D center of gravity of the pedestrian trajectory.
- the dimensional vector of the model includes ( x, y, z, h, V x , V y , V z ), where x, y, z correspond to 3D
- the three-dimensional coordinates of the center of gravity, V x , V y , and V z correspond to the movement speed of the 3D center of gravity in the coordinate direction of the corresponding dimension
- h represents the height of the pedestrian to which the 3D center of gravity belongs.
- this embodiment uses a single-camera polling method to update the multi-target tracking trajectory status, which can simply and effectively solve the problem of human body deduplication in overlapping regions in cross-camera tracking.
- the depth image of the single camera is a two-dimensional image
- the internal and external parameters of the pre-calibrated depth camera are used to obtain the coordinate system conversion formula, and the points in the two-dimensional image coordinates RGB-D are converted into three-dimensional coordinate points.
- the coordinates of the 3D center of gravity can be obtained by projecting the average center of gravity in the depth map of the head frame to a three-dimensional coordinate system.
- a 3D Kalman filter is used to model the pedestrian trajectory of the human head point movement in space, and a 6-dimensional space position state vector ( x, y, z, h, V x , V y , V z ) is used to Describe the pedestrian trajectory, x, y, z represent the three dimensions of the space coordinates of the 3D center of gravity of the pedestrian's head, h represents the height of the pedestrian, and V x , V y , and V z represent the speed of the pedestrian in the corresponding dimension.
- the predicted position of the pedestrian in the current frame can be obtained through the 3D Kalman filter using the following formula:
- variable with the estimate subscript represents the predicted output of the pedestrian position of the 3D Kalman filter in the current frame
- x, y, z and V x , V y , and V z are the state parameters of the 3D Kalman filter
- t represents the time used for two adjacent frames.
- the method of tracking and matching the predicted position of the 3D center of gravity of each pedestrian trajectory with the actual position of the 3D center of gravity of the corresponding head frame of each depth image, and updating the trajectory tracking status of each pedestrian trajectory according to the tracking matching result includes:
- the Kalman filter tracking algorithm is used to track the actual position of the 3D center of gravity of the corresponding head frame in each depth image of the current frame to obtain the actual position of the 3D center of gravity;
- the bipartite graph maximum matching algorithm is used based on the cost matrix to filter out the initial pairing of each pedestrian trajectory with the actual position of the 3D center of gravity in each depth image in the current frame;
- the primaries whose cost metric is less than or equal to the cost threshold are screened out and considered as successful, and the primaries whose cost metric is greater than the cost threshold are screened out as unmatched;
- the unpaired primary selection pairing includes the remaining unpaired head frame 3D center of gravity points and the remaining unpaired pedestrian trajectories.
- the trajectory tracking status of is updated to the lost status;
- the trajectory tracking state is the initial state
- continuous m frames of trajectory tracking state are pedestrian trajectories in the lost state
- the trajectory tracking state of the pedestrian trajectory is updated to the deleted state, where n>0, m>0, and n and m are both integers.
- the algorithm of the 3D Kalman filter needs to update the algorithm parameters according to the coordinate position of the 3D center of gravity of the pedestrian trajectory in the previous frame to calculate the predicted position of the 3D center of gravity of the head frame paired in the next frame.
- the algorithm parameters of the 3D Kalman filter are repeatedly updated to realize the continuous prediction of the 3D center of gravity position of the head frame paired in the next frame.
- all pedestrian trajectories in each depth image of the previous frame and the 3D center of gravity of the head frame in each depth image of the current frame are assigned.
- the assigned cost metric can be Mahalanobis distance, and the 3D center of gravity of each pedestrian trajectory is used to predict The position and the actual position of the 3D center of gravity of the human head frame detected by the current frame of each depth image is calculated to calculate the cost matrix.
- the KM algorithm is used as the assignment algorithm to implement the best assignment, and the assignment result is filtered by the threshold to obtain a reliable matching result.
- the matching result of threshold filtering can further prevent the pedestrian trajectory from being mismatched by adding a pedestrian height verification mechanism, and finally update the trajectory and status of the pedestrian trajectory that has been successfully matched.
- m is 3 and n is an integer greater than or equal to 5. That is to say, when the trajectory tracking state is the initial state and the trajectory tracking state in each of the 3 consecutive depth images is the pedestrian trajectory in the lost state, the pedestrian The trajectory is regarded as noise deletion.
- the pedestrian trajectory needs to be regarded as noise deleted, and there is a trajectory Pedestrian trajectories whose regional status is in the away state need to be deleted as noise.
- the trajectory tracking status in each depth image is the same regardless of how many consecutive frames.
- the method for updating the state of the trajectory area of each pedestrian trajectory based on the location of the area where the 3D center of gravity point in each depth image is located includes:
- the method for judging that the track tracking state of the pedestrian track is the lost state includes:
- the trajectory tracking state of the pedestrian trajectory is considered to be a lost state.
- this embodiment can continuously track all detected pedestrian trajectories in the depth image, and can also adopt different processing strategies for pedestrian trajectories in different locations according to actual needs, but consider the coverage of the depth camera and Application scenario characteristics, some depth cameras may capture pedestrians outside the target area, because pedestrians outside the target area may interfere with pedestrian tracking in the target area, so this embodiment designs a set of pedestrian trajectory area status management The strategy is elaborated as follows:
- the state of the pedestrian trajectory area outside the target area is set to the initial state. Pedestrians in this state will not perform additional processing on their lost trajectories. If the lost trajectory will not be retrieved through the ReID method, the remaining unmatched areas in the area outside the target The 3D center of gravity of the head frame can create a new pedestrian trajectory;
- the status of the pedestrian trajectory area is set to the registered state.
- the pedestrian in this state will complete the registration of the base library feature data table without perception, and the pedestrian trajectory in the target area after the registration is completed Always remain in the state of entry;
- the registration area is part of the target area, and this area is only used to realize the function of the registered pedestrian base library feature data table (such as the ReID base library picture), and the pedestrian trajectory after the registration is completed
- the state of the area can be set to enter the state.
- the trajectory tracking state in this embodiment is divided into the following four states: a new state, a normal state, a lost state, and a deleted state.
- a new state When the pedestrian trajectory is initially generated, its trajectory tracking state is the new state. After the pedestrian trajectory has successfully tracked the target in m frames, the trajectory tracking state is set to the normal state. The pedestrian trajectory in the normal state cannot be the same as the human head in the current frame.
- the trajectory tracking state is set to the lost state, and the corresponding trajectory tracking state in the continuous x-frame depth image is the lost state and the trajectory area state is the registered state or the entered state, the ReID method is used to match and retrieve Lost pedestrian trajectory, if the pedestrian trajectory fails to be retrieved for a long time, or the status of the trajectory area is updated to the away state, set its trajectory tracking state to the deleted state at this time. In this state, the corresponding pedestrian trajectory and its bottom library characteristic data The table is deleted.
- a Kalman filter based on spatial position information is used for tracking.
- this embodiment uses a deep learning feature-based matching strategy to detect and match the missing pedestrian trajectories and unmatched pedestrian trajectories.
- the cosine distance is used as the cost metric.
- the KM algorithm is used to solve the assignment problem between the lost pedestrian trajectory and the unmatched pedestrian trajectory.
- the matching pedestrian trajectory is updated.
- ReID mainly relies on the feature data corresponding to the bottom library feature data table and the regional position of the pedestrian trajectory in the pedestrian trajectory data table, and the regional status of the pedestrian trajectory to achieve pedestrian tracking.
- the specific implementation plan is for those skilled in the art. As is well known, this embodiment will not repeat this description.
- the application scenarios of this embodiment are very rich, such as unmanned supermarkets, smart factories, warehouse anti-theft and loss monitoring, etc.
- the human body tracking method based on RGB-D images provided by this embodiment ensures the reliability and reliability of pedestrian trajectory tracking in the target area.
- Sustainability, at the same time, track area status management provides strong technical support for practical applications, which can reduce labor costs while improving management efficiency, and has strong application value and rich application scenarios.
- This embodiment provides a human body tracking device based on RGB-D images, including:
- the partition setting unit is used to divide the monitoring area into the target area, the registration area, and the target area in sequence according to the route of travel, and use multiple distributed depth cameras to capture real-time depth images;
- the detection frame binding unit is used to detect the human body frame, human head frame and the location of the area of the pedestrian in each depth image, and bind the human body frame and human head frame of the same pedestrian in the depth image;
- Trajectory tracking state detection unit is used to track and match the predicted position of the 3D center of gravity of each pedestrian trajectory with the actual position of the 3D center of gravity of the corresponding head frame of each depth image, and update the trajectory tracking status of each pedestrian trajectory according to the tracking matching result,
- the updated status includes new status, normal status, lost status and deleted status;
- the trajectory area status detection unit updates the trajectory area status of each pedestrian trajectory based on the location of the 3D center of gravity in each depth image.
- the updated status includes the initial status, the entry status, the registration status, and the exit status;
- the trajectory tracking unit when the corresponding trajectory tracking state of any pedestrian trajectory in the continuous x-frame depth image is the lost state and the trajectory area state is the registered state or the entered state, the ReID method is used to match and retrieve the lost pedestrian trajectory and update it accordingly , Otherwise, update the pedestrian trajectory correspondingly according to the coordinates of the 3D center of gravity of the human head frame matched by the tracking in the depth image, where x>0, and x is an integer.
- the beneficial effects of the RGB-D image-based human body tracking device provided by the embodiment of the present invention are the same as the beneficial effects of the RGB-D image-based human body tracking method provided in the first embodiment, which will not be described here. Go into details.
- This embodiment provides a computer-readable storage medium on which a computer program is stored.
- the steps of the human body tracking method based on RGB-D images are executed.
- the beneficial effects of the computer-readable storage medium provided in this embodiment are the same as those of the RGB-D image-based human body tracking method provided by the above technical solutions, and will not be repeated here.
- the above-mentioned program can be stored in a computer readable storage medium.
- the storage medium may be: ROM/RAM, magnetic disk, optical disk, memory card, etc.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Image Analysis (AREA)
- Image Processing (AREA)
Abstract
Disclosed in the present invention are a human body tracking method and device based on an RGB-D image, capable of improving the accuracy of pedestrian trajectory tracking. The method comprises: dividing a monitoring area into a target outer area, a registration area, and a target inner area; detecting a human body frame, a human head frame, and an area position of a pedestrian in each depth image, respectively, and binding the human body frame and the human head frame of a same pedestrian in the depth images; performing tracking matching on a predicted position of a 3D center-of-gravity point of each pedestrian trajectory and an actual position of a 3D center-of-gravity point of the corresponding human head frame in each depth image, and updating a trajectory tracking state of each pedestrian trajectory according to the tracking matching result; updating a trajectory area state of each pedestrian trajectory on the basis of an area position of the 3D center-of-gravity point in each depth image; and if the trajectory tracking states corresponding to any pedestrian trajectory in continuous x frames of depth images are a lost state, and the trajectory area state is a registered state or an entering state, using an ReID mode to match and find the lost pedestrian trajectory and correspondingly updating the lost pedestrian trajectory, otherwise, correspondingly updating the pedestrian trajectory according to position coordinates of the 3D center-of-gravity point of the human head frame tracked and matched in each depth image.
Description
本发明涉及人体跟踪技术领域,尤其涉及一种基于RGB-D图像的人体跟踪方法及装置。The invention relates to the technical field of human body tracking, in particular to a method and device for human body tracking based on RGB-D images.
在信息技术的强势驱动下,各行各业均产生巨大变革,智慧城市、智慧工业、智慧零售等概念应运而生。利用视觉技术将人们从繁重的重复性劳动中解放出来成为潮流,视频监控则是视觉技术应用的重要领域,在视频监控领域跨摄像头多目标跟踪技术(Multi-target
Multi-camera tracking,MTMC tracking)是一个非常重要的研究课题,该技术可广泛应用于犯罪刑侦、仓库管理、无人购物、无人驾驶等场景下,具有较高的实用价值。Driven by the strong momentum of information technology, all walks of life have undergone tremendous changes, and concepts such as smart cities, smart industries, and smart retail have emerged. The use of vision technology to liberate people from heavy repetitive work has become a trend. Video surveillance is an important field of vision technology applications. In the field of video surveillance, cross-camera multi-target tracking technology (Multi-target
Multi-camera tracking (MTMC tracking) is a very important research topic. This technology can be widely used in crime and criminal investigation, warehouse management, unmanned shopping, unmanned driving and other scenarios, and has high practical value.
跨摄像头多目标跟踪技术主要是为了解决不同摄像头间行人的持续定位和追踪确认的问题。目前较为成熟的跨摄像头多目标跟踪技术多针对开阔场景下接***行视角的目标进行跟踪,而实际的监控场景多受限于环境因素,如为了获取室内较大的拍摄角度,通常将摄像头安装成斜拍的角度,这样随之而来的是行人遮挡问题,进而会导致多视角下行人的姿态差异较大,这些问题会直接影响行人轨迹的跟踪质量,解决这些问题对跨摄像头多目标跟踪技术从学术研究走向实际生产具有重要意义。The cross-camera multi-target tracking technology is mainly to solve the problem of continuous positioning and tracking confirmation of pedestrians between different cameras. At present, the more mature cross-camera multi-target tracking technology mostly tracks targets close to parallel perspective in open scenes, but the actual monitoring scenes are mostly limited by environmental factors. For example, in order to obtain a larger indoor shooting angle, the camera is usually installed as a The angle of oblique shooting, so that the problem of pedestrian occlusion will follow, which will cause large differences in the posture of pedestrians from multiple viewing angles. These problems will directly affect the tracking quality of pedestrian trajectories. Solve these problems for cross-camera multi-target tracking technology. It is of great significance to move from academic research to actual production.
本发明的目的在于提供一种基于RGB-D图像的人体跟踪方法及装置,采用多个RGB-D深度摄像头俯拍监控区域,通过追踪行人头部的3D重心点可以避免出现人体框跟踪形变大和易遮挡的问题,提高了行人轨迹跟踪的准确性。The object of the present invention is to provide a human body tracking method and device based on RGB-D images, which adopts multiple RGB-D depth cameras to shoot the monitored area from the top, and by tracking the 3D center of gravity of the pedestrian’s head, it is possible to avoid the appearance of the larger and larger shape of the human body frame tracking. The problem of easy occlusion improves the accuracy of pedestrian trajectory tracking.
为了实现上述目的,本发明的第一方面提供一种基于RGB-D图像的人体跟踪方法,包括:In order to achieve the above objective, the first aspect of the present invention provides a human body tracking method based on RGB-D images, including:
将监控区域按照行进路线依次划分为目标外区域、注册区域和目标内区域,并利用分布的多个深度摄像头俯拍实时采集深度图像;Divide the surveillance area into the target area, the registration area and the target area in sequence according to the route of travel, and use multiple distributed depth cameras to capture real-time depth images from top-down photography;
分别检测各深度图像中行人的人体框、人头框及所处区域位置,对深度图像中同一行人的人体框和人头框互做绑定;Detect the human body frame, human head frame and the location of the area of the pedestrian in each depth image, and bind the human body frame and human head frame of the same pedestrian in the depth image;
将每个行人轨迹的3D重心点预测位置与各深度图像对应人头框的3D重心点实际位置做跟踪匹配,根据跟踪匹配结果更新每个行人轨迹的轨迹跟踪状态,更新的状态包括新建状态、正常状态、丢失状态和删除状态;The predicted position of the 3D center of gravity of each pedestrian trajectory is matched with the actual position of the 3D center of gravity of the corresponding head frame of each depth image, and the trajectory tracking status of each pedestrian trajectory is updated according to the tracking matching result. The updated status includes new status and normal Status, lost status and deleted status;
基于各深度图像中3D重心点所处区域位置更新每个行人轨迹的轨迹区域状态,更新的状态包括初始状态、进入状态、注册状态和离开状态;Update the trajectory area status of each pedestrian trajectory based on the location of the 3D center of gravity in each depth image. The updated status includes the initial state, the entry state, the registration state, and the exit state;
当任一行人轨迹在连续x帧深度图像中对应的轨迹跟踪状态均为丢失状态且轨迹区域状态为注册状态或进入状态时,采用ReID方式匹配找回丢失的行人轨迹并对应更新,否则根据深度图像中跟踪匹配到的人头框的3D重心点位置坐标对应更新行人轨迹,其中x>0,且x为整数。When the tracking state of any pedestrian trajectory in the continuous x-frame depth image is lost, and the state of the trajectory area is registered or entered, the ReID method is used to match and retrieve the lost pedestrian trajectory and update accordingly, otherwise according to the depth The position coordinates of the 3D center of gravity of the head frame matched by the tracking in the image are updated correspondingly to the pedestrian trajectory, where x>0, and x is an integer.
优选地,分别检测各深度图像中行人的人体框、人头框及所处区域位置,对深度图像中同一行人的人体框和人头框互做绑定的方法包括:Preferably, detecting the human body frame, the human head frame, and the location of the area of the pedestrian in each depth image respectively, and the method of binding the human body frame and the human head frame of the same pedestrian in the depth image to each other includes:
轮询当前帧对应的各深度图像,采用 RGB-D目标检测方法获取各深度图像中行人的人体框、人头框及行人所处的区域位置;Polling each depth image corresponding to the current frame, using the RGB-D target detection method to obtain the pedestrian's human body frame, head frame, and the location of the area where the pedestrian is located in each depth image;
轮询各深度图像中出现的人体框面积和人头框面积,遍历每一对人体框和人头框的包含度;Polling the human body frame area and human head frame area appearing in each depth image, and traverse the inclusion degree of each pair of human body frame and human head frame;
基于每个深度图像对应的包含度采用二分图最大匹配算法筛选出每个深度图像中属于同一行人的人体框和人头框互做绑定。Based on the inclusion degree corresponding to each depth image, the bipartite graph maximum matching algorithm is used to screen out the human body frame and head frame belonging to the same pedestrian in each depth image to bind each other.
较佳地,基于每个深度图像对应的包含度采用二分图最大匹配算法筛选出每个深度图像中属于同一行人的人体框和人头框互做绑定的方法包括:Preferably, the method of using the bipartite graph maximum matching algorithm to filter out the human body frame and the human head frame belonging to the same pedestrian in each depth image based on the inclusion degree corresponding to each depth image includes:
根据各深度图像对应的包含度大小,利用二分图最大匹配算法筛选出每个深度图像中的人体框和人头框做初始配对;According to the degree of inclusion corresponding to each depth image, use the bipartite graph maximum matching algorithm to filter out the human body frame and the human head frame in each depth image for initial pairing;
将各深度图像对应初始配对中的包含度分别与重合度阈值比较,将包含度大于或等于重合度阈值的初始配对筛选出来做绑定确认,将包含度小于重合度阈值的初始配对筛选出来做绑定解除。Compare the inclusion degree in the initial pairings corresponding to each depth image with the coincidence degree threshold, and screen out the initial pairs whose inclusion degree is greater than or equal to the coincidence degree threshold for binding confirmation, and screen out the initial pairs whose inclusion degree is less than the coincidence degree threshold for binding confirmation. The binding is released.
较佳地,计算行人轨迹3D重心点预测位置的方法包括:Preferably, the method for calculating the predicted position of the 3D center of gravity of the pedestrian trajectory includes:
将每个深度图像进行三维坐标转换,并计算深度图像中人头框的3D重心点;Transform each depth image into three-dimensional coordinates, and calculate the 3D center of gravity of the human head frame in the depth image;
对行人轨迹3D重心点的空间位置进行多维建模,所述模型的维度向量包括(
x,y,z,h,V
x,V
y,V
z
),其中,
x,y,z对应表示3D重心点的三维坐标,
V
x,V
y,V
z
对应表示3D重心点在对应维度坐标方向上的运动速度,
h表示3D重心点所属行人的身高;
Multi-dimensional modeling is performed on the spatial position of the 3D center of gravity of the pedestrian trajectory. The dimensional vector of the model includes ( x, y, z, h, V x , V y , V z ), where x, y, z correspond to 3D The three-dimensional coordinates of the center of gravity point, V x , V y , and V z correspondingly represent the movement speed of the 3D center of gravity point in the coordinate direction of the corresponding dimension, and h represents the height of the pedestrian to which the 3D center of gravity point belongs;
基于行人轨迹3D重心点当前的
x轴坐标、
y轴坐标、
z轴坐标及对应在
x轴方向上的运动速度
V
x,
、
y轴方向上的运动速度
V
x
、
z轴方向上的运动速度
z,分别计算当前行人轨迹3D重心点在下一帧深度图像中所处
x轴方向上的预测位置、所处
y轴方向上的预测位置以及所处
z轴方向上的预测位置。
Based on the current x- axis coordinates, y- axis coordinates, z- axis coordinates of the 3D center of gravity point of the pedestrian trajectory , and the corresponding movement speed in the x- axis direction V x , the movement speed in the y- axis direction V x , the movement speed in the z- axis direction z Calculate the predicted position in the x- axis direction, the predicted position in the y- axis direction, and the predicted position in the z- axis direction of the current pedestrian trajectory 3D center of gravity in the next frame of depth image.
较佳地,将每个行人轨迹的3D重心点预测位置与各深度图像对应人头框的3D重心点实际位置做跟踪匹配,根据跟踪匹配结果更新每个行人轨迹的轨迹跟踪状态的方法包括:Preferably, the method of tracking and matching the predicted position of the 3D center of gravity of each pedestrian trajectory with the actual position of the 3D center of gravity of the corresponding head frame of each depth image, and updating the trajectory tracking status of each pedestrian trajectory according to the tracking matching result includes:
采用卡尔曼滤波器的跟踪算法对当前帧各深度图像中对应的人头框3D重心点的实际位置进行追踪,获取3D重心点的实际位置;The Kalman filter tracking algorithm is used to track the actual position of the 3D center of gravity of the corresponding head frame in each depth image of the current frame to obtain the actual position of the 3D center of gravity;
遍历计算当前帧各深度图像中每个3D重心点的实际位置与每个行人轨迹3D重心点预测位置的代价度量,得到代价矩阵;Traverse and calculate the cost metric of the actual position of each 3D center of gravity in each depth image of the current frame and the predicted position of each pedestrian trajectory 3D center of gravity to obtain a cost matrix;
在当前帧各深度图像遍历计算完成后,基于代价矩阵采用二分图最大匹配算法筛选出各行人轨迹与当前帧每个深度图像中3D重心点实际位置的初选配对;After the traversal calculation of each depth image in the current frame is completed, the bipartite graph maximum matching algorithm is used based on the cost matrix to filter out the initial pairing of each pedestrian trajectory with the actual position of the 3D center of gravity in each depth image in the current frame;
筛选出代价度量小于或等于代价阈值的初选配对认为配对成功,筛选出代价度量大于代价阈值的初选配对认为未配对成功;The primaries whose cost metric is less than or equal to the cost threshold are screened out and considered as successful, and the primaries whose cost metric is greater than the cost threshold are screened out as unmatched;
所述未配对成功的初选配对包括剩余未配对的人头框3D重心点和剩余未配对的行人轨迹,对于当前帧各深度图像中剩余未配对且处于目标外区域的人头框3D重心点,新建一行人轨迹并将轨迹跟踪状态更新为新建状态,同时将新建行人轨迹的轨迹区域状态更新为初始状态,和/或,对于当前帧各深度图像中剩余未配对的行人轨迹,将所述行人轨迹的轨迹跟踪状态更新为丢失状态; The unpaired primary selection pairing includes the remaining unpaired head frame 3D center of gravity points and the remaining unpaired pedestrian trajectories. For the remaining unpaired head frame 3D center of gravity points in the area outside the target in each depth image of the current frame, create a new Pedestrian trajectory and update the trajectory tracking status to the new state, and at the same time update the trajectory area status of the new pedestrian trajectory to the initial state, and/or, for the remaining unpaired pedestrian trajectories in each depth image of the current frame, change the pedestrian trajectory The trajectory tracking status of is updated to the lost status;
对配对成功的初选配对阈值过滤后,将其中配对的行人轨迹的轨迹跟踪状态更新为正常状态,同时将配对的人头框3D重心点的实际位置更新为当前行人轨迹的3D重心点位置;After filtering the pairing threshold for the initial selection of a successful pairing, update the trajectory tracking status of the paired pedestrian trajectory to the normal state, and at the same time update the actual position of the 3D center of gravity of the paired head frame to the 3D center of gravity of the current pedestrian trajectory;
对于配对的人头框3D重心点处于目标外区域且连续n帧轨迹跟踪状态均为丢失状态的行人轨迹,和/或,轨迹区域状态为离开状态的行人轨迹,和/或,轨迹跟踪状态为初始状态且连续m帧轨迹跟踪状态均为丢失状态的行人轨迹,将所述行人轨迹的轨迹跟踪状态更新为删除状态,其中, n>0,m>0,且n和m均为整数。For the pedestrian trajectory whose 3D center of gravity of the paired head frame is in the area outside the target and the trajectory tracking state for consecutive n frames is the lost state, and/or the trajectory area state is the pedestrian trajectory in the off state, and/or the trajectory tracking state is the initial state State and continuous m frames of trajectory tracking state are pedestrian trajectories in the lost state, and the trajectory tracking state of the pedestrian trajectory is updated to the deleted state, where n>0, m>0, and n and m are both integers.
进一步地,还包括:Further, it also includes:
当行人轨迹对应的轨迹跟踪状态为删除状态时,删除该行人轨迹及其对应的底库特征数据表;When the trajectory tracking state corresponding to the pedestrian trajectory is the deleted state, delete the pedestrian trajectory and its corresponding bottom library feature data table;
优选地,还包括:Preferably, it also includes:
基于各深度图像中3D重心点所处的区域位置更新每个行人轨迹的轨迹区域状态的方法包括:The method of updating the trajectory area status of each pedestrian trajectory based on the regional position of the 3D center of gravity in each depth image includes:
遍历当前帧各深度图像中的人头框3D重心点,识别出现在目标外区域的人头框3D重心点,将其对应行人轨迹的轨迹区域状态置为初始状态;Traverse the 3D center of gravity of the head frame in each depth image of the current frame, identify the 3D center of gravity of the head frame that appears in the area outside the target, and set the state of the trajectory area corresponding to the pedestrian trajectory to the initial state;
遍历当前帧各深度图像中的人头框3D重心点,识别出现在注册区域的人头框3D重心点,将其对应行人轨迹的轨迹区域状态置为注册状态,注册并实时更新底库特征数据表;Traverse the 3D center of gravity of the head frame in each depth image of the current frame, identify the 3D center of gravity of the head frame that appears in the registered area, set the status of the trajectory area corresponding to the pedestrian trajectory to the registered state, register and update the base library feature data table in real time;
遍历当前帧各深度图像中的人头框3D重心点,识别出现在目标内区域的人头框3D重心点,将其对应行人轨迹的轨迹区域状态置为进入状态;Traverse the 3D center of gravity of the head frame in each depth image of the current frame, identify the 3D center of gravity of the head frame that appears in the target area, and set the state of the trajectory area corresponding to the pedestrian trajectory to the entering state;
遍历当前帧各深度图像中的人头框3D重心点,识别离开目标内区域进入目标外区域的人头框3D重心点,将其对应行人轨迹的轨迹区域状态置为离开状态。Traverse the 3D center of gravity of the head frame in each depth image of the current frame, identify the 3D center of gravity of the head frame that leaves the area inside the target and enter the area outside the target, and set the state of the trajectory area corresponding to the pedestrian trajectory to the away state.
优选地,判断行人轨迹的轨迹跟踪状态为丢失状态的方法包括:Preferably, the method for judging that the track tracking state of the pedestrian track is the lost state includes:
识别当前帧各深度图像中的人头框3D重心点,若行人轨迹不能与任一个深度图像中的人头框3D重心点相匹配,则认为所述行人轨迹的轨迹跟踪状态为丢失状态。Identify the 3D center of gravity of the head frame in each depth image of the current frame, and if the pedestrian trajectory cannot match the 3D center of gravity of the head frame in any depth image, the trajectory tracking state of the pedestrian trajectory is considered to be a lost state.
本发明的第二方面提供一种基于RGB-D图像的人体跟踪装置,应用于上述技术方案所述的基于RGB-D图像的人体跟踪方法中,所述装置包括:The second aspect of the present invention provides a human body tracking device based on RGB-D images, which is applied to the RGB-D image-based human body tracking method described in the above technical solution, and the device includes:
分区设置单元,用于将监控区域按照行进路线依次划分为目标外区域、注册区域和目标内区域,并利用分布的多个深度摄像头俯拍实时采集深度图像;The partition setting unit is used to divide the monitoring area into the target area, the registration area, and the target area in sequence according to the route of travel, and use multiple distributed depth cameras to capture real-time depth images;
检测框绑定单元,用于分别检测各深度图像中行人的人体框、人头框及所处区域位置,对深度图像中同一行人的人体框和人头框互做绑定;The detection frame binding unit is used to detect the human body frame, human head frame and the location of the area of the pedestrian in each depth image, and bind the human body frame and human head frame of the same pedestrian in the depth image;
轨迹跟踪状态检测单元,用于将每个行人轨迹的3D重心点预测位置与各深度图像对应人头框的3D重心点实际位置做跟踪匹配,根据跟踪匹配结果更新每个行人轨迹的轨迹跟踪状态,更新的状态包括新建状态、正常状态、丢失状态和删除状态;Trajectory tracking state detection unit is used to track and match the predicted position of the 3D center of gravity of each pedestrian trajectory with the actual position of the 3D center of gravity of the corresponding head frame of each depth image, and update the trajectory tracking status of each pedestrian trajectory according to the tracking matching result, The updated status includes new status, normal status, lost status and deleted status;
轨迹区域状态检测单元,基于各深度图像中3D重心点所处区域位置更新每个行人轨迹的轨迹区域状态,更新的状态包括初始状态、进入状态、注册状态和离开状态;The trajectory area status detection unit updates the trajectory area status of each pedestrian trajectory based on the location of the 3D center of gravity in each depth image. The updated status includes the initial status, the entry status, the registration status, and the exit status;
轨迹追踪单元,当任一行人轨迹在连续x帧深度图像中对应的轨迹跟踪状态均为丢失状态且轨迹区域状态为注册状态或进入状态时,采用ReID方式匹配找回丢失的行人轨迹并对应更新,否则根据深度图像中跟踪匹配到的人头框的3D重心点位置坐标对应更新行人轨迹,其中x>0,且x为整数。The trajectory tracking unit, when the corresponding trajectory tracking state of any pedestrian trajectory in the continuous x-frame depth image is the lost state and the trajectory area state is the registered state or the entered state, the ReID method is used to match and retrieve the lost pedestrian trajectory and update it accordingly , Otherwise, update the pedestrian trajectory correspondingly according to the coordinates of the 3D center of gravity of the human head frame matched by the tracking in the depth image, where x>0, and x is an integer.
本发明的第三方面提供一种计算机可读存储介质,计算机可读存储介质上存储有计算机程序,计算机程序被处理器运行时执行上述基于RGB-D图像的人体跟踪方法的步骤。A third aspect of the present invention provides a computer-readable storage medium on which a computer program is stored. When the computer program is run by a processor, the steps of the human body tracking method based on RGB-D images are executed.
与现有技术相比,本发明具有以下有益效果:Compared with the prior art, the present invention has the following beneficial effects:
本发明提供的基于RGB-D图像的人体跟踪方法中,将监控区域按照行进路线依次划分为目标外区域、注册区域和目标内区域,也就是说行人最先进入的监控区域为目标外区域,然后通过目标外区域经过注册区域进入到目标内区域,行人离开监控区域的路线与上述进入路线相反,监控区域上分布有多个俯拍的深度摄像头用于实时采集各区域的深度图像,通过轮询检测各深度图像中行人的人体框、人头框及所处区域位置,并对深度图像中同一行人的人体框和人头框互做绑定,接着根据各深度图像中人头框3D重心点的实际位置与行人轨迹的3D重心点预测位置做跟踪匹配,并根据得到的跟踪匹配结果对每个行人轨迹的轨迹跟踪状态进行更新,以及根据当前帧各深度图像中人头框的3D重心点所处的区域位置对行人轨迹的轨迹区域状态进行更新,从而在人头框的3D重心点追踪匹配正常时,采用位置信息法(人头框3D重心点的位置坐标)对行人轨迹进行更新,而在人头框的3D重心点追踪匹配失败时,也即行人轨迹在连续x帧深度图像中对应的轨迹跟踪状态均为丢失状态且轨迹区域状态为注册状态或进入状态时,采用ReID策略对丢失的行人轨迹进行匹配找回并更新。In the human body tracking method based on RGB-D images provided by the present invention, the monitoring area is divided into the target outside area, the registration area and the target area in sequence according to the travel route, that is to say, the monitoring area that pedestrians first enter is the target outside area. Then enter the target area through the registered area through the outside area of the target. The pedestrian leaving the surveillance area is the opposite of the above entry route. There are multiple overhead depth cameras distributed on the surveillance area to collect depth images of each area in real time. Detect the human body frame, head frame, and location of the pedestrian in each depth image, and bind the human body frame and head frame of the same pedestrian in the depth image to each other, and then according to the actual 3D center of gravity of the head frame in each depth image The position is matched with the predicted position of the 3D center of gravity of the pedestrian trajectory, and the tracking status of each pedestrian trajectory is updated according to the obtained tracking matching result, and according to the 3D center of gravity of the head frame in each depth image of the current frame. The location of the area updates the status of the trajectory area of the pedestrian trajectory, so that when the tracking and matching of the 3D center of gravity of the head frame is normal, the position information method (the position coordinate of the 3D center of gravity of the head frame) is used to update the pedestrian trajectory. When the 3D center-of-gravity point tracking and matching fails, that is, when the corresponding trajectory tracking status of the pedestrian trajectory in the continuous x-frame depth image is lost, and the trajectory area status is registered or entered, the ReID strategy is used to match the lost pedestrian trajectory Retrieve and update.
可见,本发明采用俯拍深度图片并轮询计算的方式对头部框的3D重心点进行有效的跟踪匹配,解决了跨摄像头行人追踪过程中因遮挡导致的追踪失败问题。另外,通过设置注册区域,能够在行人进入注册区域时自动为行人注册底库特征数据表,以在基于位置坐标匹配行人轨迹失败时,自动切换至采用了深度学习的ReID策略进行行人轨迹的匹配找回,提高了轨迹跟踪结果的准确性和可靠性。It can be seen that, in the present invention, the 3D center of gravity point of the head frame is effectively tracked and matched by the way of taking the depth picture overhead and polling calculation, and solves the problem of tracking failure caused by occlusion in the process of cross-camera pedestrian tracking. In addition, by setting the registration area, it is possible to automatically register the base library feature data table for the pedestrian when the pedestrian enters the registration area, so that when the pedestrian trajectory matching based on the location coordinate fails, it will automatically switch to the ReID strategy using deep learning to match the pedestrian trajectory. Retrieval improves the accuracy and reliability of trajectory tracking results.
此处所说明的附图用来提供对本发明的进一步理解,构成本发明的一部分,本发明的示意性实施例及其说明用于解释本发明,并不构成对本发明的不当限定。在附图中:The drawings described here are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and the description thereof are used to explain the present invention, and do not constitute an improper limitation of the present invention. In the attached picture:
图1为本发明实施例一中基于RGB-D图像的人体跟踪方法的流程示意图;FIG. 1 is a schematic flowchart of a human body tracking method based on RGB-D images in Embodiment 1 of the present invention;
图2为图1中对当前帧各深度图像中同一行人的人体框和人头框互做绑定处理的流程示意图。Fig. 2 is a schematic flow chart of the mutual binding processing of the human body frame and the human head frame of the same pedestrian in each depth image of the current frame in Fig. 1.
为使本发明的上述目的、特征和优点能够更加明显易懂,下面将结合本发明实施例中的附图,对本发明实施例中的技术方案进行清楚、完整地描述。显然,所描述的实施例仅仅是本发明一部分实施例,而不是全部的实施例。基于本发明中的实施例,本领域普通技术人员在没有作出创造性劳动的前提下所获得的所有其它实施例,均属于本发明保护的范围。In order to make the foregoing objectives, features, and advantages of the present invention more obvious and understandable, the technical solutions in the embodiments of the present invention will be described clearly and completely in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the protection scope of the present invention.
实施例一Example one
请参阅图1,本实施例提供一种基于RGB-D图像的人体跟踪方法,包括:Please refer to FIG. 1. This embodiment provides a human body tracking method based on RGB-D images, including:
将监控区域按照行进路线依次划分为目标外区域、注册区域和目标内区域,并利用分布的多个深度摄像头俯拍实时采集深度图像;Divide the surveillance area into the target area, the registration area and the target area in sequence according to the route of travel, and use multiple distributed depth cameras to capture real-time depth images from top-down photography;
分别检测各深度图像中行人的人体框、人头框及所处区域位置,对深度图像中同一行人的人体框和人头框互做绑定;Detect the human body frame, human head frame and the location of the area of the pedestrian in each depth image, and bind the human body frame and human head frame of the same pedestrian in the depth image;
将每个行人轨迹的3D重心点预测位置与各深度图像对应人头框的3D重心点实际位置做跟踪匹配,根据跟踪匹配结果更新每个行人轨迹的轨迹跟踪状态,更新的状态包括新建状态、正常状态、丢失状态和删除状态;The predicted position of the 3D center of gravity of each pedestrian trajectory is matched with the actual position of the 3D center of gravity of the corresponding head frame of each depth image, and the trajectory tracking status of each pedestrian trajectory is updated according to the tracking matching result. The updated status includes new status and normal Status, lost status and deleted status;
基于各深度图像中3D重心点所处区域位置更新每个行人轨迹的轨迹区域状态,更新的状态包括初始状态、进入状态、注册状态和离开状态;Update the trajectory area status of each pedestrian trajectory based on the location of the 3D center of gravity in each depth image. The updated status includes the initial state, the entry state, the registration state, and the exit state;
当任一行人轨迹对应的轨迹跟踪状态为丢失状态且轨迹区域状态为注册状态时,采用ReID方式匹配找回丢失的行人轨迹并对应更新,否则根据深度图像中跟踪匹配到的人头框的3D重心点位置坐标对应更新行人轨迹。When the tracking status of any pedestrian trajectory is lost and the status of the trajectory area is registered, the lost pedestrian trajectory is retrieved and updated according to the ReID method. Otherwise, the 3D center of gravity of the matched head frame is tracked in the depth image. The point position coordinates update the pedestrian trajectory correspondingly.
本实施例提供的基于RGB-D图像的人体跟踪方法中,将监控区域按照行进路线依次划分为目标外区域、注册区域和目标内区域,也就是说行人最先进入的监控区域为目标外区域,然后通过目标外区域经过注册区域进入到目标内区域,行人离开监控区域的路线与上述进入路线相反,监控区域上分布有多个俯拍的深度摄像头用于实时采集各区域的深度图像,通过轮询检测各深度图像中行人的人体框、人头框及所处区域位置,并对深度图像中同一行人的人体框和人头框互做绑定,接着根据各深度图像中人头框3D重心点的实际位置与行人轨迹的3D重心点预测位置做跟踪匹配,并根据得到的跟踪匹配结果对每个行人轨迹的轨迹跟踪状态进行更新,以及根据当前帧各深度图像中人头框的3D重心点所处的区域位置对行人轨迹的轨迹区域状态进行更新,从而在人头框的3D重心点追踪匹配正常时,采用位置信息法(人头框3D重心点的位置坐标)对行人轨迹进行更新,而在人头框的3D重心点追踪匹配失败时,也即行人轨迹在连续x帧深度图像中对应的轨迹跟踪状态均为丢失状态且轨迹区域状态为注册状态或进入状态时,采用ReID策略对丢失的行人轨迹进行匹配找回并更新。In the human body tracking method based on RGB-D images provided in this embodiment, the monitoring area is divided into the target outside area, the registration area, and the target area in sequence according to the travel route, that is to say, the monitored area that pedestrians first enter is the target outside area. , And then enter the target inner area through the registered area through the outside area of the target. The route for pedestrians leaving the surveillance area is opposite to the above entry route. There are multiple overhead depth cameras distributed on the surveillance area to collect depth images of each area in real time. Polling detects the human body frame, head frame and the location of the area of the pedestrian in each depth image, and binds the human body frame and head frame of the same pedestrian in the depth image, and then according to the 3D center of gravity of the head frame in each depth image The actual position and the predicted position of the 3D center of gravity of the pedestrian trajectory are tracked and matched, and the tracking status of each pedestrian trajectory is updated according to the obtained tracking matching results, and the 3D center of gravity of the head frame in each depth image of the current frame is located Update the status of the trajectory area of the pedestrian trajectory, so that when the tracking and matching of the 3D center of gravity of the head frame is normal, the position information method (the position coordinate of the 3D center of gravity of the head frame) is used to update the pedestrian trajectory, and the pedestrian trajectory is updated in the head frame. When the 3D center-of-gravity point tracking matching fails, that is, when the corresponding trajectory tracking state of the pedestrian trajectory in the continuous x-frame depth image is the lost state and the trajectory area state is the registered state or the entered state, the ReID strategy is used to perform the lost pedestrian trajectory The match is retrieved and updated.
可见,本实施例采用俯拍深度图片并轮询计算的方式对头部框的3D重心点进行有效的跟踪匹配,解决了跨摄像头行人追踪过程中因遮挡导致的追踪失败问题。另外,通过设置注册区域,能够在行人进入注册区域时自动为行人注册底库特征数据表,以在基于位置坐标匹配行人轨迹失败时,自动切换至采用了深度学习的ReID策略进行行人轨迹的匹配找回,提高了轨迹跟踪结果的准确性和可靠性。It can be seen that, in this embodiment, the 3D center of gravity point of the head frame is effectively tracked and matched by the way of taking the depth picture overhead and polling calculation, which solves the problem of tracking failure caused by occlusion in the process of cross-camera pedestrian tracking. In addition, by setting the registration area, it is possible to automatically register the base library feature data table for the pedestrian when the pedestrian enters the registration area, so that when the pedestrian trajectory matching based on the location coordinate fails, it will automatically switch to the ReID strategy using deep learning to match the pedestrian trajectory. Retrieval improves the accuracy and reliability of trajectory tracking results.
具体实施时,上述目标外区域、注册区域和目标内区域通过在监控区域上设置3D坐标范围分界线进行划分,且注册区域属于目标内区域的一部分。目标外区域、注册区域和目标内区域的上方均设置有至少一台俯拍的深度摄像头,用于实时采集所拍区域的深度图像,通过多台深度摄像头的设置,能够实时采集多角度的深度图像。In specific implementation, the above-mentioned target area, registration area, and target area are divided by setting a 3D coordinate range boundary line on the monitoring area, and the registration area is a part of the target area. At least one depth camera is set above the area outside the target, the registration area and the area inside the target. It is used to collect the depth image of the captured area in real time. Through the setting of multiple depth cameras, it can collect the depth from multiple angles in real time. image.
请参阅图2,上述实施例中分别检测各深度图像中行人的人体框、人头框及所处区域位置,对深度图像中同一行人的人体框和人头框互做绑定的方法包括:Referring to FIG. 2, in the above embodiment, the human body frame, human head frame, and the location of the area of the pedestrian in each depth image are respectively detected, and the method of binding the human body frame and human head frame of the same pedestrian in the depth image includes:
轮询当前帧对应的各深度图像,采用 RGB-D目标检测方法获取各深度图像中行人的人体框、人头框及行人所处的区域位置;Polling each depth image corresponding to the current frame, using the RGB-D target detection method to obtain the pedestrian's human body frame, head frame, and the location of the area where the pedestrian is located in each depth image;
轮询各深度图像中出现的人体框面积和人头框面积,遍历每一对人体框和人头框的包含度;Polling the human body frame area and human head frame area appearing in each depth image, and traverse the inclusion degree of each pair of human body frame and human head frame;
基于每个深度图像对应的包含度采用二分图最大匹配算法筛选出每个深度图像中属于同一行人的人体框和人头框互做绑定。Based on the inclusion degree corresponding to each depth image, the bipartite graph maximum matching algorithm is used to screen out the human body frame and head frame belonging to the same pedestrian in each depth image to bind each other.
可以理解的是,假设一共设置有k台深度摄像头,那么当前帧各深度图像就对应为当前帧一共采集的k个深度图像,通过依次计算每个深度图像中出现的人体框面积和人头框面积,并分别计算每个深度图像中两两成对的人体框和人头框的包含度,此处的两两成对包括了深度图像中人体框和人头框的全部成对组合,然后针对每个深度图像对应的包含度采用二分图最大匹配算法筛选出每个深度图像中属于同一行人的人体框和人头框互做绑定。其中,包含度的计算方法为:将一对人体框和人头框中的重叠面积除以其中的人头框面积,得到该对人体框和人头框的包含度。It is understandable that, assuming that there are a total of k depth cameras, each depth image of the current frame corresponds to a total of k depth images collected in the current frame, and the area of the human body frame and the area of the human head frame appearing in each depth image are sequentially calculated. , And calculate the inclusion degree of the human body frame and the human head frame in each depth image. The two pairs here include all the paired combinations of the human body frame and the human head frame in the depth image, and then for each The inclusion degree corresponding to the depth image uses the bipartite graph maximum matching algorithm to filter out the human body frame and the head frame belonging to the same pedestrian in each depth image to bind each other. The method for calculating the inclusion degree is: dividing the overlapping area of a pair of human body frame and human head frame by the area of the human head frame therein to obtain the inclusion degree of the pair of human body frame and human head frame.
具体地,基于每个深度图像对应的包含度采用二分图最大匹配算法筛选出每个深度图像中属于同一行人的人体框和人头框互做绑定的方法包括:Specifically, the method of using the bipartite graph maximum matching algorithm to filter out the human body frame and the human head frame belonging to the same pedestrian in each depth image based on the inclusion degree corresponding to each depth image includes:
根据各深度图像对应的包含度大小,利用二分图最大匹配算法筛选出每个深度图像中的人体框和人头框做初始配对;According to the degree of inclusion corresponding to each depth image, use the bipartite graph maximum matching algorithm to filter out the human body frame and the human head frame in each depth image for initial pairing;
将各深度图像对应初始配对中的包含度分别与重合度阈值比较,将包含度大于或等于重合度阈值的初始配对筛选出来做绑定确认,将包含度小于重合度阈值的初始配对筛选出来做绑定解除。Compare the inclusion degree in the initial pairings corresponding to each depth image with the coincidence degree threshold, and screen out the initial pairs whose inclusion degree is greater than or equal to the coincidence degree threshold for binding confirmation, and screen out the initial pairs whose inclusion degree is less than the coincidence degree threshold for binding confirmation. The binding is released.
具体实施时,采用RGB-D目标检测技术轮询检测各深度摄像头对应深度图像中行人的人体框、人头框及所处区域位置,并轮询计算各深度图像中人体框和人头框的绑定结果。根据先验知识可知,人头是人体的一部分,因此目标检测输出每个实例的人体框和人头框会具有较大的重叠度,但现有技术并不能直接输出实例的人头框和人体框的绑定结果,因此同一深度图像中的人体框和人头框的匹配问题可建模为指派问题进行绑定,本实施例采用二分图最大匹配算法(KM算法)作为指派问题的建模求解方法,并设计了一种新的代价度量方法-包含度来解决指派代价计算问题。该代价度量的计算可用如下公式表述:In specific implementation, the RGB-D target detection technology is used to poll and detect the human body frame, head frame, and location of the pedestrian in the depth image corresponding to each depth camera, and polling to calculate the binding of the human frame and head frame in each depth image result. According to prior knowledge, the human head is a part of the human body. Therefore, the human body frame and the human head frame of each instance of the target detection output will have a greater degree of overlap, but the prior art cannot directly output the binding of the human head frame and the human body frame. Therefore, the matching problem of the human body frame and the human head frame in the same depth image can be modeled as an assignment problem for binding. This embodiment uses the bipartite graph maximum matching algorithm (KM algorithm) as the modeling solution method for the assignment problem, and A new cost measurement method-inclusion degree is designed to solve the problem of calculation of assignment cost. The calculation of the cost metric can be expressed by the following formula:
D
inclusion=S
h
D inclusion =S h
∩∩
bb
/S
h
/S h
式中
S
h
∩
b 表示检测框和人头检测框的重叠面积,
S
h
为人头检测框的面积,
D
inclusion
表示包含度值。
Wherein S h ∩ b represent detection frames and frame head detection area of overlap, S h is the area of the detection head frame, D inclusion contains values represents.
举例说明:假设某一深度摄像头采集的某一深度图像中存在两个人体框{
B
body1,B
body2
}和两个人头框{
B
head1,B
head2
},总共存在4对绑定组合,遍历计算每一对人体框和人头框组合的包含度,得到包含度集合为{
D
body1head1,D
body1head2,D
body2head1,D
body2head2
},KM算法的优化目标是尽可能匹配较多的人体框和人头框,同时得到匹配结果的包含度之和应尽可能大。若假设的包含度集合{
D
body1head1,D
body1head2,D
body2head1,D
body2head2
}对应为{0.4,0.5,0.9,0.1},故KM算法指派的结果是
B
body1
和
B
head2
为同一行人的人体框和人头框,
B
body2
和
B
head1
为另一行人的人体框和人头框,总的代价值和1.4。
Example: Suppose there are two human body frames {B body1 , B body2 } and two human head frames { B head1 , B head2 } in a certain depth image collected by a certain depth camera, there are a total of 4 pairs of binding combinations, traverse calculation For the inclusion degree of each pair of human body frame and human head frame, the inclusion degree set is { D body1head1 , D body1head2 , D body2head1 , D body2head2 }. The optimization goal of the KM algorithm is to match as many human body frames and human head frames as possible. At the same time, the sum of the inclusion degrees of the matching results should be as large as possible. If the hypothetical set of inclusion degrees { D body1head1 , D body1head2 , D body2head1 , D body2head2 } corresponds to {0.4, 0.5, 0.9, 0.1}, the result assigned by the KM algorithm is that B body1 and B head2 are the body frame sum of the same pedestrian Human head frame, B body2 and B head1 are the human body frame and head frame of another pedestrian, the total cost value is 1.4.
在具体实施过程中,由于检测可能存在误检人头框或人体框的情况,因此设定重合度阈值来对指派结果进行过滤,具体可由如下公式表示: In the specific implementation process, because the detection may misdetect the human head frame or the human frame, the coincidence degree threshold is set to filter the assignment result, which can be expressed by the following formula:
上述公式中,
D
matchedBodyN_HeadM
表示经过KM算法指派后配对的人体框
B
bodyN
和人头框
B
HeadM
的包含度,
Filter_Thresh是包含度阈值,低于阈值的配对结果M(
D
matchedBodyN_HeadM
)判定为0,解除对应人体框和人头框的配对关系,高于阈值的配对结果M(
D
matchedBodyN_HeadM
)判定为1,作为合法输出保持配对关系。
In the above formula, D matchedBodyN_HeadM represents the inclusion degree of the human body frame B bodyN and the human head frame B HeadM that are paired after being assigned by the KM algorithm , Filter_Thresh is the inclusion threshold, and the matching result M ( D matchedBodyN_HeadM ) below the threshold is judged to be 0, and the correspondence is cancelled For the pairing relationship between the human body frame and the human head frame, the matching result M ( D matchedBodyN_HeadM ) that is higher than the threshold is judged to be 1, and the pairing relationship is maintained as a legal output.
通过上述绑定策略,可以将当前帧各深度图像中同一行人实例的人头框和人体框相互绑定,为后续的跟踪匹配提供可靠的预处理输入。Through the above-mentioned binding strategy, the human head frame and the human body frame of the same pedestrian instance in each depth image of the current frame can be bound to each other, providing reliable preprocessing input for subsequent tracking and matching.
上述实施例中,计算行人轨迹3D重心点预测位置的方法包括:In the foregoing embodiment, the method for calculating the predicted position of the 3D center of gravity of the pedestrian trajectory includes:
将每个深度图像进行三维坐标转换,并计算深度图像中人头框的3D重心点;Transform each depth image into three-dimensional coordinates, and calculate the 3D center of gravity of the human head frame in the depth image;
对行人轨迹3D重心点的空间位置进行多维建模,所述模型的维度向量包括(
x,y,z,h,V
x,V
y,V
z
),其中,
x,y,z对应表示3D重心点的三维坐标,
V
x,V
y,V
z
对应表示3D重心点在对应维度坐标方向上的运动速度,
h表示3D重心点所属行人的身高。
Multi-dimensional modeling is performed on the spatial position of the 3D center of gravity of the pedestrian trajectory. The dimensional vector of the model includes ( x, y, z, h, V x , V y , V z ), where x, y, z correspond to 3D The three-dimensional coordinates of the center of gravity, V x , V y , and V z correspond to the movement speed of the 3D center of gravity in the coordinate direction of the corresponding dimension, and h represents the height of the pedestrian to which the 3D center of gravity belongs.
基于行人轨迹3D重心点当前的
x轴坐标、
y轴坐标、
z轴坐标及对应在
x轴方向上的运动速度
V
x,
、
y轴方向上的运动速度
V
x
、
z轴方向上的运动速度
z,分别计算当前行人轨迹3D重心点在下一帧深度图像中所处
x轴方向上的预测位置、所处
y轴方向上的预测位置以及所处
z轴方向上的预测位置。
Based on the current x- axis coordinates, y- axis coordinates, z- axis coordinates of the 3D center of gravity point of the pedestrian trajectory , and the corresponding movement speed in the x- axis direction V x , the movement speed in the y- axis direction V x , the movement speed in the z- axis direction z Calculate the predicted position in the x- axis direction, the predicted position in the y- axis direction, and the predicted position in the z- axis direction of the current pedestrian trajectory 3D center of gravity in the next frame of depth image.
具体实施时,本实施例采用单摄像头轮询的方式进行多目标跟踪轨迹状态的更新,能够简单有效地解决了跨摄像头跟踪中重叠区域的人体去重问题。另外,由于单摄像头的深度图像为二维画面,采用预先标定好的深度摄像头的内参外参获取坐标系转换公式,将二维画面坐标RGB-D中的点转换为三维坐标点,人头框的3D重心点坐标可通过人头框深度图中的平均重心投影到三维坐标系获得。During specific implementation, this embodiment uses a single-camera polling method to update the multi-target tracking trajectory status, which can simply and effectively solve the problem of human body deduplication in overlapping regions in cross-camera tracking. In addition, since the depth image of the single camera is a two-dimensional image, the internal and external parameters of the pre-calibrated depth camera are used to obtain the coordinate system conversion formula, and the points in the two-dimensional image coordinates RGB-D are converted into three-dimensional coordinate points. The coordinates of the 3D center of gravity can be obtained by projecting the average center of gravity in the depth map of the head frame to a three-dimensional coordinate system.
本实施例采用3D卡尔曼滤波器对空间中人体头点运动的行人轨迹进行建模,采用一个6维空间位置状态向量(
x,y,z,h,V
x,V
y,V
z
)来描述行人轨迹,
x,y,z分别表示行人人头3D重心点空间坐标的三个维度,
h表示行人身高,
V
x,V
y,V
z
则表示行人在对应维度上的运动速度。通过3D卡尔曼滤波器利用下述公式可得到当前帧行人的预测位置:
In this embodiment, a 3D Kalman filter is used to model the pedestrian trajectory of the human head point movement in space, and a 6-dimensional space position state vector ( x, y, z, h, V x , V y , V z ) is used to Describe the pedestrian trajectory, x, y, z represent the three dimensions of the space coordinates of the 3D center of gravity of the pedestrian's head, h represents the height of the pedestrian, and V x , V y , and V z represent the speed of the pedestrian in the corresponding dimension. The predicted position of the pedestrian in the current frame can be obtained through the 3D Kalman filter using the following formula:
x
estimate=x+V
x*t,y
estimate=y+V
y*t,z
estimate=z+V
z*t
x estimate = x+V x *t,y estimate =y+V y *t,z estimate = z+V z *t
上式中,带estimate下标的变量表示3D卡尔曼滤波器在当前帧的行人位置的预测输出,
x,y,z以及
V
x,V
y,V
z
则是3D卡尔曼滤波器的状态参量,
t表示相邻两帧所用的时间。
In the above formula, the variable with the estimate subscript represents the predicted output of the pedestrian position of the 3D Kalman filter in the current frame, and x, y, z and V x , V y , and V z are the state parameters of the 3D Kalman filter. t represents the time used for two adjacent frames.
上述实施中,将每个行人轨迹的3D重心点预测位置与各深度图像对应人头框的3D重心点实际位置做跟踪匹配,根据跟踪匹配结果更新每个行人轨迹的轨迹跟踪状态的方法包括:In the above implementation, the method of tracking and matching the predicted position of the 3D center of gravity of each pedestrian trajectory with the actual position of the 3D center of gravity of the corresponding head frame of each depth image, and updating the trajectory tracking status of each pedestrian trajectory according to the tracking matching result includes:
采用卡尔曼滤波器的跟踪算法对当前帧各深度图像中对应的人头框3D重心点的实际位置进行追踪,获取3D重心点的实际位置;The Kalman filter tracking algorithm is used to track the actual position of the 3D center of gravity of the corresponding head frame in each depth image of the current frame to obtain the actual position of the 3D center of gravity;
遍历计算当前帧各深度图像中每个3D重心点的实际位置与每个行人轨迹3D重心点预测位置的代价度量,得到代价矩阵;Traverse and calculate the cost metric of the actual position of each 3D center of gravity in each depth image of the current frame and the predicted position of each pedestrian trajectory 3D center of gravity to obtain a cost matrix;
在当前帧各深度图像遍历计算完成后,基于代价矩阵采用二分图最大匹配算法筛选出各行人轨迹与当前帧每个深度图像中3D重心点实际位置的初选配对;After the traversal calculation of each depth image in the current frame is completed, the bipartite graph maximum matching algorithm is used based on the cost matrix to filter out the initial pairing of each pedestrian trajectory with the actual position of the 3D center of gravity in each depth image in the current frame;
筛选出代价度量小于或等于代价阈值的初选配对认为配对成功,筛选出代价度量大于代价阈值的初选配对认为未配对成功;The primaries whose cost metric is less than or equal to the cost threshold are screened out and considered as successful, and the primaries whose cost metric is greater than the cost threshold are screened out as unmatched;
所述未配对成功的初选配对包括剩余未配对的人头框3D重心点和剩余未配对的行人轨迹,对于当前帧各深度图像中剩余未配对且处于目标外区域的人头框3D重心点,新建一行人轨迹并将轨迹跟踪状态更新为新建状态,同时将新建行人轨迹的轨迹区域状态更新为初始状态,和/或,对于当前帧各深度图像中剩余未配对的行人轨迹,将所述行人轨迹的轨迹跟踪状态更新为丢失状态; The unpaired primary selection pairing includes the remaining unpaired head frame 3D center of gravity points and the remaining unpaired pedestrian trajectories. For the remaining unpaired head frame 3D center of gravity points in the area outside the target in each depth image of the current frame, create a new Pedestrian trajectory and update the trajectory tracking status to the new state, and at the same time update the trajectory area status of the new pedestrian trajectory to the initial state, and/or, for the remaining unpaired pedestrian trajectories in each depth image of the current frame, change the pedestrian trajectory The trajectory tracking status of is updated to the lost status;
对配对成功的初选配对阈值过滤后,将其中配对的行人轨迹的轨迹跟踪状态更新为正常状态,同时将配对的人头框3D重心点的实际位置更新为当前行人轨迹的3D重心点位置;After filtering the pairing threshold for the initial selection of a successful pairing, update the trajectory tracking status of the paired pedestrian trajectory to the normal state, and at the same time update the actual position of the 3D center of gravity of the paired head frame to the 3D center of gravity of the current pedestrian trajectory;
对于配对的人头框3D重心点处于目标外区域且连续n帧轨迹跟踪状态均为丢失状态的行人轨迹,和/或,轨迹区域状态为离开状态的行人轨迹,和/或,轨迹跟踪状态为初始状态且连续m帧轨迹跟踪状态均为丢失状态的行人轨迹,将所述行人轨迹的轨迹跟踪状态更新为删除状态,其中, n>0,m>0,且n和m均为整数。For the pedestrian trajectory whose 3D center of gravity of the paired head frame is in the area outside the target and the trajectory tracking state for consecutive n frames is the lost state, and/or the trajectory area state is the pedestrian trajectory in the off state, and/or the trajectory tracking state is the initial state State and continuous m frames of trajectory tracking state are pedestrian trajectories in the lost state, and the trajectory tracking state of the pedestrian trajectory is updated to the deleted state, where n>0, m>0, and n and m are both integers.
具体实施时,3D卡尔曼滤波器的算法需根据行人轨迹在上一帧中的3D重心点的坐标位置更新算法参数,以计算下一帧配对的人头框3D重心点的预测位置。以此循环,反复更新3D卡尔曼滤波器的算法参数,实现持续对下一帧配对的人头框3D重心点位置的预测。In specific implementation, the algorithm of the 3D Kalman filter needs to update the algorithm parameters according to the coordinate position of the 3D center of gravity of the pedestrian trajectory in the previous frame to calculate the predicted position of the 3D center of gravity of the head frame paired in the next frame. In this cycle, the algorithm parameters of the 3D Kalman filter are repeatedly updated to realize the continuous prediction of the 3D center of gravity position of the head frame paired in the next frame.
本实施例对上一帧各深度图像中所有的行人轨迹与当前帧各深度图像中的人头框3D重心点进行指派,指派的代价度量可以为马氏距离,利用各行人轨迹的3D重心点预测位置和当前帧各深度图像检测的人头框3D重心点实际位置计算代价矩阵,如采用KM算法作为指派算法实施最佳指派,对指派结果采用阈值过滤后得到可靠的匹配结果,对于指派成功以及通过阈值过滤的匹配结果可通过增加行人身高校验机制进一步防止行人轨迹出现误匹配的情况,最终对匹配成功的行人轨迹进行轨迹更新和状态更新。In this embodiment, all pedestrian trajectories in each depth image of the previous frame and the 3D center of gravity of the head frame in each depth image of the current frame are assigned. The assigned cost metric can be Mahalanobis distance, and the 3D center of gravity of each pedestrian trajectory is used to predict The position and the actual position of the 3D center of gravity of the human head frame detected by the current frame of each depth image is calculated to calculate the cost matrix. For example, the KM algorithm is used as the assignment algorithm to implement the best assignment, and the assignment result is filtered by the threshold to obtain a reliable matching result. For assignment success and passing The matching result of threshold filtering can further prevent the pedestrian trajectory from being mismatched by adding a pedestrian height verification mechanism, and finally update the trajectory and status of the pedestrian trajectory that has been successfully matched.
优选地,m取3,n取大或等于5的整数,也就是说当轨迹跟踪状态为初始状态且连续3帧各深度图像中的轨迹跟踪状态均为丢失状态的行人轨迹,需将该行人轨迹视为噪音删除,当人头框3D重心点处于目标外区域且连续5帧各深度图像中的轨迹跟踪状态均为丢失状态的行人轨迹,需将该行人轨迹视为噪音删除,还有就是轨迹区域状态为离开状态的行人轨迹,需将该行人轨迹视为噪音删除,而对于其他复杂场景,如人头框3D重心点处于目标内区域,其无论连续多少帧各深度图像中的轨迹跟踪状态均为丢失状态该行人轨迹也不会被删除,而是持续进行轨迹匹配,直至匹配成功为止。这样设置主要是考虑到无人商店这种追踪场景,在目标内区域中的人不会凭空消失,而追踪失败多是因为技术问题导致,故后续还需持续追踪找回。Preferably, m is 3 and n is an integer greater than or equal to 5. That is to say, when the trajectory tracking state is the initial state and the trajectory tracking state in each of the 3 consecutive depth images is the pedestrian trajectory in the lost state, the pedestrian The trajectory is regarded as noise deletion. When the 3D center of gravity of the head frame is in the area outside the target and the trajectory tracking state in each depth image for 5 consecutive frames is a pedestrian trajectory in a lost state, the pedestrian trajectory needs to be regarded as noise deleted, and there is a trajectory Pedestrian trajectories whose regional status is in the away state need to be deleted as noise. For other complex scenes, such as the 3D center of gravity of the head frame in the target area, the trajectory tracking status in each depth image is the same regardless of how many consecutive frames. In the lost state, the pedestrian trajectory will not be deleted, but the trajectory matching will continue until the matching is successful. This setting mainly takes into account the unmanned store tracking scene, people in the target area will not disappear out of thin air, and tracking failures are mostly caused by technical problems, so follow-up tracking needs to be continued.
进一步地还包括:当行人轨迹对应的轨迹跟踪状态为删除状态时,删除该行人轨迹及其对应的底库特征数据表。It further includes: when the track tracking state corresponding to the pedestrian track is in the deleted state, deleting the pedestrian track and its corresponding bottom library feature data table.
上述实施例中基于各深度图像中3D重心点所处的区域位置更新每个行人轨迹的轨迹区域状态的方法包括:In the foregoing embodiment, the method for updating the state of the trajectory area of each pedestrian trajectory based on the location of the area where the 3D center of gravity point in each depth image is located includes:
遍历当前帧各深度图像中的人头框3D重心点,识别出现在目标外区域的人头框3D重心点,将其对应行人轨迹的轨迹区域状态置为初始状态;Traverse the 3D center of gravity of the head frame in each depth image of the current frame, identify the 3D center of gravity of the head frame that appears in the area outside the target, and set the state of the trajectory area corresponding to the pedestrian trajectory to the initial state;
遍历当前帧各深度图像中的人头框3D重心点,识别出现在注册区域的人头框3D重心点,将其对应行人轨迹的轨迹区域状态置为注册状态,注册并实时更新底库特征数据表;Traverse the 3D center of gravity of the head frame in each depth image of the current frame, identify the 3D center of gravity of the head frame that appears in the registered area, set the status of the trajectory area corresponding to the pedestrian trajectory to the registered state, register and update the base library feature data table in real time;
遍历当前帧各深度图像中的人头框3D重心点,识别出现在目标内区域的人头框3D重心点,将其对应行人轨迹的轨迹区域状态置为进入状态;Traverse the 3D center of gravity of the head frame in each depth image of the current frame, identify the 3D center of gravity of the head frame that appears in the target area, and set the state of the trajectory area corresponding to the pedestrian trajectory to the entering state;
遍历当前帧各深度图像中的人头框3D重心点,识别离开目标内区域进入目标外区域的人头框3D重心点,将其对应行人轨迹的轨迹区域状态置为离开状态。Traverse the 3D center of gravity of the head frame in each depth image of the current frame, identify the 3D center of gravity of the head frame that leaves the area inside the target and enter the area outside the target, and set the state of the trajectory area corresponding to the pedestrian trajectory to the away state.
上述实施例中判断行人轨迹的轨迹跟踪状态为丢失状态的方法包括:In the foregoing embodiment, the method for judging that the track tracking state of the pedestrian track is the lost state includes:
识别当前帧各深度图像中的人头框3D重心点,若行人轨迹不能与当前帧任一个深度图像中的人头框3D重心点相匹配,则认为所述行人轨迹的轨迹跟踪状态为丢失状态。Identify the 3D center of gravity of the head frame in each depth image of the current frame. If the pedestrian trajectory cannot match the 3D center of gravity of the head frame in any depth image of the current frame, the trajectory tracking state of the pedestrian trajectory is considered to be a lost state.
具体实施时,本实施例可对深度图像中所有检测到的行人轨迹进行持续跟踪,还可根据实际需求对处于不同区域位置的行人轨迹采用不同的处理策略,但考虑到深度摄像头的覆盖范围以及应用场景特点,部分深度摄像头可能会拍摄到目标外区域的行人,由于目标外区域的行人可能会对目标内区域的行人追踪产生一定的干扰,因此本实施例设计了一套行人轨迹区域状态管理策略,具体阐述如下:In specific implementation, this embodiment can continuously track all detected pedestrian trajectories in the depth image, and can also adopt different processing strategies for pedestrian trajectories in different locations according to actual needs, but consider the coverage of the depth camera and Application scenario characteristics, some depth cameras may capture pedestrians outside the target area, because pedestrians outside the target area may interfere with pedestrian tracking in the target area, so this embodiment designs a set of pedestrian trajectory area status management The strategy is elaborated as follows:
1、目标外区域的行人轨迹区域状态设置为初始状态,该种状态的行人不会对其丢失轨迹进行额外处理,如不会对丢失轨迹通过ReID方式找回,处于目标外区域的剩余未配的人头框3D重心点,可以新建一行人轨迹;1. The state of the pedestrian trajectory area outside the target area is set to the initial state. Pedestrians in this state will not perform additional processing on their lost trajectories. If the lost trajectory will not be retrieved through the ReID method, the remaining unmatched areas in the area outside the target The 3D center of gravity of the head frame can create a new pedestrian trajectory;
2、从目标外区域进入目标内区域的行人轨迹区域状态设置为进入状态,该种状态的行人是跟踪关注的重点;2. Set the status of the pedestrian trajectory area from the area outside the target into the area inside the target to enter the state, and pedestrians in this state are the focus of tracking;
3、行人进入到注册区域后,行人的轨迹区域状态设置为注册状态,该种状态的行人会在无感知的情况下完成底库特征数据表的注册,注册完成后在目标内区域的行人轨迹始终保持为进入状态;3. After the pedestrian enters the registration area, the status of the pedestrian trajectory area is set to the registered state. The pedestrian in this state will complete the registration of the base library feature data table without perception, and the pedestrian trajectory in the target area after the registration is completed Always remain in the state of entry;
4、从目标内区域走到目标外区域的行人,其行人轨迹的区域状态更新为离开状态,该状态下的行人轨迹将会被删除,同时对应的底库特征数据表也会被删除,以避免对其他跟踪目标产生影响;4. For pedestrians who walk from the area inside the target to the area outside the target, the area status of their pedestrian trajectory is updated to the Away state, the pedestrian trajectory in this state will be deleted, and the corresponding base library feature data table will also be deleted. Avoid affecting other tracking targets;
5、除上述正常跟踪流程外未加说明的非法行为,如行人轨迹的初始位置出现在目标内区域,可根据实际需求进行相应的报警操作。5. In addition to the above-mentioned normal tracking process, illegal behaviors that are not explained, such as the initial position of the pedestrian trajectory appear in the target area, and the corresponding alarm operation can be carried out according to actual needs.
对上述策略3需进一步说明的是,注册区域属于目标内区域的一部分,该区域仅用于实现注册行人底库特征数据表(如ReID底库图片)的功能,在完成注册行为后行人的轨迹区域状态可设置为进入状态。For the above strategy 3, it needs to be further explained that the registration area is part of the target area, and this area is only used to realize the function of the registered pedestrian base library feature data table (such as the ReID base library picture), and the pedestrian trajectory after the registration is completed The state of the area can be set to enter the state.
本实施例中的轨迹跟踪状态分为以下四种:新建状态、正常状态、丢失状态和删除状态。行人轨迹在初始生成时其轨迹跟踪状态为新建状态,行人轨迹在连续成功跟踪m帧目标后,轨迹跟踪状态置为正常状态,正常状态下的行人轨迹不能与当前帧任一个深度图像中的人头框3D重心点相匹配时,轨迹跟踪状态设置为丢失状态,在连续x帧深度图像中对应的轨迹跟踪状态均为丢失状态且轨迹区域状态为注册状态或进入状态时,采用ReID方式匹配找回丢失的行人轨迹,如果行人轨迹长时间找回失败,或轨迹区域状态更新为离开状态,此时将其轨迹跟踪状态设置为删除状态,该状态下会对相应的行人轨迹以及其底库特征数据表进行删除处理。The trajectory tracking state in this embodiment is divided into the following four states: a new state, a normal state, a lost state, and a deleted state. When the pedestrian trajectory is initially generated, its trajectory tracking state is the new state. After the pedestrian trajectory has successfully tracked the target in m frames, the trajectory tracking state is set to the normal state. The pedestrian trajectory in the normal state cannot be the same as the human head in the current frame. When the frame 3D center of gravity points match, the trajectory tracking state is set to the lost state, and the corresponding trajectory tracking state in the continuous x-frame depth image is the lost state and the trajectory area state is the registered state or the entered state, the ReID method is used to match and retrieve Lost pedestrian trajectory, if the pedestrian trajectory fails to be retrieved for a long time, or the status of the trajectory area is updated to the away state, set its trajectory tracking state to the deleted state at this time. In this state, the corresponding pedestrian trajectory and its bottom library characteristic data The table is deleted.
综上,本实施例在行人轨迹的初始跟踪阶段采用的是基于空间位置信息的卡尔曼滤波器进行跟踪,但由于人头重心点3D坐标估计的误差、漏检以及密集人群的遮挡干扰等问题,可能会导致目标内区域正常行走行人的行人轨迹丢失,该种情况下本实施例采用了基于深度学习特征的匹配策略对丢失的行人轨迹和无匹配的行人轨迹进行检测匹配,利用深度学习特征间的距离,如余弦距离作为代价度量,采用KM算法解决丢失的行人轨迹和无匹配的行人轨迹之间的指派问题,对指派结果采用阈值过滤得到可靠的匹配结果,以对丢失的行人轨迹和无匹配的行人轨迹进行更新。In summary, in the initial tracking stage of the pedestrian trajectory in this embodiment, a Kalman filter based on spatial position information is used for tracking. However, due to errors in the estimation of the 3D coordinates of the center of gravity of the human head, missed detection, and occlusion interference from dense crowds, etc. Pedestrian trajectories of normal walking pedestrians in the target area may be lost. In this case, this embodiment uses a deep learning feature-based matching strategy to detect and match the missing pedestrian trajectories and unmatched pedestrian trajectories. For example, the cosine distance is used as the cost metric. The KM algorithm is used to solve the assignment problem between the lost pedestrian trajectory and the unmatched pedestrian trajectory. The matching pedestrian trajectory is updated.
需要说明的是,ReID主要是依靠底库特征数据表对应的特征数据以及行人轨迹数据表中行人轨迹的区域位置、行人轨迹的区域状态来实现行人跟踪的,其具体实现方案为本领域技术人员所公知的,本实施例对此不做赘述。It should be noted that ReID mainly relies on the feature data corresponding to the bottom library feature data table and the regional position of the pedestrian trajectory in the pedestrian trajectory data table, and the regional status of the pedestrian trajectory to achieve pedestrian tracking. The specific implementation plan is for those skilled in the art. As is well known, this embodiment will not repeat this description.
本实施例的应用场景十分丰富,如无人超市、智慧工厂、仓库防盗损监控等,本实施例提供的基于RGB-D图像的人体跟踪方法,保证行人在目标内区域轨迹跟踪的可靠性和持续性,同时轨迹区域状态管理为实际应用提供了扩展性强的技术支撑,能够在降低人工成本的同时提升管理效率,具有较强的应用价值和丰富的应用场景。The application scenarios of this embodiment are very rich, such as unmanned supermarkets, smart factories, warehouse anti-theft and loss monitoring, etc. The human body tracking method based on RGB-D images provided by this embodiment ensures the reliability and reliability of pedestrian trajectory tracking in the target area. Sustainability, at the same time, track area status management provides strong technical support for practical applications, which can reduce labor costs while improving management efficiency, and has strong application value and rich application scenarios.
实施例二Example two
本实施例提供一种基于RGB-D图像的人体跟踪装置,包括:This embodiment provides a human body tracking device based on RGB-D images, including:
分区设置单元,用于将监控区域按照行进路线依次划分为目标外区域、注册区域和目标内区域,并利用分布的多个深度摄像头俯拍实时采集深度图像;The partition setting unit is used to divide the monitoring area into the target area, the registration area, and the target area in sequence according to the route of travel, and use multiple distributed depth cameras to capture real-time depth images;
检测框绑定单元,用于分别检测各深度图像中行人的人体框、人头框及所处区域位置,对深度图像中同一行人的人体框和人头框互做绑定;The detection frame binding unit is used to detect the human body frame, human head frame and the location of the area of the pedestrian in each depth image, and bind the human body frame and human head frame of the same pedestrian in the depth image;
轨迹跟踪状态检测单元,用于将每个行人轨迹的3D重心点预测位置与各深度图像对应人头框的3D重心点实际位置做跟踪匹配,根据跟踪匹配结果更新每个行人轨迹的轨迹跟踪状态,更新的状态包括新建状态、正常状态、丢失状态和删除状态;Trajectory tracking state detection unit is used to track and match the predicted position of the 3D center of gravity of each pedestrian trajectory with the actual position of the 3D center of gravity of the corresponding head frame of each depth image, and update the trajectory tracking status of each pedestrian trajectory according to the tracking matching result, The updated status includes new status, normal status, lost status and deleted status;
轨迹区域状态检测单元,基于各深度图像中3D重心点所处区域位置更新每个行人轨迹的轨迹区域状态,更新的状态包括初始状态、进入状态、注册状态和离开状态;The trajectory area status detection unit updates the trajectory area status of each pedestrian trajectory based on the location of the 3D center of gravity in each depth image. The updated status includes the initial status, the entry status, the registration status, and the exit status;
轨迹追踪单元,当任一行人轨迹在连续x帧深度图像中对应的轨迹跟踪状态均为丢失状态且轨迹区域状态为注册状态或进入状态时,采用ReID方式匹配找回丢失的行人轨迹并对应更新,否则根据深度图像中跟踪匹配到的人头框的3D重心点位置坐标对应更新行人轨迹,其中x>0,且x为整数。The trajectory tracking unit, when the corresponding trajectory tracking state of any pedestrian trajectory in the continuous x-frame depth image is the lost state and the trajectory area state is the registered state or the entered state, the ReID method is used to match and retrieve the lost pedestrian trajectory and update it accordingly , Otherwise, update the pedestrian trajectory correspondingly according to the coordinates of the 3D center of gravity of the human head frame matched by the tracking in the depth image, where x>0, and x is an integer.
与现有技术相比,本发明实施例提供的基于RGB-D图像的人体跟踪装置的有益效果与上述实施例一提供的基于RGB-D图像的人体跟踪方法的有益效果相同,在此不做赘述。Compared with the prior art, the beneficial effects of the RGB-D image-based human body tracking device provided by the embodiment of the present invention are the same as the beneficial effects of the RGB-D image-based human body tracking method provided in the first embodiment, which will not be described here. Go into details.
实施例三Example three
本实施例提供一种计算机可读存储介质,计算机可读存储介质上存储有计算机程序,计算机程序被处理器运行时执行上述基于RGB-D图像的人体跟踪方法的步骤。This embodiment provides a computer-readable storage medium on which a computer program is stored. When the computer program is run by a processor, the steps of the human body tracking method based on RGB-D images are executed.
与现有技术相比,本实施例提供的计算机可读存储介质的有益效果与上述技术方案提供的基于RGB-D图像的人体跟踪方法的有益效果相同,在此不做赘述。Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this embodiment are the same as those of the RGB-D image-based human body tracking method provided by the above technical solutions, and will not be repeated here.
本领域普通技术人员可以理解,实现上述发明方法中的全部或部分步骤是可以通过程序来指令相关的硬件来完成,上述程序可以存储于计算机可读取存储介质中,该程序在执行时,包括上述实施例方法的各步骤,而的存储介质可以是:ROM/RAM、磁碟、光盘、存储卡等。A person of ordinary skill in the art can understand that all or part of the steps in the above-mentioned inventive method can be implemented by a program instructing relevant hardware. The above-mentioned program can be stored in a computer readable storage medium. When the program is executed, it includes For each step of the method in the foregoing embodiment, the storage medium may be: ROM/RAM, magnetic disk, optical disk, memory card, etc.
以上,仅为本发明的具体实施方式,但本发明的保护范围并不局限于此,任何熟悉本技术领域的技术人员在本发明揭露的技术范围内,可轻易想到变化或替换,都应涵盖在本发明的保护范围之内。因此,本发明的保护范围应以所述权利要求的保护范围为准。The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited to this. Any person skilled in the art can easily conceive of changes or substitutions within the technical scope disclosed by the present invention, which shall cover Within the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.
Claims (10)
- 一种基于RGB-D图像的人体跟踪方法,其特征在于,包括:A human body tracking method based on RGB-D images, which is characterized in that it includes:将监控区域按照行进路线依次划分为目标外区域、注册区域和目标内区域,并利用分布的多个深度摄像头俯拍实时采集深度图像;Divide the surveillance area into the target area, the registration area and the target area in sequence according to the route of travel, and use multiple distributed depth cameras to capture real-time depth images from top-down photography;分别检测各深度图像中行人的人体框、人头框及所处区域位置,对深度图像中同一行人的人体框和人头框互做绑定;Detect the human body frame, human head frame and the location of the area of the pedestrian in each depth image, and bind the human body frame and human head frame of the same pedestrian in the depth image;将每个行人轨迹的3D重心点预测位置与各深度图像对应人头框的3D重心点实际位置做跟踪匹配,根据跟踪匹配结果更新每个行人轨迹的轨迹跟踪状态,更新的状态包括新建状态、正常状态、丢失状态和删除状态;The predicted position of the 3D center of gravity of each pedestrian trajectory is matched with the actual position of the 3D center of gravity of the corresponding head frame of each depth image, and the trajectory tracking status of each pedestrian trajectory is updated according to the tracking matching result. The updated status includes new status and normal Status, lost status and deleted status;基于各深度图像中3D重心点所处区域位置更新每个行人轨迹的轨迹区域状态,更新的状态包括初始状态、进入状态、注册状态和离开状态;Update the trajectory area status of each pedestrian trajectory based on the location of the 3D center of gravity in each depth image. The updated status includes the initial state, the entry state, the registration state, and the exit state;当任一行人轨迹在连续x帧深度图像中对应的轨迹跟踪状态均为丢失状态且轨迹区域状态为注册状态或进入状态时,采用ReID方式匹配找回丢失的行人轨迹并对应更新,否则根据深度图像中跟踪匹配到的人头框的3D重心点位置坐标对应更新行人轨迹,其中x>0,且x为整数。When the tracking state of any pedestrian trajectory in the continuous x-frame depth image is lost, and the state of the trajectory area is registered or entered, the ReID method is used to match and retrieve the lost pedestrian trajectory and update accordingly, otherwise according to the depth The position coordinates of the 3D center of gravity of the head frame matched by the tracking in the image are updated correspondingly to the pedestrian trajectory, where x>0, and x is an integer.
- 根据权利要求1所述的方法,其特征在于,分别检测各深度图像中行人的人体框、人头框及所处区域位置,对深度图像中同一行人的人体框和人头框互做绑定的方法包括:The method according to claim 1, wherein the human body frame, head frame, and location of the pedestrian in each depth image are respectively detected, and the human body frame and head frame of the same pedestrian in the depth image are bound to each other include:轮询当前帧对应的各深度图像,采用 RGB-D目标检测方法获取各深度图像中行人的人体框、人头框及行人所处的区域位置;Polling each depth image corresponding to the current frame, using the RGB-D target detection method to obtain the pedestrian's human body frame, head frame, and the location of the area where the pedestrian is located in each depth image;轮询各深度图像中出现的人体框面积和人头框面积,遍历每一对人体框和人头框的包含度;Polling the human body frame area and human head frame area appearing in each depth image, and traverse the inclusion degree of each pair of human body frame and human head frame;基于每个深度图像对应的包含度采用二分图最大匹配算法筛选出每个深度图像中属于同一行人的人体框和人头框互做绑定。Based on the inclusion degree corresponding to each depth image, the bipartite graph maximum matching algorithm is used to screen out the human body frame and head frame belonging to the same pedestrian in each depth image to bind each other.
- 根据权利要求2所述的方法,其特征在于,基于每个深度图像对应的包含度采用二分图最大匹配算法筛选出每个深度图像中属于同一行人的人体框和人头框互做绑定的方法包括:The method according to claim 2, characterized in that, based on the inclusion degree corresponding to each depth image, a bipartite graph maximum matching algorithm is used to screen out the human body frame and the human head frame belonging to the same pedestrian in each depth image to bind each other include:根据各深度图像对应的包含度大小,利用二分图最大匹配算法筛选出每个深度图像中的人体框和人头框做初始配对;According to the degree of inclusion corresponding to each depth image, use the bipartite graph maximum matching algorithm to filter out the human body frame and the human head frame in each depth image for initial pairing;将各深度图像对应初始配对中的包含度分别与重合度阈值比较,将包含度大于或等于重合度阈值的初始配对筛选出来做绑定确认,将包含度小于重合度阈值的初始配对筛选出来做绑定解除。Compare the inclusion degree in the initial pairings corresponding to each depth image with the coincidence degree threshold, and screen out the initial pairs whose inclusion degree is greater than or equal to the coincidence degree threshold for binding confirmation, and screen out the initial pairs whose inclusion degree is less than the coincidence degree threshold for binding confirmation. The binding is released.
- 根据权利要求1所述的方法,其特征在于,计算行人轨迹3D重心点预测位置的方法包括:The method according to claim 1, wherein the method for calculating the predicted position of the 3D center of gravity of the pedestrian trajectory comprises:将每个深度图像进行三维坐标转换,并计算深度图像中人头框的3D重心点;Transform each depth image into three-dimensional coordinates, and calculate the 3D center of gravity of the human head frame in the depth image;对行人轨迹3D重心点的空间位置进行多维建模,所述模型的维度向量包括 ( x,y,z,h,V x,V y,V z ),其中, x,y,z对应表示3D重心点的三维坐标, V x,V y,V z 对应表示3D重心点在对应维度坐标方向上的运动速度, h表示3D重心点所属行人的身高; Multi-dimensional modeling is performed on the spatial position of the 3D center of gravity of the pedestrian trajectory. The dimensional vector of the model includes ( x, y, z, h, V x , V y , V z ), where x, y, z correspond to 3D The three-dimensional coordinates of the center of gravity point, V x , V y , and V z correspondingly represent the movement speed of the 3D center of gravity point in the coordinate direction of the corresponding dimension, and h represents the height of the pedestrian to which the 3D center of gravity point belongs;基于行人轨迹3D重心点当前的 x轴坐标、 y轴坐标、 z轴坐标及对应在 x轴方向上的运动速度 V x, 、 y轴方向上的运动速度 V x 、 z轴方向上的运动速度 z,分别计算当前行人轨迹3D重心点在下一帧深度图像中所处 x轴方向上的预测位置、所处 y轴方向上的预测位置以及所处 z轴方向上的预测位置。 Based on the current x- axis coordinates, y- axis coordinates, z- axis coordinates of the 3D center of gravity point of the pedestrian trajectory , and the corresponding movement speed in the x- axis direction V x , the movement speed in the y- axis direction V x , the movement speed in the z- axis direction z Calculate the predicted position in the x- axis direction, the predicted position in the y- axis direction, and the predicted position in the z- axis direction of the current pedestrian trajectory 3D center of gravity in the next frame of depth image.
- 根据权利要求4所述的方法,其特征在于,将每个行人轨迹的3D重心点预测位置与各深度图像对应人头框的3D重心点实际位置做跟踪匹配,根据跟踪匹配结果更新每个行人轨迹的轨迹跟踪状态的方法包括:The method of claim 4, wherein the predicted position of the 3D center of gravity of each pedestrian trajectory is matched with the actual position of the 3D center of gravity of the corresponding head frame of each depth image, and each pedestrian trajectory is updated according to the tracking matching result The methods of trajectory tracking state include:采用卡尔曼滤波器的跟踪算法对当前帧各深度图像中对应的人头框3D重心点的实际位置进行追踪,获取3D重心点的实际位置;The Kalman filter tracking algorithm is used to track the actual position of the 3D center of gravity of the corresponding head frame in each depth image of the current frame to obtain the actual position of the 3D center of gravity;遍历计算当前帧各深度图像中每个3D重心点的实际位置与每个行人轨迹3D重心点预测位置的代价度量,得到代价矩阵;Traverse and calculate the cost metric of the actual position of each 3D center of gravity in each depth image of the current frame and the predicted position of each pedestrian trajectory 3D center of gravity to obtain a cost matrix;在当前帧各深度图像遍历计算完成后,基于代价矩阵采用二分图最大匹配算法筛选出各行人轨迹与当前帧每个深度图像中3D重心点实际位置的初选配对;After the traversal calculation of each depth image in the current frame is completed, the bipartite graph maximum matching algorithm is used based on the cost matrix to filter out the initial pairing of each pedestrian trajectory with the actual position of the 3D center of gravity in each depth image in the current frame;筛选出代价度量小于或等于代价阈值的初选配对认为配对成功,筛选出代价度量大于代价阈值的初选配对认为未配对成功;The primaries whose cost metric is less than or equal to the cost threshold are screened out and considered as successful, and the primaries whose cost metric is greater than the cost threshold are screened out as unmatched;所述未配对成功的初选配对包括剩余未配对的人头框3D重心点和剩余未配对的行人轨迹,对于当前帧各深度图像中剩余未配对且处于目标外区域的人头框3D重心点,新建一行人轨迹并将轨迹跟踪状态更新为新建状态,同时将新建行人轨迹的轨迹区域状态更新为初始状态,和/或,对于当前帧各深度图像中剩余未配对的行人轨迹,将所述行人轨迹的轨迹跟踪状态更新为丢失状态; The unpaired primary selection pairing includes the remaining unpaired head frame 3D center of gravity points and the remaining unpaired pedestrian trajectories. For the remaining unpaired head frame 3D center of gravity points in the area outside the target in each depth image of the current frame, create a new Pedestrian trajectory and update the trajectory tracking status to the new state, and at the same time update the trajectory area status of the new pedestrian trajectory to the initial state, and/or, for the remaining unpaired pedestrian trajectories in each depth image of the current frame, change the pedestrian trajectory The trajectory tracking status of is updated to the lost status;对配对成功的初选配对阈值过滤后,将其中配对的行人轨迹的轨迹跟踪状态更新为正常状态,同时将配对的人头框3D重心点的实际位置更新为当前行人轨迹的3D重心点位置;After filtering the pairing threshold for the initial selection of a successful pairing, update the trajectory tracking status of the paired pedestrian trajectory to the normal state, and at the same time update the actual position of the 3D center of gravity of the paired head frame to the 3D center of gravity of the current pedestrian trajectory;对于配对的人头框3D重心点处于目标外区域且连续n帧轨迹跟踪状态均为丢失状态的行人轨迹,和/或,轨迹区域状态为离开状态的行人轨迹,和/或,轨迹跟踪状态为初始状态且连续m帧轨迹跟踪状态均为丢失状态的行人轨迹,将所述行人轨迹的轨迹跟踪状态更新为删除状态,其中, n>0,m>0,且n和m均为整数。For the pedestrian trajectory whose 3D center of gravity of the paired head frame is in the area outside the target and the trajectory tracking state for consecutive n frames is the lost state, and/or the trajectory area state is the pedestrian trajectory in the off state, and/or the trajectory tracking state is the initial state State and continuous m frames of trajectory tracking state are pedestrian trajectories in the lost state, and the trajectory tracking state of the pedestrian trajectory is updated to the deleted state, where n>0, m>0, and n and m are both integers.
- 根据权利要求5所述的方法,其特征在于,还包括:The method according to claim 5, further comprising:当行人轨迹对应的轨迹跟踪状态为删除状态时,删除该行人轨迹及其对应的底库特征数据表。When the track tracking state corresponding to the pedestrian track is the deleted state, delete the pedestrian track and its corresponding bottom library feature data table.
- 根据权利要求6所述的方法,其特征在于,基于各深度图像中3D重心点所处的区域位置更新每个行人轨迹的轨迹区域状态的方法包括:The method according to claim 6, wherein the method of updating the state of the trajectory area of each pedestrian trajectory based on the position of the area where the 3D center of gravity point is located in each depth image comprises:遍历当前帧各深度图像中的人头框3D重心点,识别出现在目标外区域的人头框3D重心点,将其对应行人轨迹的轨迹区域状态置为初始状态;Traverse the 3D center of gravity of the head frame in each depth image of the current frame, identify the 3D center of gravity of the head frame that appears in the area outside the target, and set the state of the trajectory area corresponding to the pedestrian trajectory to the initial state;遍历当前帧各深度图像中的人头框3D重心点,识别出现在注册区域的人头框3D重心点,将其对应行人轨迹的轨迹区域状态置为注册状态,注册并实时更新底库特征数据表;Traverse the 3D center of gravity of the head frame in each depth image of the current frame, identify the 3D center of gravity of the head frame that appears in the registered area, set the status of the trajectory area corresponding to the pedestrian trajectory to the registered state, register and update the base library feature data table in real time;遍历当前帧各深度图像中的人头框3D重心点,识别出现在目标内区域的人头框3D重心点,将其对应行人轨迹的轨迹区域状态置为进入状态;Traverse the 3D center of gravity of the head frame in each depth image of the current frame, identify the 3D center of gravity of the head frame that appears in the target area, and set the state of the trajectory area corresponding to the pedestrian trajectory to the entering state;遍历当前帧各深度图像中的人头框3D重心点,识别离开目标内区域进入目标外区域的人头框3D重心点,将其对应行人轨迹的轨迹区域状态置为离开状态。Traverse the 3D center of gravity of the head frame in each depth image of the current frame, identify the 3D center of gravity of the head frame that leaves the area inside the target and enter the area outside the target, and set the state of the trajectory area corresponding to the pedestrian trajectory to the away state.
- 根据权利要求1所述的方法,其特征在于,判断行人轨迹的轨迹跟踪状态为丢失状态的方法包括:The method according to claim 1, wherein the method for judging that the tracking state of the pedestrian trajectory is a lost state comprises:识别当前帧各深度图像中的人头框3D重心点,若行人轨迹不能与任一个深度图像中的人头框3D重心点相匹配,则认为所述行人轨迹的轨迹跟踪状态为丢失状态。Identify the 3D center of gravity of the head frame in each depth image of the current frame, and if the pedestrian trajectory cannot match the 3D center of gravity of the head frame in any depth image, the trajectory tracking state of the pedestrian trajectory is considered to be a lost state.
- 一种基于RGB-D图像的人体跟踪装置,其特征在于,包括:A human body tracking device based on RGB-D images, which is characterized in that it comprises:分区设置单元,用于将监控区域按照行进路线依次划分为目标外区域、注册区域和目标内区域,并利用分布的多个深度摄像头俯拍实时采集深度图像;The partition setting unit is used to divide the monitoring area into the target area, the registration area, and the target area in sequence according to the route of travel, and use multiple distributed depth cameras to capture real-time depth images;检测框绑定单元,用于分别检测各深度图像中行人的人体框、人头框及所处区域位置,对深度图像中同一行人的人体框和人头框互做绑定;The detection frame binding unit is used to detect the human body frame, human head frame and the location of the area of the pedestrian in each depth image, and bind the human body frame and human head frame of the same pedestrian in the depth image;轨迹跟踪状态检测单元,用于将每个行人轨迹的3D重心点预测位置与各深度图像对应人头框的3D重心点实际位置做跟踪匹配,根据跟踪匹配结果更新每个行人轨迹的轨迹跟踪状态,更新的状态包括新建状态、正常状态、丢失状态和删除状态;Trajectory tracking state detection unit is used to track and match the predicted position of the 3D center of gravity of each pedestrian trajectory with the actual position of the 3D center of gravity of the corresponding head frame of each depth image, and update the trajectory tracking status of each pedestrian trajectory according to the tracking matching result, The updated status includes new status, normal status, lost status and deleted status;轨迹区域状态检测单元,基于各深度图像中3D重心点所处区域位置更新每个行人轨迹的轨迹区域状态,更新的状态包括初始状态、进入状态、注册状态和离开状态;The trajectory area status detection unit updates the trajectory area status of each pedestrian trajectory based on the location of the 3D center of gravity in each depth image. The updated status includes the initial status, the entry status, the registration status, and the exit status;轨迹追踪单元,当任一行人轨迹在连续x帧深度图像中对应的轨迹跟踪状态均为丢失状态且轨迹区域状态为注册状态或进入状态时,采用ReID方式匹配找回丢失的行人轨迹并对应更新,否则根据深度图像中跟踪匹配到的人头框的3D重心点位置坐标对应更新行人轨迹,其中x>0,且x为整数。The trajectory tracking unit, when the corresponding trajectory tracking state of any pedestrian trajectory in the continuous x-frame depth image is the lost state and the trajectory area state is the registered state or the entered state, the ReID method is used to match and retrieve the lost pedestrian trajectory and update it accordingly , Otherwise, update the pedestrian trajectory correspondingly according to the coordinates of the 3D center of gravity of the human head frame matched by the tracking in the depth image, where x>0, and x is an integer.
- 一种计算机可读存储介质,计算机可读存储介质上存储有计算机程序,其特征在于,计算机程序被处理器运行时执行上述权利要求1至8任一项所述方法的步骤。A computer-readable storage medium with a computer program stored on the computer-readable storage medium, wherein the computer program executes the steps of the method according to any one of claims 1 to 8 when the computer program is run by a processor.
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202010573284.9 | 2020-06-22 | ||
CN202010573284.9A CN111709974B (en) | 2020-06-22 | 2020-06-22 | Human body tracking method and device based on RGB-D image |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2021259055A1 true WO2021259055A1 (en) | 2021-12-30 |
Family
ID=72541731
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CN2021/098724 WO2021259055A1 (en) | 2020-06-22 | 2021-06-07 | Human body tracking method and device based on rgb-d image |
Country Status (2)
Country | Link |
---|---|
CN (1) | CN111709974B (en) |
WO (1) | WO2021259055A1 (en) |
Cited By (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN114926859A (en) * | 2022-01-04 | 2022-08-19 | 天津理工大学 | Pedestrian multi-target tracking method in dense scene combined with head tracking |
CN114972417A (en) * | 2022-04-02 | 2022-08-30 | 江南大学 | Multi-target tracking method for dynamic track quality quantification and feature re-planning |
CN115019241A (en) * | 2022-08-05 | 2022-09-06 | 江西中业智能科技有限公司 | Pedestrian identification and tracking method and device, readable storage medium and equipment |
CN115994925A (en) * | 2023-02-14 | 2023-04-21 | 成都理工大学工程技术学院 | Multi-row person rapid tracking method based on key point detection |
CN116188357A (en) * | 2022-09-27 | 2023-05-30 | 珠海视熙科技有限公司 | Entrance and exit human body detection method, imaging equipment, device and storage medium |
Families Citing this family (14)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN111709974B (en) * | 2020-06-22 | 2022-08-02 | 苏宁云计算有限公司 | Human body tracking method and device based on RGB-D image |
CN112446355B (en) * | 2020-12-15 | 2023-10-17 | 中电海康集团有限公司 | Pedestrian recognition method and people stream statistics system in public place |
KR102580281B1 (en) * | 2020-12-29 | 2023-09-18 | 센스타임 인터내셔널 피티이. 리미티드. | Related object detection method and device |
WO2022144601A1 (en) * | 2020-12-29 | 2022-07-07 | Sensetime International Pte. Ltd. | Method and apparatus for detecting associated objects |
WO2022144600A1 (en) * | 2020-12-29 | 2022-07-07 | Sensetime International Pte. Ltd. | Object detection method and apparatus, and electronic device |
CN112686178B (en) * | 2020-12-30 | 2024-04-16 | 中国电子科技集团公司信息科学研究院 | Multi-view target track generation method and device and electronic equipment |
CN112581507A (en) * | 2020-12-31 | 2021-03-30 | 北京澎思科技有限公司 | Target tracking method, system and computer readable storage medium |
CN113034544A (en) * | 2021-03-19 | 2021-06-25 | 奥比中光科技集团股份有限公司 | People flow analysis method and device based on depth camera |
CN113377192B (en) * | 2021-05-20 | 2023-06-20 | 广州紫为云科技有限公司 | Somatosensory game tracking method and device based on deep learning |
CN113963375A (en) * | 2021-10-20 | 2022-01-21 | 中国石油大学(华东) | Multi-feature matching multi-target tracking method for fast skating athletes based on regions |
CN114240997B (en) * | 2021-11-16 | 2023-07-28 | 南京云牛智能科技有限公司 | Intelligent building online trans-camera multi-target tracking method |
CN114708304B (en) * | 2022-06-06 | 2022-10-28 | 苏州浪潮智能科技有限公司 | Cross-camera multi-target tracking method, device, equipment and medium |
WO2024004197A1 (en) * | 2022-07-01 | 2024-01-04 | 日本電気株式会社 | Information processing device, information processing method, and recording medium |
CN116528062B (en) * | 2023-07-05 | 2023-09-15 | 合肥中科类脑智能技术有限公司 | Multi-target tracking method |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20130121526A1 (en) * | 2011-11-11 | 2013-05-16 | Microsoft Corporation | Computing 3d shape parameters for face animation |
CN110163889A (en) * | 2018-10-15 | 2019-08-23 | 腾讯科技(深圳)有限公司 | Method for tracking target, target tracker, target following equipment |
CN110443116A (en) * | 2019-06-19 | 2019-11-12 | 平安科技(深圳)有限公司 | Video pedestrian detection method, device, server and storage medium |
CN111709974A (en) * | 2020-06-22 | 2020-09-25 | 苏宁云计算有限公司 | Human body tracking method and device based on RGB-D image |
Family Cites Families (14)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9098110B2 (en) * | 2011-06-06 | 2015-08-04 | Microsoft Technology Licensing, Llc | Head rotation tracking from depth-based center of mass |
US8855369B2 (en) * | 2012-06-22 | 2014-10-07 | Microsoft Corporation | Self learning face recognition using depth based tracking for database generation and update |
JP6448223B2 (en) * | 2014-06-12 | 2019-01-09 | キヤノン株式会社 | Image recognition system, image recognition apparatus, image recognition method, and computer program |
GB201501311D0 (en) * | 2015-01-27 | 2015-03-11 | Apical Ltd | Method, system and computer program product |
CN104751491B (en) * | 2015-04-10 | 2018-01-23 | 中国科学院宁波材料技术与工程研究所 | A kind of crowd's tracking and people flow rate statistical method and device |
CN107180435B (en) * | 2017-05-09 | 2020-05-26 | 杭州电子科技大学 | Human body target tracking method suitable for depth image |
CN108053427B (en) * | 2017-10-31 | 2021-12-14 | 深圳大学 | Improved multi-target tracking method, system and device based on KCF and Kalman |
CN109146923B (en) * | 2018-07-13 | 2021-08-24 | 高新兴科技集团股份有限公司 | Processing method and system for target tracking broken frame |
CN109583373B (en) * | 2018-11-29 | 2022-08-19 | 成都索贝数码科技股份有限公司 | Pedestrian re-identification implementation method |
CN109919974B (en) * | 2019-02-21 | 2023-07-14 | 上海理工大学 | Online multi-target tracking method based on R-FCN frame multi-candidate association |
CN110490161B (en) * | 2019-08-23 | 2022-01-07 | 安徽农业大学 | Captive animal behavior analysis method based on deep learning |
CN110728702B (en) * | 2019-08-30 | 2022-05-20 | 深圳大学 | High-speed cross-camera single-target tracking method and system based on deep learning |
CN111027462A (en) * | 2019-12-06 | 2020-04-17 | 长沙海格北斗信息技术有限公司 | Pedestrian track identification method across multiple cameras |
CN111079600A (en) * | 2019-12-06 | 2020-04-28 | 长沙海格北斗信息技术有限公司 | Pedestrian identification method and system with multiple cameras |
-
2020
- 2020-06-22 CN CN202010573284.9A patent/CN111709974B/en active Active
-
2021
- 2021-06-07 WO PCT/CN2021/098724 patent/WO2021259055A1/en active Application Filing
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20130121526A1 (en) * | 2011-11-11 | 2013-05-16 | Microsoft Corporation | Computing 3d shape parameters for face animation |
CN110163889A (en) * | 2018-10-15 | 2019-08-23 | 腾讯科技(深圳)有限公司 | Method for tracking target, target tracker, target following equipment |
CN110443116A (en) * | 2019-06-19 | 2019-11-12 | 平安科技(深圳)有限公司 | Video pedestrian detection method, device, server and storage medium |
CN111709974A (en) * | 2020-06-22 | 2020-09-25 | 苏宁云计算有限公司 | Human body tracking method and device based on RGB-D image |
Non-Patent Citations (1)
Title |
---|
"Master's Thesis of Shenzhen University", 15 July 2017, SHENZHEN UNIVERSITY, CN, article LIU LEI: "Study on the Pedestrians Detection and Statistical Method with the Depth Sequence Images", XP055885793 * |
Cited By (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN114926859A (en) * | 2022-01-04 | 2022-08-19 | 天津理工大学 | Pedestrian multi-target tracking method in dense scene combined with head tracking |
CN114972417A (en) * | 2022-04-02 | 2022-08-30 | 江南大学 | Multi-target tracking method for dynamic track quality quantification and feature re-planning |
CN115019241A (en) * | 2022-08-05 | 2022-09-06 | 江西中业智能科技有限公司 | Pedestrian identification and tracking method and device, readable storage medium and equipment |
CN115019241B (en) * | 2022-08-05 | 2022-11-01 | 江西中业智能科技有限公司 | Pedestrian identification and tracking method and device, readable storage medium and equipment |
CN116188357A (en) * | 2022-09-27 | 2023-05-30 | 珠海视熙科技有限公司 | Entrance and exit human body detection method, imaging equipment, device and storage medium |
CN115994925A (en) * | 2023-02-14 | 2023-04-21 | 成都理工大学工程技术学院 | Multi-row person rapid tracking method based on key point detection |
CN115994925B (en) * | 2023-02-14 | 2023-09-29 | 成都理工大学工程技术学院 | Multi-row person rapid tracking method based on key point detection |
Also Published As
Publication number | Publication date |
---|---|
CN111709974A (en) | 2020-09-25 |
CN111709974B (en) | 2022-08-02 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
WO2021259055A1 (en) | Human body tracking method and device based on rgb-d image | |
Cohen et al. | Detecting and tracking moving objects for video surveillance | |
US10664706B2 (en) | System and method for detecting, tracking, and classifying objects | |
Huitl et al. | TUMindoor: An extensive image and point cloud dataset for visual indoor localization and mapping | |
CN105405154B (en) | Target object tracking based on color-structure feature | |
Ermis et al. | Activity based matching in distributed camera networks | |
US20120209514A1 (en) | Change invariant scene recognition by an agent | |
CN101344965A (en) | Tracking system based on binocular camera shooting | |
KR102376479B1 (en) | Method, device and system for controlling for automatic recognition of object based on artificial intelligence | |
US11348276B2 (en) | Mobile robot control method | |
Pareek et al. | Re-projected SURF features based mean-shift algorithm for visual tracking | |
Zhang et al. | A fast and robust people counting method in video surveillance | |
Jiao et al. | Anatomy of a multicamera video surveillance system | |
KR101456172B1 (en) | Localization of a mobile robot device, method and mobile robot | |
Prokaj et al. | Using 3d scene structure to improve tracking | |
Masoud et al. | The use of computer vision in monitoring weaving sections | |
Liu et al. | Multi-camera vehicle tracking based on occlusion-aware and inter-vehicle information | |
Liu et al. | A person-following method based on monocular camera for quadruped robots | |
CN113989761A (en) | Object tracking method and device, electronic equipment and storage medium | |
Amer | Voting-based simultaneous tracking of multiple video objects | |
CN109344685A (en) | A kind of wisdom pallet and its intelligent positioning method for tracing | |
CN115731287A (en) | Moving target retrieval method based on set and topological space | |
Peng et al. | Accurate visual-inertial slam by feature re-identification | |
Vokhmintcev et al. | Real-time visual loop-closure detection using fused iterative close point algorithm and extended Kalman filter | |
Vasuhi et al. | Object detection and tracking in secured area with wireless and multimedia sensor network |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 21830221 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 21830221 Country of ref document: EP Kind code of ref document: A1 |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 21830221 Country of ref document: EP Kind code of ref document: A1 |