CN112446355A - Public place pedestrian identification method and pedestrian flow statistical system - Google Patents

Public place pedestrian identification method and pedestrian flow statistical system Download PDF

Info

Publication number
CN112446355A
CN112446355A CN202011477711.XA CN202011477711A CN112446355A CN 112446355 A CN112446355 A CN 112446355A CN 202011477711 A CN202011477711 A CN 202011477711A CN 112446355 A CN112446355 A CN 112446355A
Authority
CN
China
Prior art keywords
pedestrian
dimensional
historical
current
bounding box
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202011477711.XA
Other languages
Chinese (zh)
Other versions
CN112446355B (en
Inventor
舒元昊
张一杨
马小雯
刘倚剑
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
CETHIK Group Ltd
Original Assignee
CETHIK Group Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by CETHIK Group Ltd filed Critical CETHIK Group Ltd
Priority to CN202011477711.XA priority Critical patent/CN112446355B/en
Priority to PCT/CN2020/137803 priority patent/WO2022126668A1/en
Publication of CN112446355A publication Critical patent/CN112446355A/en
Application granted granted Critical
Publication of CN112446355B publication Critical patent/CN112446355B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects
    • G06V20/53—Recognition of crowd images, e.g. recognition of crowd congestion
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/40—Scenes; Scene-specific elements in video content
    • G06V20/46—Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/103—Static body considered as a whole, e.g. static pedestrian or occupant recognition
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30196—Human being; Person
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30244—Camera pose

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • General Health & Medical Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Biomedical Technology (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Biophysics (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Health & Medical Sciences (AREA)
  • Human Computer Interaction (AREA)
  • Image Analysis (AREA)

Abstract

The invention discloses a pedestrian identification method and a pedestrian flow statistical system in a public place, wherein the method comprises the following steps: acquiring an optical image, detecting a pedestrian in the optical image, and outputting a three-dimensional bounding box of the pedestrian and a corresponding timestamp; acquiring pedestrian characteristics based on the optical image and the three-dimensional bounding box; identifying the pedestrian based on the pedestrian characteristics in the historical characteristic library; and marking the pedestrian state of the pedestrian corresponding to the three-dimensional bounding box as a first matching success, a lost pedestrian, a re-matching success after the loss, a continuous matching success or a shooting range according to the matching result of the time and the historical matching result. The invention comprehensively considers the pedestrian appearance characteristics, the three-dimensional motion characteristics and the motion mode, accurately identifies the pedestrian, obtains the time and the position of the pedestrian entering and exiting the statistical range and the moving track in the statistical range, provides a pedestrian flow statistical system based on the method, and can accurately count the pedestrian flow entering and exiting the statistical range in unit time.

Description

Public place pedestrian identification method and pedestrian flow statistical system
Technical Field
The invention belongs to the technical field of computer vision, and particularly relates to a pedestrian identification method and a pedestrian flow statistical system in a public place.
Background
People flow statistics relates to pedestrian identification, residence time and access tracks of pedestrians in a statistical area, and currently, common statistical methods include base station-based statistical methods such as a Bluetooth base station and a 4G base station, but the positioning accuracy of the methods is not accurate enough; there are statistical methods based on non-optical imaging devices, such as statistical methods of infrared arrays and millimeter wave radars, which have relatively high positioning accuracy, but cannot accurately identify pedestrians, thus easily causing repeated statistics; the statistical method based on the optical imaging equipment, such as a camera, is high in positioning accuracy and capable of accurately identifying pedestrians, but has the problem that the pedestrians are shielded, and the statistical method based on the pedestrian re-identification partially has the repeated statistical problem caused by the fact that the motion mode of the pedestrians is not consistent with the filtering prediction track.
Disclosure of Invention
The invention aims to provide a pedestrian identification method and a pedestrian flow statistical system in a public place, which can accurately identify pedestrians and have high pedestrian flow statistical accuracy.
In order to achieve the purpose, the technical scheme adopted by the invention is as follows:
a public place pedestrian identification method, comprising:
step 1, acquiring an optical image, detecting a pedestrian in the optical image, and outputting a three-dimensional bounding box of the pedestrian and a corresponding timestamp;
step 2, acquiring pedestrian characteristics based on the optical image and the three-dimensional bounding box, and the method comprises the following steps:
step 2.1, extracting the human body shape and the features of the pedestrians in the optical image to serve as the pedestrian apparent features of each pedestrian, and storing the pedestrian apparent features in a historical feature library;
2.2, extracting the pedestrian three-dimensional motion characteristics of each pedestrian based on the current three-dimensional bounding boxes of the pedestrians and the three-dimensional bounding boxes distributed in the historical characteristic library according to the time sequence, and storing the pedestrian three-dimensional motion characteristics into the historical characteristic library;
step 2.3, predicting the three-dimensional motion characteristics of the pedestrian at the next moment based on the three-dimensional motion characteristics of the pedestrian and the three-dimensional motion characteristics of the pedestrian within the specified time in the historical characteristic library, and storing the three-dimensional motion characteristics of the pedestrian into the historical characteristic library;
and 3, identifying the pedestrians based on the pedestrian characteristics in the historical characteristic library, wherein the identification comprises the following steps:
step 3.1, calculating apparent feature distances one by one based on the current apparent features of the pedestrians and the historical apparent features of the pedestrians in the historical feature library, if the apparent feature distances are larger than an apparent threshold value, judging that the current apparent features of the pedestrians and the apparent features of the pedestrians in the historical feature library belong to the same pedestrian, and determining the current apparent feature distances as the apparent feature distances of the pedestrians;
step 3.2, calculating spatial feature distances one by one based on the current three-dimensional motion features of the pedestrians and the three-dimensional motion features of the pedestrians at the next moment predicted by the previous moment of each pedestrian in the historical feature library, if the spatial feature distances are larger than a spatial threshold, judging that the current three-dimensional motion features of the pedestrians and the three-dimensional motion features of the pedestrians at the next moment predicted by the previous moment in the historical feature library belong to the same pedestrian, and determining the current spatial feature distances as the spatial feature distances of the pedestrians;
3.3, judging whether the pedestrian meets the motion mode of a pedestrian according with the same contract based on the current three-dimensional motion characteristics, apparent characteristic distance and spatial characteristic distance of the pedestrian and the historical three-dimensional motion characteristics of the pedestrian in a historical characteristic library, and outputting the motion mode matching degree as the motion mode matching degree of the pedestrian;
step 3.4, performing weighted calculation on the apparent characteristic distance, the spatial characteristic distance and the motion pattern matching degree of the same pedestrian to obtain a matching result of the pedestrian in the current three-dimensional bounding box and the pedestrian in the historical characteristic library, wherein the matching result comprises successful matching or failed matching, and the pedestrian information obtained by matching is also included when the matching is successful;
and 4, marking the pedestrian state of the pedestrian corresponding to the three-dimensional bounding box as a pedestrian state which is successfully matched for the first time, lost, re-successfully matched after being lost, successfully matched continuously or out of the shooting range according to the matching result of the current time and the historical matching result.
Several alternatives are provided below, but not as an additional limitation to the above general solution, but merely as a further addition or preference, each alternative being combinable individually for the above general solution or among several alternatives without technical or logical contradictions.
Preferably, the detecting a pedestrian in the optical image and outputting a three-dimensional bounding box of the pedestrian includes:
calibrating a camera for acquiring an optical image to obtain a mapping relation between pixels in the optical image and the distance of the camera;
detecting a pedestrian in the optical image, and acquiring a two-dimensional surrounding frame of the pedestrian in the optical image;
and obtaining the three-dimensional bounding box of the pedestrian based on the two-dimensional bounding box and the mapping relation.
Preferably, the extracting the pedestrian three-dimensional motion feature of each pedestrian based on the current three-dimensional bounding box of the pedestrian and the three-dimensional bounding boxes distributed in the historical feature library according to the time series includes:
step 2.2.1, extracting direction vectors: extracting the moving direction of the pedestrian in the horizontal direction and the moving direction of the pedestrian in the vertical direction through the current three-dimensional bounding box and the historical three-dimensional bounding box;
step 2.2.2, extracting the movement speed: extracting the movement speed of a person in the horizontal direction and the movement speed of the person in the vertical direction through the current three-dimensional bounding box and the historical three-dimensional bounding box;
step 2.2.3, extracting relative positions: outputting coordinates of the pedestrian in a three-dimensional coordinate system with the camera as the center based on the current three-dimensional bounding box and the historical three-dimensional bounding box according to a mapping relation obtained after the camera is calibrated;
and 2.2.4, taking the direction vector, the motion speed and the relative position extracted in the step 2.2.1-2.2.3 as the three-dimensional motion characteristics of the pedestrian.
Preferably, the marking the pedestrian state of the pedestrian corresponding to the three-dimensional bounding box according to the matching result of this time and the historical matching result is that the pedestrian state is successfully matched for the first time, lost, successfully matched again after lost, successfully matched continuously or goes out of the shooting range includes:
if the pedestrian features are successfully extracted, but the matching result is matching failure, marking the current pedestrian state as successful primary matching;
if the same pedestrian in the history matching result is not matched for M times, marking the state of the pedestrian as lost;
if the matching result of the pedestrian marked as lost is successfully re-matched in the current matching result, updating the state of the pedestrian as the state of the lost pedestrian which is successfully re-matched;
if the same pedestrian in the history matching result is matched for L times, updating the state of the pedestrian as successful continuous matching;
and if the same pedestrian in the history matching result is not matched for N times, marking the state of the pedestrian as the exit shooting range, and M < N.
Preferably, if the status flag of the current pedestrian is that the initial matching is successful, new pedestrian information is assigned to the pedestrian in the historical feature library, and the pedestrian feature of the pedestrian is associated with the newly assigned pedestrian information.
The invention also provides a people stream statistical system, which comprises:
the pedestrian detection module is used for acquiring an optical image, detecting a pedestrian in the optical image and outputting a three-dimensional bounding box of the pedestrian and a corresponding timestamp;
the feature extraction module is used for acquiring pedestrian features based on the optical images and the three-dimensional bounding box, and specifically executes the following steps:
a. extracting the human body shape and the features of the pedestrians in the optical image to serve as the pedestrian apparent features of each pedestrian, and storing the pedestrian apparent features in a historical feature library;
b. extracting the pedestrian three-dimensional motion characteristics of each pedestrian based on the current three-dimensional bounding boxes of the pedestrians and the three-dimensional bounding boxes distributed in the historical characteristic library according to the time sequence, and storing the pedestrian three-dimensional motion characteristics into the historical characteristic library;
c. predicting the three-dimensional motion characteristics of the pedestrian at the next moment based on the three-dimensional motion characteristics of the pedestrian and the three-dimensional motion characteristics of the pedestrian within the specified time in the historical characteristic library, and storing the three-dimensional motion characteristics of the pedestrian into the historical characteristic library;
the pedestrian recognition module is used for recognizing pedestrians based on pedestrian features in the historical feature library, and specifically executes the following steps:
a. calculating apparent feature distances one by one based on the current apparent features of the pedestrians and the historical apparent features of the pedestrians in the historical feature library, if the apparent feature distances are larger than an apparent threshold value, judging that the current apparent features of the pedestrians and the apparent features of the pedestrians in the historical feature library belong to the same pedestrian, and determining the current apparent feature distances as the apparent feature distances of the pedestrians;
b. calculating a spatial feature distance one by one based on the current three-dimensional motion feature of the pedestrian and the three-dimensional motion feature of the pedestrian at the next moment predicted by each pedestrian at the previous moment in the historical feature library, if the spatial feature distance is greater than a spatial threshold, judging that the current three-dimensional motion feature of the pedestrian and the three-dimensional motion feature of the pedestrian at the next moment predicted by the pedestrian at the previous moment in the historical feature library belong to the same pedestrian, and determining the current spatial feature distance as the spatial feature distance of the pedestrian;
c. judging whether a pedestrian motion mode is matched with a pedestrian on the basis of the current three-dimensional motion characteristic, the apparent characteristic distance, the spatial characteristic distance of the pedestrian and the historical three-dimensional motion characteristic of the pedestrian in a historical characteristic library, and outputting a motion mode matching degree as the motion mode matching degree of the pedestrian;
d. weighting and calculating the apparent characteristic distance, the spatial characteristic distance and the motion pattern matching degree which belong to the same pedestrian to obtain a matching result of the pedestrian in the current three-dimensional bounding box and the pedestrian in the historical characteristic library, wherein the matching result comprises successful matching or failed matching, and the pedestrian information obtained by matching is also included when the matching is successful;
the pedestrian marking module is used for marking the pedestrian state of the pedestrian corresponding to the three-dimensional bounding box as a pedestrian state which is successfully matched for the first time, lost or re-matched after being lost, successfully matched for the continuous time or out of a shooting range according to the matching result of the current time and the historical matching result;
and the pedestrian flow counting module is used for counting the pedestrian flow in the counting range corresponding to the optical image within the preset time according to the pedestrian state.
Preferably, the detecting the pedestrian in the optical image, outputting a three-dimensional bounding box of the pedestrian, and performing the following operations:
calibrating a camera for acquiring an optical image to obtain a mapping relation between pixels in the optical image and the distance of the camera;
detecting a pedestrian in the optical image, and acquiring a two-dimensional surrounding frame of the pedestrian in the optical image;
and obtaining the three-dimensional bounding box of the pedestrian based on the two-dimensional bounding box and the mapping relation.
Preferably, the pedestrian three-dimensional motion feature of each pedestrian is extracted based on the current three-dimensional bounding box of the pedestrian and three-dimensional bounding boxes distributed in time series in the historical feature library, and the following operations are executed:
extracting a direction vector: extracting the moving direction of the pedestrian in the horizontal direction and the moving direction of the pedestrian in the vertical direction through the current three-dimensional bounding box and the historical three-dimensional bounding box;
extracting the movement speed: extracting the movement speed of a person in the horizontal direction and the movement speed of the person in the vertical direction through the current three-dimensional bounding box and the historical three-dimensional bounding box;
extracting relative positions: outputting coordinates of the pedestrian in a three-dimensional coordinate system with the camera as the center based on the current three-dimensional bounding box and the historical three-dimensional bounding box according to a mapping relation obtained after the camera is calibrated;
feature integration: and taking the extracted direction vector, the motion speed and the relative position as the three-dimensional motion characteristics of the pedestrian.
Preferably, the pedestrian state of the pedestrian corresponding to the three-dimensional bounding box is marked as the initial matching success, the loss, the re-matching success after the loss, the continuous matching success or the image pickup range according to the matching result of the current time and the historical matching result, and the following operations are executed:
if the pedestrian features are successfully extracted, but the matching result is matching failure, marking the current pedestrian state as successful primary matching;
if the same pedestrian in the history matching result is not matched for M times, marking the state of the pedestrian as lost;
if the matching result of the pedestrian marked as lost is successfully re-matched in the current matching result, updating the state of the pedestrian as the state of the lost pedestrian which is successfully re-matched;
if the same pedestrian in the history matching result is matched for L times, updating the state of the pedestrian as successful continuous matching;
and if the same pedestrian in the history matching result is not matched for N times, marking the state of the pedestrian as the exit shooting range, and M < N.
Preferably, if the status flag of the current pedestrian is that the initial matching is successful, new pedestrian information is assigned to the pedestrian in the historical feature library, and the pedestrian feature of the pedestrian is associated with the newly assigned pedestrian information.
According to the pedestrian identification method in the public place, the pedestrian apparent characteristics, the three-dimensional motion characteristics and the motion mode are comprehensively considered, the pedestrian is accurately identified, the time and the position of the pedestrian entering and exiting the statistical range and the moving track of the pedestrian in the statistical range are obtained, and the pedestrian flow statistical system is provided based on the method, so that the pedestrian flow entering and exiting the statistical range in unit time can be accurately counted.
Drawings
FIG. 1 is a flow chart of a public pedestrian identification method of the present invention;
FIG. 2 is a flow chart of the present invention outputting a three-dimensional bounding box of a pedestrian;
FIG. 3 is a flow chart of the present invention for obtaining pedestrian features based on optical images and three-dimensional bounding boxes;
FIG. 4 is a schematic diagram of the present invention for extracting motion features according to human anatomy in a right-hand coordinate system;
FIG. 5 is a flow chart of pedestrian identification based on pedestrian features in the historical feature library according to the present invention;
FIG. 6 is a flow chart of the present invention for pedestrian status marking;
fig. 7 is a block diagram of the people flow statistical system of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. The terminology used in the description of the invention herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.
In one embodiment, the method for identifying the pedestrians in the public places is accurate in pedestrian identification, and can be used for carrying out urban planning based on pedestrian identification and pedestrian flow statistics, carrying out business strategy adjustment on mall pedestrian flow statistics, carrying out subway shift adjustment on subway station pedestrian flow statistics and the like.
As shown in fig. 1, the method for identifying pedestrians in public places in the embodiment includes the following steps:
step 1, acquiring an optical image, detecting a pedestrian in the optical image, and outputting a three-dimensional bounding box of the pedestrian and a corresponding timestamp.
In this embodiment, the optical image is acquired based on the camera, and the obtained timestamp is the time when the optical image is shot by the camera. It is easily understood that the optical image acquisition can be based on any image acquisition device, and the embodiment is described by taking a camera as an example.
Since the three-dimensional bounding box is provided with depth information, but the optical image does not contain depth information, the present embodiment includes the following steps in forming the three-dimensional bounding box, as shown in fig. 2:
calibrating a camera for acquiring an optical image to obtain a mapping relation between pixels in the optical image and the distance of the camera; detecting a pedestrian in the optical image, and acquiring a two-dimensional Bounding Box (BBox) of the pedestrian in the optical image; and obtaining a three-dimensional Bounding Box (3D Bounding Box, 3D BBox) of the pedestrian based on the two-dimensional Bounding Box and the mapping relation.
According to the embodiment, the mapping relation between the pedestrian pixels in the optical image and the distance between the pedestrian pixels and the camera is obtained through calibration of the camera, corresponding depth information is obtained based on the mapping relation, the depth information reflects the actual distance between the pedestrian and the camera, movement change of the pedestrian is included, and the three-dimensional motion characteristic of the pedestrian can be conveniently extracted based on the depth information subsequently.
In this embodiment, a monocular fixed-focus camera is used to capture a video, a cube with each side length of 1 meter is used to calibrate the mapping relationship, each face of the cube is averagely divided into 100 grids with black and white alternating, and the camera capture range is a statistical range.
It should be noted that, the conventional technical means used for calibrating the camera is not limited to the specific steps of calibration in this embodiment, and the depth information is obtained based on the mapping relationship calibrated by the camera, which is a preferred method provided in this embodiment, but is not limited to a unique means, for example, the depth information may be superimposed by combining a video camera and a depth camera.
In this embodiment, the pedestrian in the optical image is identified based on a pedestrian detection method and the two-dimensional bounding box is output, where the pedestrian detection method is a conventional method in image identification, for example, a recognition network based on Yolo expansion obtained by training on a pedestrian data set is used. And when the three-dimensional bounding box is obtained, inputting the two-dimensional bounding box and the mapping relation into a three-dimensional estimation method, and outputting the three-dimensional bounding box.
The three-dimensional estimation method used in this embodiment is an optical flow-based monocular depth estimation method, which can output an inverse depth from which depth information can be calculated. The depth information has different error coefficients in different ranges from the camera, and in this embodiment, an error matrix is used.
The problem of pedestrian shielding can be effectively overcome by carrying out pedestrian identification based on the three-dimensional bounding box, and because the body structure of a person conforms to geometric constraint, the partially shielded two-dimensional bounding box can be restored into a complete three-dimensional bounding box, and the spatial error of the three-dimensional bounding box is within an allowable range.
Step 2, acquiring pedestrian characteristics based on the optical image and the three-dimensional bounding box, as shown in fig. 3, including:
and 2.1, extracting the human body shape and the features of the pedestrians in the optical image to serve as the pedestrian apparent features of each pedestrian, and storing the pedestrian apparent features in a historical feature library.
Since the shape and features of the human body are important features for distinguishing different pedestrians, the method for extracting the apparent features of the pedestrians is adopted in the embodiment, the human body characters and features which can be observed visually are mainly extracted, the distinguishing is convenient, and the apparent features of the pedestrians are marked as Fappearance。
In the present embodiment, an embedding structure (embedding) of a recognition network based on a Yolo expansion is used as a pedestrian apparent feature extraction method.
And 2.2, extracting the pedestrian three-dimensional motion characteristics of each pedestrian based on the current three-dimensional bounding boxes of the pedestrians and the three-dimensional bounding boxes distributed in the historical characteristic library according to the time sequence, and storing the pedestrian three-dimensional motion characteristics in the historical characteristic library.
The three-dimensional motion characteristic of the pedestrian is the position change characteristic of the pedestrian in the three-dimensional space, and is marked as F for convenient distinctiondisplacemenrIt is an important feature for performing data correlation in time series.
As shown in fig. 4, in this embodiment, a right-hand coordinate system is established, the three-dimensional bounding box is divided into three parts, namely a head part, an upper body part and a lower body part, according to the human body structure, the three-dimensional bounding box is input to the pedestrian three-dimensional motion feature extraction method, the position change feature of a person in a three-dimensional space is mainly extracted, and the pedestrian three-dimensional motion feature is output.
In this embodiment, the three-dimensional motion feature extraction method is composed of the following components:
step 2.2.1, extracting direction vectors: and extracting the moving direction of the pedestrian in the horizontal direction and the moving direction of the pedestrian in the vertical direction through the current three-dimensional bounding box and the historical three-dimensional bounding box.
Since the three-dimensional bounding box has a corresponding time stamp, the change in position of the three-dimensional bounding box based on the time distribution can result in a direction vector of the pedestrian. The plurality of three-dimensional bounding boxes may use only the first two bounding boxes for direction vector determination, or may use a plurality of pairs of three-dimensional bounding boxes, and take the average value, median value or other values of the plurality of direction vectors as the finally determined direction vectors.
Step 2.2.2, extracting the movement speed: and extracting the movement speed of the person in the horizontal direction and the movement speed of the person in the vertical direction through the current three-dimensional bounding box and the historical three-dimensional bounding box.
Similar to the direction vector extraction, the motion speed of the pedestrian is obtained according to the time difference and the position difference of the corresponding three-dimensional bounding boxes based on the three-dimensional bounding boxes of the time distribution, and the plurality of three-dimensional bounding boxes can be used for calculating the motion speed only by using the first two bounding boxes, or a plurality of pairs of three-dimensional bounding boxes can be used, and the average value, the median value or other values of the plurality of motion speeds are taken as the finally determined motion speed.
Step 2.2.3, extracting relative positions: and outputting the coordinates of the pedestrian in a three-dimensional coordinate system taking the camera as the center based on the current three-dimensional bounding box and the historical three-dimensional bounding box according to the mapping relation obtained after the camera is calibrated.
In determining the coordinates, each three-dimensional bounding box is equivalent to a point, and the coordinates of the point are obtained as the coordinates of the pedestrian, and the point can be the center point, a certain vertex or any point of the three-dimensional bounding box.
And 2.2.4, taking the direction vector, the motion speed and the relative position extracted in the step 2.2.1-2.2.3 as the three-dimensional motion characteristics of the pedestrian.
Since a plurality of three-dimensional bounding boxes of pedestrians usually exist in the historical feature library, when extracting three-dimensional motion features of pedestrians, feature matching is firstly performed on the current and historical three-dimensional bounding boxes (for example, a hungarian matching algorithm is adopted), and the historical three-dimensional bounding box which is unused and has the highest matching degree is taken for extracting the three-dimensional motion features of the pedestrians.
If the current three-dimensional bounding box is a new pedestrian three-dimensional bounding box newly entering the statistical range, and the matching fails to obtain the corresponding historical three-dimensional bounding box, setting the direction vector and the motion speed of the new pedestrian as default values (for example, the direction vector is none, the transportation speed is 0), and setting the coordinates of the current three-dimensional bounding box as the three-dimensional motion characteristics of the pedestrian.
And 2.3, predicting the three-dimensional motion characteristics of the pedestrian at the next moment based on the three-dimensional motion characteristics of the pedestrian and the three-dimensional motion characteristics of the pedestrian within the specified time in the historical characteristic library, and storing the three-dimensional motion characteristics of the pedestrian in the historical characteristic library.
The pedestrian three-dimensional motion characteristic mark at the next moment is FpredictedThe target loss problem caused by the fact that the pedestrians are shielded can be further solved by using the characteristics expressed by the motion trend of the pedestrians on the space-time through prediction of a track prediction algorithm; in the embodiment, the three-dimensional motion characteristics of the pedestrian at the next moment are predicted by using Kalman filtering.
Similar to the extraction of the three-dimensional motion features of the pedestrians, because the three-dimensional motion features of the pedestrians of a plurality of pedestrians usually exist in the historical feature library, when the three-dimensional motion features of the pedestrians are predicted, feature matching is firstly carried out on the three-dimensional motion features of the pedestrians in the current and historical states (for example, Hungary matching algorithm is adopted), and the three-dimensional motion features of the pedestrians in the historical states which are not used and have the highest matching degree are taken to predict the three-dimensional motion features of the pedestrians.
If the current three-dimensional pedestrian motion characteristic is the pedestrian three-dimensional motion characteristic of a new pedestrian entering the statistical range, the corresponding historical pedestrian three-dimensional motion characteristic cannot be obtained through the matching, and prediction is directly carried out on the basis of the current pedestrian three-dimensional motion characteristic. And 3, identifying the pedestrian based on the pedestrian characteristics in the historical characteristic library, as shown in fig. 5, wherein the method comprises the following steps:
and 3.1, calculating apparent feature distances one by one based on the current apparent features of the pedestrians and the historical apparent features of the pedestrians in the historical feature library, judging that the current apparent features of the pedestrians and the apparent features of the pedestrians in the historical feature library belong to the same pedestrian if the apparent feature distances are larger than an apparent threshold value, and determining the apparent feature distances as the apparent feature distances of the pedestrians.
In the embodiment, a pedestrian apparent feature matching method is used, and the current pedestrian apparent feature is calculated
Figure BDA0002836152720000091
And the apparent features of the pedestrians in the historical feature library
Figure BDA0002836152720000092
For example, the apparent characteristic distance is calculated by using the weighting of the mahalanobis distance and the cosine distance, and the coefficients are 0.02 and 0.98 respectively. The historical pedestrian appearance features in the embodiment are mainly taken as the pedestrian appearance features at the previous moment. In order to improve the matching result, in another embodiment, if the apparent features of the pedestrians belonging to the same pedestrian are determined, an apparent similar pedestrian list may be established, each pedestrian corresponds to one apparent similar pedestrian list, and the apparent feature distance is further distinguished based on the list.
For example, the current time has the appearance characteristics of two persons, namely A and A { t-1} (the appearance characteristics of the pedestrian at the previous time), the similarity between B and A { t-1} is high, even the similarity between B and A in a plurality of time periods in the appearance similar pedestrian list is high, but the similarity between B and B in a plurality of time periods in the appearance similar pedestrian list is higher, then B can be judged to be B. However, since the search method is time-consuming, it is generally used in a scenario where pedestrian recognition is more demanding.
And 3.2, calculating spatial feature distances one by one based on the current three-dimensional motion features of the pedestrians and the three-dimensional motion features of the pedestrians at the next moment predicted by the previous moment of each pedestrian in the historical feature library, if the spatial feature distances are larger than a spatial threshold, judging that the current three-dimensional motion features of the pedestrians and the three-dimensional motion features of the pedestrians at the next moment predicted by the previous moment in the historical feature library belong to the same pedestrian, and determining the current spatial feature distances as the spatial feature distances of the pedestrians.
The pedestrian three-dimensional motion characteristic of the next moment predicted at the previous moment is the predicted current pedestrian three-dimensional motion characteristic, the predicted and actual current pedestrian three-dimensional motion characteristics are matched, and the matching can be used as one of judgment bases for judging whether the pedestrian belongs to the same pedestrian, and the matching has reference because the change of the three-dimensional motion characteristic of the same pedestrian is not too large. In the embodiment, the Hungarian algorithm is used as a pedestrian three-dimensional motion feature matching method for calculation, and whether the pedestrians belong to the same pedestrian is judged.
Similar to the apparent similar pedestrian list, in another embodiment, if the three-dimensional motion characteristics of the pedestrians belonging to the same pedestrian are determined, a spatial similar pedestrian list may be established.
And 3.3, judging whether the pedestrian meets the motion mode of a pedestrian according with the same contract based on the current three-dimensional motion characteristics, apparent characteristic distance and spatial characteristic distance of the pedestrian and the historical three-dimensional motion characteristics of the pedestrian in a historical characteristic library, and outputting the motion mode matching degree as the motion mode matching degree of the pedestrian.
For the motion pattern of the pedestrian, the present embodiment focuses on the moving logic of the pedestrian in time sequence of changing speed and spatial position, and the moving logic includes, but is not limited to, common behaviors such as turning back, staying in place, jogging, squatting, and the like. Meanwhile, in consideration of the regular movement of the pedestrian, the present embodiment focuses on the change speed of the object within 3 seconds and the moving logic of the object within the shooting space. Since the camera acquires the optical image based on the preset interval, the pedestrian with a reasonable speed change can be determined as the same pedestrian.
Inputting the three-dimensional motion characteristics of the current pedestrian and the three-dimensional motion characteristics, apparent characteristic distances and spatial characteristic distances in the historical characteristic library into a motion pattern matching method, judging whether the behaviors of the pedestrian accord with common motion patterns of the pedestrian in public places, if the matching degree of the motion patterns is smaller than a motion threshold value, determining that the pedestrian belongs to the same pedestrian, and establishing a pedestrian list with similar motion patterns.
The calculated pedestrian motion pattern matching degree can be directly output based on a pre-trained neural network, and a preset matching rule can be predicted to be directly judged. The judgment of the former is relatively flexible, but the training of the neural network is required to be carried out based on a large number of samples, and the latter can be directly generated and used, is convenient for addition, deletion and modification, but has relatively low flexibility, and a proper mode can be selected according to actual requirements.
In one embodiment, based on actual observation and statistics, a matching rule (a specific probability value is omitted) is established as shown in table 1, and the table represents the probability of the transition from the behavior pattern of the previous stage to the corresponding behavior pattern of the current stage.
TABLE 1 probability of transition from the behavior pattern of the previous stage to the corresponding behavior pattern of the current stage
Figure BDA0002836152720000111
When the motion pattern matching degree is carried out based on the table 1, historical pedestrian three-dimensional motion features of pedestrians corresponding to the apparent feature distances are taken, historical pedestrian three-dimensional motion features of pedestrians corresponding to the spatial feature distances are taken, if the pedestrians corresponding to the two taken historical pedestrian three-dimensional motion features are not the same pedestrian, the matching is given up, if the pedestrians are the same pedestrian, the behavior pattern of the pedestrian at the previous stage and the behavior pattern of the pedestrian at the current stage are judged according to the taken historical pedestrian three-dimensional motion features and the current pedestrian three-dimensional motion features, and the probability value can be obtained by looking up the table to serve as the motion pattern matching degree.
It should be noted that the behavior pattern of a stage is determined by at least two three-dimensional motion characteristics of the pedestrian, and since one three-dimensional motion characteristic of the pedestrian has a direction vector, a motion speed and a coordinate, the current stage can be determined to be forward, turn back or turn by the change of the two direction vectors, and can be further distinguished to be forward or stop by combining the change of the coordinate, and further distinguished to be forward or accelerated by combining the motion speed.
Of course, the table is a preferred matching rule adopted in the embodiment, and may be further optimized in actual use, for example, refining a turn into a left turn or a right turn, and the probability value in the table may also be updated according to the probability counted in actual use, so as to improve the pedestrian recognition rate.
And 3.4, performing weighted calculation (namely inputting the weighted calculation into a weighted calculator) on the apparent characteristic distance, the spatial characteristic distance and the motion pattern matching degree which belong to the same pedestrian to obtain a matching result of the pedestrian in the current three-dimensional bounding box and the pedestrian in the historical characteristic library, wherein the matching result comprises matching success or matching failure, and the matching result further comprises pedestrian information obtained by matching when the matching is successful.
In this embodiment, the weights of the apparent feature distance, the spatial feature distance, and the motion pattern matching degree are 0.6, 0.2, and 0.2, respectively, and since the apparent feature is the most intuitive feature for distinguishing different pedestrians, the apparent feature distance is set to have the highest weight in this embodiment. In actual use, the weight can be adjusted, for example, the weight of the motion pattern matching degree is increased, so as to avoid misjudgment caused by two persons with too similar apparent features.
The matching failure in the final matching result indicates that the characteristic of the current pedestrian has no history, namely the pedestrian is a pedestrian newly entering the statistical range; and the successful matching indicates that the characteristic of the current pedestrian has a history record, so that the pedestrian information obtained by matching is output to associate the new characteristic and the history characteristic of the same pedestrian. The pedestrian information may be a unique identifier (e.g., an ID value), a spatial location, a time, etc.
And 4, marking the pedestrian state of the pedestrian corresponding to the three-dimensional bounding box as a pedestrian state which is successfully matched for the first time, lost, re-successfully matched after being lost, successfully matched continuously or out of the shooting range according to the matching result of the current time and the historical matching result.
As shown in fig. 6, a specific matching method provided in this embodiment may be:
if the pedestrian features are successfully extracted (namely the identification at the current moment is successful), but the matching result is that the matching is failed (namely no history exists), marking the current pedestrian state as the first matching is successful;
if the same pedestrian is not matched to the history matching result for M times (for example, 50 times (10 seconds) by 5 times/second) continuously (for example, the current identification fails and the identification also fails at the last moment, or the current identification fails and the number of times of continuous identification fails is not greater than a threshold), marking the state of the pedestrian as lost;
if the pedestrian marked as lost is successfully re-matched in the matching result at this time (for example, the current identification is successful, a history record exists, but the matching at the last time is failed), updating the state of the pedestrian as the state of the lost pedestrian which is successfully re-matched;
if the same pedestrian is matched to the history matching result for L times (for example, 50 times (10 seconds) by 5 times/second) continuously (for example, the current time is successfully identified, a history record exists, and the last time is also successfully matched), updating the state of the pedestrian to be the successful continuous matching;
if the same pedestrian is not matched to the history matching result for N times (for example, 150 times (10 seconds) by 15 times/second), the state of the pedestrian is marked as the image capturing range, and M is less than N.
If the state mark of the current pedestrian is successful in primary matching, new pedestrian information is distributed to the pedestrian in the historical characteristic library, the pedestrian characteristic of the pedestrian is associated with the newly distributed pedestrian information, and the pedestrian can be used as historical data for identifying and tracking the pedestrian at the next moment.
In another implementation, as shown in fig. 7, there is provided a people flow statistics system, comprising:
the pedestrian detection module is used for acquiring an optical image, detecting a pedestrian in the optical image and outputting a three-dimensional bounding box of the pedestrian and a corresponding timestamp;
the feature extraction module is used for acquiring pedestrian features based on the optical images and the three-dimensional bounding box, and specifically executes the following steps:
a. extracting the human body shape and the features of the pedestrians in the optical image to serve as the pedestrian apparent features of each pedestrian, and storing the pedestrian apparent features in a historical feature library;
b. extracting the pedestrian three-dimensional motion characteristics of each pedestrian based on the current three-dimensional bounding boxes of the pedestrians and the three-dimensional bounding boxes distributed in the historical characteristic library according to the time sequence, and storing the pedestrian three-dimensional motion characteristics into the historical characteristic library;
c. predicting the three-dimensional motion characteristics of the pedestrian at the next moment based on the three-dimensional motion characteristics of the pedestrian and the three-dimensional motion characteristics of the pedestrian within the specified time in the historical characteristic library, and storing the three-dimensional motion characteristics of the pedestrian into the historical characteristic library;
the pedestrian recognition module is used for recognizing pedestrians based on pedestrian features in the historical feature library, and specifically executes the following steps:
a. calculating apparent feature distances one by one based on the current apparent features of the pedestrians and the historical apparent features of the pedestrians in the historical feature library, if the apparent feature distances are larger than an apparent threshold value, judging that the current apparent features of the pedestrians and the apparent features of the pedestrians in the historical feature library belong to the same pedestrian, and determining the current apparent feature distances as the apparent feature distances of the pedestrians;
b. calculating a spatial feature distance one by one based on the current three-dimensional motion feature of the pedestrian and the three-dimensional motion feature of the pedestrian at the next moment predicted by each pedestrian at the previous moment in the historical feature library, if the spatial feature distance is greater than a spatial threshold, judging that the current three-dimensional motion feature of the pedestrian and the three-dimensional motion feature of the pedestrian at the next moment predicted by the pedestrian at the previous moment in the historical feature library belong to the same pedestrian, and determining the current spatial feature distance as the spatial feature distance of the pedestrian;
c. judging whether a pedestrian motion mode is matched with a pedestrian on the basis of the current three-dimensional motion characteristic, the apparent characteristic distance, the spatial characteristic distance of the pedestrian and the historical three-dimensional motion characteristic of the pedestrian in a historical characteristic library, and outputting a motion mode matching degree as the motion mode matching degree of the pedestrian;
d. weighting and calculating the apparent characteristic distance, the spatial characteristic distance and the motion pattern matching degree which belong to the same pedestrian to obtain a matching result of the pedestrian in the current three-dimensional bounding box and the pedestrian in the historical characteristic library, wherein the matching result comprises successful matching or failed matching, and the pedestrian information obtained by matching is also included when the matching is successful;
the pedestrian marking module is used for marking the pedestrian state of the pedestrian corresponding to the three-dimensional bounding box as a pedestrian state which is successfully matched for the first time, lost or re-matched after being lost, successfully matched for the continuous time or out of a shooting range according to the matching result of the current time and the historical matching result;
and the pedestrian flow counting module is used for counting the pedestrian flow in the counting range corresponding to the optical image within the preset time according to the pedestrian state.
For specific limitations in the pedestrian flow statistical system, refer to the above specific limitations for the pedestrian identification method in the public place, and are not described herein again.
In a preferred embodiment, the detecting the pedestrian in the optical image and outputting the three-dimensional bounding box of the pedestrian perform the following operations:
calibrating a camera for acquiring an optical image to obtain a mapping relation between pixels in the optical image and the distance of the camera;
detecting a pedestrian in the optical image, and acquiring a two-dimensional surrounding frame of the pedestrian in the optical image;
and obtaining the three-dimensional bounding box of the pedestrian based on the two-dimensional bounding box and the mapping relation.
In other embodiments, the camera calibration device may be used as a part independent from the people flow statistical system of this embodiment, and send the calibrated device external parameters and the mapping relationship to the parameter management module of the people flow statistical system of this embodiment.
It should be noted that, in this embodiment, people flow statistics is based on an optical image process, that is, the people flow statistics system further includes a video acquisition module, and the video acquisition module is connected to a peripheral video acquisition device, and after acquiring a real-time video within a statistical range, sends an optical picture of each frame to the pedestrian detection module.
In another embodiment, the pedestrian three-dimensional motion feature of each pedestrian is extracted based on the current three-dimensional bounding box of the pedestrian and three-dimensional bounding boxes distributed in time series in the historical feature library, and the following operations are performed:
extracting a direction vector: extracting the moving direction of the pedestrian in the horizontal direction and the moving direction of the pedestrian in the vertical direction through the current three-dimensional bounding box and the historical three-dimensional bounding box;
extracting the movement speed: extracting the movement speed of a person in the horizontal direction and the movement speed of the person in the vertical direction through the current three-dimensional bounding box and the historical three-dimensional bounding box;
extracting relative positions: outputting coordinates of the pedestrian in a three-dimensional coordinate system with the camera as the center based on the current three-dimensional bounding box and the historical three-dimensional bounding box according to a mapping relation obtained after the camera is calibrated;
feature integration: and taking the extracted direction vector, the motion speed and the relative position as the three-dimensional motion characteristics of the pedestrian.
In another embodiment, the marking of the pedestrian state of the pedestrian corresponding to the three-dimensional bounding box as successful initial matching, missing, successful re-matching after missing, successful continuous matching or exiting of the shooting range according to the matching result of this time and the historical matching result is performed as follows:
if the pedestrian features are successfully extracted, but the matching result is matching failure, marking the current pedestrian state as successful primary matching;
if the same pedestrian in the history matching result is not matched for M times, marking the state of the pedestrian as lost;
if the matching result of the pedestrian marked as lost is successfully re-matched in the current matching result, updating the state of the pedestrian as the state of the lost pedestrian which is successfully re-matched;
if the same pedestrian in the history matching result is matched for L times, updating the state of the pedestrian as successful continuous matching;
and if the same pedestrian in the history matching result is not matched for N times, marking the state of the pedestrian as the exit shooting range, and M < N.
In another embodiment, if the status of the current pedestrian is marked as successful in the initial matching, new pedestrian information is assigned to the pedestrian in the historical feature library, and the pedestrian feature of the pedestrian is associated with the newly assigned pedestrian information.
The technical features of the embodiments described above may be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the embodiments described above are not described, but should be considered as being within the scope of the present specification as long as there is no contradiction between the combinations of the technical features.
The above-mentioned embodiments only express several embodiments of the present invention, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the inventive concept, which falls within the scope of the present invention. Therefore, the protection scope of the present patent shall be subject to the appended claims.

Claims (10)

1. A public place pedestrian recognition method is characterized by comprising the following steps:
step 1, acquiring an optical image, detecting a pedestrian in the optical image, and outputting a three-dimensional bounding box of the pedestrian and a corresponding timestamp;
step 2, acquiring pedestrian characteristics based on the optical image and the three-dimensional bounding box, and the method comprises the following steps:
step 2.1, extracting the human body shape and the features of the pedestrians in the optical image to serve as the pedestrian apparent features of each pedestrian, and storing the pedestrian apparent features in a historical feature library;
2.2, extracting the pedestrian three-dimensional motion characteristics of each pedestrian based on the current three-dimensional bounding boxes of the pedestrians and the three-dimensional bounding boxes distributed in the historical characteristic library according to the time sequence, and storing the pedestrian three-dimensional motion characteristics into the historical characteristic library;
step 2.3, predicting the three-dimensional motion characteristics of the pedestrian at the next moment based on the three-dimensional motion characteristics of the pedestrian and the three-dimensional motion characteristics of the pedestrian within the specified time in the historical characteristic library, and storing the three-dimensional motion characteristics of the pedestrian into the historical characteristic library;
and 3, identifying the pedestrians based on the pedestrian characteristics in the historical characteristic library, wherein the identification comprises the following steps:
step 3.1, calculating apparent feature distances one by one based on the current apparent features of the pedestrians and the historical apparent features of the pedestrians in the historical feature library, if the apparent feature distances are larger than an apparent threshold value, judging that the current apparent features of the pedestrians and the apparent features of the pedestrians in the historical feature library belong to the same pedestrian, and determining the current apparent feature distances as the apparent feature distances of the pedestrians;
step 3.2, calculating spatial feature distances one by one based on the current three-dimensional motion features of the pedestrians and the three-dimensional motion features of the pedestrians at the next moment predicted by the previous moment of each pedestrian in the historical feature library, if the spatial feature distances are larger than a spatial threshold, judging that the current three-dimensional motion features of the pedestrians and the three-dimensional motion features of the pedestrians at the next moment predicted by the previous moment in the historical feature library belong to the same pedestrian, and determining the current spatial feature distances as the spatial feature distances of the pedestrians;
3.3, judging whether the pedestrian meets the motion mode of a pedestrian according with the same contract based on the current three-dimensional motion characteristics, apparent characteristic distance and spatial characteristic distance of the pedestrian and the historical three-dimensional motion characteristics of the pedestrian in a historical characteristic library, and outputting the motion mode matching degree as the motion mode matching degree of the pedestrian;
step 3.4, performing weighted calculation on the apparent characteristic distance, the spatial characteristic distance and the motion pattern matching degree of the same pedestrian to obtain a matching result of the pedestrian in the current three-dimensional bounding box and the pedestrian in the historical characteristic library, wherein the matching result comprises successful matching or failed matching, and the pedestrian information obtained by matching is also included when the matching is successful;
and 4, marking the pedestrian state of the pedestrian corresponding to the three-dimensional bounding box as a pedestrian state which is successfully matched for the first time, lost, re-successfully matched after being lost, successfully matched continuously or out of the shooting range according to the matching result of the current time and the historical matching result.
2. The public place pedestrian recognition method according to claim 1, wherein the detecting a pedestrian in the optical image, outputting a three-dimensional bounding box of the pedestrian, comprises:
calibrating a camera for acquiring an optical image to obtain a mapping relation between pixels in the optical image and the distance of the camera;
detecting a pedestrian in the optical image, and acquiring a two-dimensional surrounding frame of the pedestrian in the optical image;
and obtaining the three-dimensional bounding box of the pedestrian based on the two-dimensional bounding box and the mapping relation.
3. The method for identifying pedestrians in public places according to claim 2, wherein the step of extracting the pedestrian three-dimensional motion characteristics of each pedestrian based on the current three-dimensional bounding box of the pedestrian and the three-dimensional bounding boxes distributed in time series in the historical characteristic library comprises the following steps:
step 2.2.1, extracting direction vectors: extracting the moving direction of the pedestrian in the horizontal direction and the moving direction of the pedestrian in the vertical direction through the current three-dimensional bounding box and the historical three-dimensional bounding box;
step 2.2.2, extracting the movement speed: extracting the movement speed of a person in the horizontal direction and the movement speed of the person in the vertical direction through the current three-dimensional bounding box and the historical three-dimensional bounding box;
step 2.2.3, extracting relative positions: outputting coordinates of the pedestrian in a three-dimensional coordinate system with the camera as the center based on the current three-dimensional bounding box and the historical three-dimensional bounding box according to a mapping relation obtained after the camera is calibrated;
and 2.2.4, taking the direction vector, the motion speed and the relative position extracted in the step 2.2.1-2.2.3 as the three-dimensional motion characteristics of the pedestrian.
4. The method for identifying pedestrians in public places according to claim 1, wherein the method for marking the pedestrian state of the pedestrian corresponding to the three-dimensional bounding box according to the matching result of the current time and the historical matching result is that the pedestrian state is successfully matched for the first time, lost, successfully matched again after being lost, successfully matched continuously or out of a shooting range, and comprises the following steps:
if the pedestrian features are successfully extracted, but the matching result is matching failure, marking the current pedestrian state as successful primary matching;
if the same pedestrian in the history matching result is not matched for M times, marking the state of the pedestrian as lost;
if the matching result of the pedestrian marked as lost is successfully re-matched in the current matching result, updating the state of the pedestrian as the state of the lost pedestrian which is successfully re-matched;
if the same pedestrian in the history matching result is matched for L times, updating the state of the pedestrian as successful continuous matching;
and if the same pedestrian in the history matching result is not matched for N times, marking the state of the pedestrian as the exit shooting range, and M < N.
5. The public place pedestrian recognition method according to claim 1, wherein if the status of the current pedestrian is marked as the initial matching success, new pedestrian information is assigned to the pedestrian in the history feature library, and the pedestrian feature of the pedestrian is associated with the newly assigned pedestrian information.
6. A people flow statistical system, characterized in that the people flow statistical system comprises:
the pedestrian detection module is used for acquiring an optical image, detecting a pedestrian in the optical image and outputting a three-dimensional bounding box of the pedestrian and a corresponding timestamp;
the feature extraction module is used for acquiring pedestrian features based on the optical images and the three-dimensional bounding box, and specifically executes the following steps:
a. extracting the human body shape and the features of the pedestrians in the optical image to serve as the pedestrian apparent features of each pedestrian, and storing the pedestrian apparent features in a historical feature library;
b. extracting the pedestrian three-dimensional motion characteristics of each pedestrian based on the current three-dimensional bounding boxes of the pedestrians and the three-dimensional bounding boxes distributed in the historical characteristic library according to the time sequence, and storing the pedestrian three-dimensional motion characteristics into the historical characteristic library;
c. predicting the three-dimensional motion characteristics of the pedestrian at the next moment based on the three-dimensional motion characteristics of the pedestrian and the three-dimensional motion characteristics of the pedestrian within the specified time in the historical characteristic library, and storing the three-dimensional motion characteristics of the pedestrian into the historical characteristic library;
the pedestrian recognition module is used for recognizing pedestrians based on pedestrian features in the historical feature library, and specifically executes the following steps:
a. calculating apparent feature distances one by one based on the current apparent features of the pedestrians and the historical apparent features of the pedestrians in the historical feature library, if the apparent feature distances are larger than an apparent threshold value, judging that the current apparent features of the pedestrians and the apparent features of the pedestrians in the historical feature library belong to the same pedestrian, and determining the current apparent feature distances as the apparent feature distances of the pedestrians;
b. calculating a spatial feature distance one by one based on the current three-dimensional motion feature of the pedestrian and the three-dimensional motion feature of the pedestrian at the next moment predicted by each pedestrian at the previous moment in the historical feature library, if the spatial feature distance is greater than a spatial threshold, judging that the current three-dimensional motion feature of the pedestrian and the three-dimensional motion feature of the pedestrian at the next moment predicted by the pedestrian at the previous moment in the historical feature library belong to the same pedestrian, and determining the current spatial feature distance as the spatial feature distance of the pedestrian;
c. judging whether a pedestrian motion mode is matched with a pedestrian on the basis of the current three-dimensional motion characteristic, the apparent characteristic distance, the spatial characteristic distance of the pedestrian and the historical three-dimensional motion characteristic of the pedestrian in a historical characteristic library, and outputting a motion mode matching degree as the motion mode matching degree of the pedestrian;
d. weighting and calculating the apparent characteristic distance, the spatial characteristic distance and the motion pattern matching degree which belong to the same pedestrian to obtain a matching result of the pedestrian in the current three-dimensional bounding box and the pedestrian in the historical characteristic library, wherein the matching result comprises successful matching or failed matching, and the pedestrian information obtained by matching is also included when the matching is successful;
the pedestrian marking module is used for marking the pedestrian state of the pedestrian corresponding to the three-dimensional bounding box as a pedestrian state which is successfully matched for the first time, lost or re-matched after being lost, successfully matched for the continuous time or out of a shooting range according to the matching result of the current time and the historical matching result;
and the pedestrian flow counting module is used for counting the pedestrian flow in the counting range corresponding to the optical image within the preset time according to the pedestrian state.
7. The people flow statistical system of claim 6, wherein the detecting of the pedestrian in the optical image, outputting a three-dimensional bounding box of the pedestrian, performs the following operations:
calibrating a camera for acquiring an optical image to obtain a mapping relation between pixels in the optical image and the distance of the camera;
detecting a pedestrian in the optical image, and acquiring a two-dimensional surrounding frame of the pedestrian in the optical image;
and obtaining the three-dimensional bounding box of the pedestrian based on the two-dimensional bounding box and the mapping relation.
8. The people flow statistical system according to claim 7, wherein the pedestrian three-dimensional motion feature of each pedestrian is extracted based on the current three-dimensional bounding box of the pedestrian and three-dimensional bounding boxes distributed in time series in the historical feature library, and the following operations are performed:
extracting a direction vector: extracting the moving direction of the pedestrian in the horizontal direction and the moving direction of the pedestrian in the vertical direction through the current three-dimensional bounding box and the historical three-dimensional bounding box;
extracting the movement speed: extracting the movement speed of a person in the horizontal direction and the movement speed of the person in the vertical direction through the current three-dimensional bounding box and the historical three-dimensional bounding box;
extracting relative positions: outputting coordinates of the pedestrian in a three-dimensional coordinate system with the camera as the center based on the current three-dimensional bounding box and the historical three-dimensional bounding box according to a mapping relation obtained after the camera is calibrated;
feature integration: and taking the extracted direction vector, the motion speed and the relative position as the three-dimensional motion characteristics of the pedestrian.
9. The people flow statistical system according to claim 6, wherein the pedestrian state of the pedestrian marked by the three-dimensional bounding box according to the matching result of this time and the historical matching result is the initial matching success, the loss, the re-matching success after the loss, the continuous matching success or the camera shooting range, and the following operations are executed:
if the pedestrian features are successfully extracted, but the matching result is matching failure, marking the current pedestrian state as successful primary matching;
if the same pedestrian in the history matching result is not matched for M times, marking the state of the pedestrian as lost;
if the matching result of the pedestrian marked as lost is successfully re-matched in the current matching result, updating the state of the pedestrian as the state of the lost pedestrian which is successfully re-matched;
if the same pedestrian in the history matching result is matched for L times, updating the state of the pedestrian as successful continuous matching;
and if the same pedestrian in the history matching result is not matched for N times, marking the state of the pedestrian as the exit shooting range, and M < N.
10. The pedestrian flow statistical system of claim 6, wherein if the status of the current pedestrian is marked as a successful initial match, new pedestrian information is assigned to the pedestrian in the historical feature library and the pedestrian feature of the pedestrian is associated with the newly assigned pedestrian information.
CN202011477711.XA 2020-12-15 2020-12-15 Pedestrian recognition method and people stream statistics system in public place Active CN112446355B (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CN202011477711.XA CN112446355B (en) 2020-12-15 2020-12-15 Pedestrian recognition method and people stream statistics system in public place
PCT/CN2020/137803 WO2022126668A1 (en) 2020-12-15 2020-12-19 Method for pedestrian identification in public places and human flow statistics system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202011477711.XA CN112446355B (en) 2020-12-15 2020-12-15 Pedestrian recognition method and people stream statistics system in public place

Publications (2)

Publication Number Publication Date
CN112446355A true CN112446355A (en) 2021-03-05
CN112446355B CN112446355B (en) 2023-10-17

Family

ID=74739432

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202011477711.XA Active CN112446355B (en) 2020-12-15 2020-12-15 Pedestrian recognition method and people stream statistics system in public place

Country Status (2)

Country Link
CN (1) CN112446355B (en)
WO (1) WO2022126668A1 (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113011329A (en) * 2021-03-19 2021-06-22 陕西科技大学 Pyramid network based on multi-scale features and dense crowd counting method

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106650562A (en) * 2016-06-14 2017-05-10 西安电子科技大学 Online continuous human behavior identification method based on Kinect
CN109829476A (en) * 2018-12-27 2019-05-31 青岛中科慧畅信息科技有限公司 End-to-end three-dimension object detection method based on YOLO
CN110490901A (en) * 2019-07-15 2019-11-22 武汉大学 The pedestrian detection tracking of anti-attitudes vibration
US10839203B1 (en) * 2016-12-27 2020-11-17 Amazon Technologies, Inc. Recognizing and tracking poses using digital imagery captured from multiple fields of view
CN111968235A (en) * 2020-07-08 2020-11-20 杭州易现先进科技有限公司 Object attitude estimation method, device and system and computer equipment

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111709974B (en) * 2020-06-22 2022-08-02 苏宁云计算有限公司 Human body tracking method and device based on RGB-D image

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106650562A (en) * 2016-06-14 2017-05-10 西安电子科技大学 Online continuous human behavior identification method based on Kinect
US10839203B1 (en) * 2016-12-27 2020-11-17 Amazon Technologies, Inc. Recognizing and tracking poses using digital imagery captured from multiple fields of view
CN109829476A (en) * 2018-12-27 2019-05-31 青岛中科慧畅信息科技有限公司 End-to-end three-dimension object detection method based on YOLO
CN110490901A (en) * 2019-07-15 2019-11-22 武汉大学 The pedestrian detection tracking of anti-attitudes vibration
CN111968235A (en) * 2020-07-08 2020-11-20 杭州易现先进科技有限公司 Object attitude estimation method, device and system and computer equipment

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
李随伟;李刚柱;: "行人动态实时监测***的设计与实现", 信息技术, no. 05, pages 15 - 20 *

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113011329A (en) * 2021-03-19 2021-06-22 陕西科技大学 Pyramid network based on multi-scale features and dense crowd counting method
CN113011329B (en) * 2021-03-19 2024-03-12 陕西科技大学 Multi-scale feature pyramid network-based and dense crowd counting method

Also Published As

Publication number Publication date
WO2022126668A1 (en) 2022-06-23
CN112446355B (en) 2023-10-17

Similar Documents

Publication Publication Date Title
CN108416250B (en) People counting method and device
US10417503B2 (en) Image processing apparatus and image processing method
US9646212B2 (en) Methods, devices and systems for detecting objects in a video
CN112836640B (en) Single-camera multi-target pedestrian tracking method
CN105745687B (en) Context aware Moving target detection
WO2012127815A1 (en) Moving object detecting apparatus and moving object detecting method
CN110874583A (en) Passenger flow statistics method and device, storage medium and electronic equipment
JP2016099941A (en) System and program for estimating position of object
JP2011100175A (en) Device and program for deciding personal action
WO2010030814A2 (en) Pixel-level based micro-feature extraction
US20220366570A1 (en) Object tracking device and object tracking method
CN112613668A (en) Scenic spot dangerous area management and control method based on artificial intelligence
CN111652900A (en) Scene flow-based passenger flow counting method, system, equipment and storage device
CN111291646A (en) People flow statistical method, device, equipment and storage medium
US20170053172A1 (en) Image processing apparatus, and image processing method
CN114581990A (en) Intelligent running test method and device
CN111950507B (en) Data processing and model training method, device, equipment and medium
CN112446355B (en) Pedestrian recognition method and people stream statistics system in public place
CN116824641B (en) Gesture classification method, device, equipment and computer storage medium
JP2021149687A (en) Device, method and program for object recognition
CN111695404A (en) Pedestrian falling detection method and device, electronic equipment and storage medium
Seer et al. Kinects and human kinetics: a new approach for studying crowd behavior
CN106981072B (en) Based on the training sample selection method of multi-instance learning thought in target following
CN113763418B (en) Multi-target tracking method based on head and shoulder detection
CN113536891B (en) Pedestrian traffic statistical method, storage medium and electronic equipment

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant