CN112541440A

CN112541440A - Subway pedestrian flow network fusion method based on video pedestrian recognition and pedestrian flow prediction method

Info

Publication number: CN112541440A
Application number: CN202011485904.XA
Authority: CN
Inventors: 徐超; 高思斌; 李少利; 李永强; 戴李杰
Original assignee: CETHIK Group Ltd
Current assignee: CETHIK Group Ltd
Priority date: 2020-12-16
Filing date: 2020-12-16
Publication date: 2021-03-23
Anticipated expiration: 2040-12-16
Also published as: WO2022126669A1; CN112541440B

Abstract

The invention discloses a fusion method of subway pedestrian flow networks and a pedestrian flow prediction method based on video pedestrian identification. And 4, deducing the people flow change of the whole traffic network by using the graph neural network so as to analyze and predict each station, the number of people flows and the direction of people flow. The method has the advantages that deeper pedestrian flow analysis of the subway station entrance and exit is realized, resource scheduling of each station and the entrance and exit is facilitated, and mutual influences of the overground pedestrian flow and the underground pedestrian flow can be judged in advance in time, so that traffic early warning is performed on the ground or underground in time, traffic jam is avoided, station security measures are deployed in advance, and the like.

Description

Subway pedestrian flow network fusion method based on video pedestrian recognition and pedestrian flow prediction method

Technical Field

The invention belongs to the technical field of smart cities, and particularly relates to a subway pedestrian flow network fusion method and a pedestrian flow prediction method based on video pedestrian recognition.

Background

The application of video monitoring is increasingly applied in the field of digital security prevention, people counting through videos is increasingly important, and people can be effectively mobilized, resources can be allocated and better security guarantee can be provided by utilizing people flow counting data in stations, tourist attractions, exhibition areas, commercial streets and other places.

The existing subway people flow prediction is generally carried out based on the card swiping data of an inlet and an outlet of each station, and the obtained result can only obtain the people flow prediction condition of the inlet and the outlet of the station. The passenger flow prediction model of the subway station is generally constructed by analyzing historical card swiping data of the subway station and a road network map, and the future passenger flow change of the station is predicted, for example, the number of people entering and leaving the station in each time period by taking 10 minutes as a unit from 00 hours to 24 hours in the future is predicted. In addition, the pedestrian flow statistical scheme is mainly used for monitoring the pedestrian flow density of the subway carriage in real time through infrared rays, a camera, communication data and the like. For example, the image method generally obtains crowd distribution images under different density conditions by using different degrees of shielding of a camera-acquired image light source by different crowd densities, performs image analysis and processing on the crowd distribution images, performs front-to-back comparison on the crowd distribution images, and finally obtains the condition of the passenger flow density of the subway carriage comprehensively.

However, the existing schemes have the following problems: each station generally has a plurality of entrances and exits, and the access of each entrance and exit has various conditions, so that the current scheme cannot analyze the flow of people entering and exiting the subway station in a subdivision direction, thus causing that the overground and underground traffic networks cannot be deeply fused, and the situation that the resources of each entrance and exit of the station cannot be reasonably distributed exists; meanwhile, the influence of the pedestrian flow entering and exiting the station on the ground traffic cannot be obtained.

Disclosure of Invention

The invention aims to provide a subway pedestrian flow network fusion method and a pedestrian flow prediction method based on video pedestrian identification, which are used for communicating an overground traffic route and an underground traffic route with a subway route, refining the pedestrian flow and the outbound or inbound direction of pedestrian flow of each station and improving the traffic prediction accuracy.

In order to achieve the purpose, the technical scheme adopted by the invention is as follows:

a fusion method of subway people flow network based on video pedestrian recognition is used for realizing fusion statistics of subway and ground people flow to assist traffic early warning, and comprises the following steps:

step 1, receiving monitoring images of each entrance and exit of a subway station, wherein the monitoring images are acquired by image acquisition equipment arranged at each entrance and exit;

step 2, extracting pedestrian target coordinate frame information and pedestrian target characteristic information in the monitored image, wherein the pedestrian target characteristic information comprises pedestrian characteristics, pedestrian in-and-out states, and pedestrian out-of-station or in-station directions;

step 3, based on the monitoring image of the same image acquisition equipment, carrying out similarity calculation according to the pedestrian target coordinate frame information and the pedestrian target characteristic information to obtain a pedestrian track for the same pedestrian;

step 4, carrying out similarity matching on pedestrian target characteristic information based on pedestrian tracks corresponding to different image acquisition devices, combining the successfully matched pedestrian tracks, and updating the pedestrian tracks corresponding to pedestrians;

step 5, acquiring subway lines, subway stations, entrances and exits of the stations and overground traffic lines corresponding to the entrances and exits in the designated area, and fusing and constructing a subway traffic network map in the designated area;

step 6, according to the latest pedestrian track in a preset time period, counting the total incoming and outgoing pedestrian flow of each station and the incoming and outgoing pedestrian flow on the traffic route corresponding to each entrance and exit in each station;

and 7, superposing the total incoming and outgoing pedestrian flow of each station of the subway station and the incoming and outgoing pedestrian flow on the traffic route corresponding to each entrance and exit in each station on the basis of the subway traffic network map to obtain a pedestrian flow mobile network map integrating subway and ground pedestrian flow.

Several alternatives are provided below, but not as an additional limitation to the above general solution, but merely as a further addition or preference, each alternative being combinable individually for the above general solution or among several alternatives without technical or logical contradictions.

Preferably, the obtaining of the pedestrian trajectory for the same pedestrian based on the monitoring image of the same image acquisition device by performing similarity calculation according to the pedestrian target coordinate frame information and the pedestrian target feature information includes:

step 3.1, acquiring pedestrian target coordinate frame information and pedestrian target characteristic information of the current image acquisition equipment in the current monitoring image;

step 3.2, judging whether a tracking track set corresponding to the image acquisition equipment is empty, wherein the tracking track set is used for storing the pedestrian track of the pedestrian, and if the tracking track set is not empty, executing step 3.3; otherwise, directly adding the pedestrian target coordinate frame information and the pedestrian target characteristic information acquired this time into the tracking track set and finishing;

3.3, obtaining estimated target coordinate frame information by adopting unscented Kalman filtering based on the pedestrian track in the tracking track set;

step 3.4, calculating the coordinate frame similarity between the current pedestrian target and the stored pedestrian target one by one according to the pedestrian target coordinate frame information and the estimated target coordinate frame information, calculating the feature similarity between the current pedestrian target and the stored pedestrian target one by one based on the pedestrian target feature information of the pedestrian track and the pedestrian target feature information in the current monitoring image, and obtaining the similarity between the current pedestrian target and the stored pedestrian target based on the weighted summation of the coordinate frame similarity and the feature similarity;

3.5, matching the pedestrian tracks in the tracking track set and the pedestrian targets obtained at this time by adopting a Hungarian matching algorithm based on the similarity between the current pedestrian target and the stored pedestrian targets;

step 3.6, if the unsuccessfully matched pedestrian target exists, directly adding the pedestrian target coordinate frame information and the pedestrian target characteristic information corresponding to the pedestrian target into the tracking track set, and marking the tracking track set as a new track; if the pedestrian track and the pedestrian target are successfully matched, updating the pedestrian track of the pedestrian according to the pedestrian target coordinate frame information and the pedestrian target characteristic information corresponding to the pedestrian target; if the pedestrian track marked as the new track exists in the tracking track set and is successfully matched for multiple times, removing the new mark of the pedestrian track; if continuous multiframe unsuccessfully matched pedestrian tracks exist in the tracking track set, the pedestrian tracks are marked as leaving tracks if the pedestrian targets are considered to leave the monitoring range of the current image acquisition equipment; and if the pedestrian track marked as the departure track is not successfully matched within the specified time threshold, the pedestrian track is considered to be finished, and the pedestrian track is deleted from the tracking track set.

Preferably, the performing similarity matching of the pedestrian target feature information based on the pedestrian trajectories corresponding to different image acquisition devices, combining the pedestrian trajectories successfully matched, and updating the pedestrian trajectories of the corresponding pedestrians includes:

step 4.1, a tracking track set corresponding to the image acquisition equipment is taken, and the pedestrian tracks marked as the new tracks in the tracking track set are subjected to similarity calculation with the pedestrian tracks marked as the leaving tracks in the tracking track sets corresponding to the other image acquisition equipment one by one;

4.2, if the similarity is greater than a preset threshold value, indicating that the two pedestrian tracks are successfully matched;

and 4.3, combining the two successfully matched pedestrian tracks to obtain a new pedestrian track of the pedestrian, and replacing the corresponding pedestrian track in the tracking track set where the new track is located by using the new pedestrian track.

The invention also provides a people stream prediction method for carrying out people stream prediction to assist traffic early warning based on fusion of subway and ground people stream, which comprises the following steps:

acquiring a pedestrian flow mobile network diagram in a specified time period by using a subway pedestrian flow network fusion method based on video pedestrian identification;

predicting the total incoming and outgoing predicted pedestrian volume of each station in a specified time period in the future by utilizing a graph neural network based on the pedestrian flow mobile network diagram;

obtaining an entrance and exit proportion mean value of the traffic route corresponding to each entrance and exit of each station based on the entrance and exit pedestrian flow on the traffic route corresponding to each entrance and exit of each station in the pedestrian flow mobile network diagram;

and distributing the total inbound and outbound predicted pedestrian volume of each station according to the inbound and outbound proportion mean value to obtain the inbound and outbound predicted pedestrian volume on the traffic route corresponding to each entrance and exit of each station.

Preferably, the obtaining of the pedestrian flow mobile network map within the specified time period by using the subway pedestrian flow network fusion method based on video pedestrian recognition includes:

Preferably, the predicting the total incoming and outgoing predicted pedestrian volume of each station in a specified time period in the future by using the graph neural network based on the pedestrian flow mobile network diagram comprises:

in the people flow mobile network diagram, subway stations are used as vertexes, traffic routes corresponding to all entrances and exits of the stations are used as sides, each vertex has a characteristic vector containing total incoming and outgoing people flow, and a model of the people flow mobile network diagram is constructed as follows:

G_t＝(V_t，ε，W)

in the formula, G_tIs a people flow mobile network diagram at time t, V_tThe method is characterized in that the method is a vector composed of characteristic vectors of all vertexes, epsilon represents an edge set between the vertexes, W is the weight of an adjacent matrix, and t is the current moment;

when the pedestrian flow of each vertex is predicted, predicting the feature vector from the time t +1 to the time t + H of a specified time period in the future based on the feature vector from the time t-M +1 to the time t of the vertex in the historical time period, wherein M, H is a preset coefficient, and constructing a pedestrian flow prediction target model as follows:

in the formula (I), the compound is shown in the specification,

for the predicted eigenvectors, v, from time t +1 to time t + H_t-M+1，...，v_tRepresenting the input feature vector from-M +1 time to t time;

and solving the people flow prediction target model by adopting a graph neural network based on the constructed people flow prediction target model to obtain the total incoming and outgoing predicted people flow of each station in a future specified time period.

The invention provides a subway people flow network fusion method and a people flow prediction method based on video pedestrian identification, which utilize video data of a subway station to carry out statistical analysis on the specific direction of people flow in and out of the subway station, and open up an overground and underground road traffic line and a subway line to form a complete people flow statistical traffic network which is quantifiable both on the ground and underground; the fusion of the subway network and the ground traffic network enables the movement of people to be carried out in a huge network, each subway station becomes a node in the network, and each subway line and ground road traffic line become edges of the network. And 4, deducing the people flow change of the whole traffic network by using the graph neural network so as to analyze and predict each station, the number of people flows and the direction of people flow. The method has the advantages that deeper pedestrian flow analysis of the subway station entrance and exit is realized, resource scheduling of each station and the entrance and exit is facilitated, and mutual influences of the overground pedestrian flow and the underground pedestrian flow can be judged in advance in time, so that traffic early warning is performed on the ground or underground in time, traffic jam is avoided, station security measures are deployed in advance, and the like.

Drawings

Fig. 1 is a flow chart of a subway pedestrian flow network fusion method based on video pedestrian identification according to the present invention;

FIG. 2 is a schematic diagram illustrating the training of an SSD target detection network in accordance with the present invention;

FIG. 3 is a schematic diagram of the training of the MobileNet neural network of the present invention;

FIG. 4 is a schematic diagram of the distillation operation of the neural network of the present invention;

FIG. 5 is a flow chart of pedestrian trajectory tracking in accordance with the present invention;

FIG. 6 is a flow chart of a multi-factor fusion pedestrian target tracking method of the present invention;

FIG. 7 is a flow chart of a target tracking method based on pedestrian image features according to the present invention;

FIG. 8 is a schematic diagram of an embodiment of a subway traffic network diagram according to the present invention;

FIG. 9 is a schematic diagram of problem modeling based on spatio-temporal sequences in accordance with the present invention;

FIG. 10 is a schematic structural diagram of the frame of the STGCN of the present invention.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. The terminology used in the description of the invention herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

In one embodiment, a subway pedestrian flow network fusion method based on video pedestrian identification is provided, people flow contact of an overground traffic route and an underground subway route is established, and incoming directions and outgoing directions of incoming people flows of all entrances and exits of all stations of a subway are analyzed. The correlation of the pedestrian flow rate on the ground (namely the ground) and the underground (namely the subway) overcomes the defect that the prior terrestrial or underground pedestrian flow statistics only considers the pedestrian flow of a single layer on the ground or the underground, but does not further consider the factors of mutual influence of the terrestrial and underground pedestrian flows, so that the precision of the terrestrial and underground statistics or prediction is not enough. The pedestrian flow statistical method based on the fusion of the overground and underground networks not only can utilize resource scheduling of each entrance and exit of each station of the subway, but also can be used for early warning pedestrian flow of the subway station by combining with the overground traffic network, and can be used for early warning pedestrian flow of an overground traffic route by combining with the underground subway network, so that the traffic control prospect and timeliness are improved.

As shown in fig. 1, the method for fusing subway pedestrian flow networks based on video pedestrian identification in the embodiment includes the following steps:

step 1, receiving monitoring images of each entrance and exit of a subway station, wherein the monitoring images are acquired by image acquisition equipment arranged at each entrance and exit.

Because the data of the ground and underground people flows need to be associated, the monitoring range of the image acquisition equipment comprises the whole entrance and exit and the ground traffic route corresponding to the entrance and exit when the image acquisition equipment is deployed, and a foundation is laid for identifying the entrance and exit states, the exit or the entrance direction of pedestrians.

The image capturing devices in the present embodiment may be optical cameras, binocular cameras, TOF cameras, and the like, and each image capturing device has a unique device id in order to distinguish each image capturing device. Therefore, while receiving the monitoring image, the device id of the image acquisition device corresponding to the monitoring image and the corresponding timestamp are obtained.

It should be noted that, usually, one image acquisition device is installed at each entrance and exit of the subway station to meet the use requirements of the present invention, but the present invention is not limited to only installing one image acquisition device at each entrance and exit, and under the condition of actual monitoring requirement or monitoring precision requirement, a plurality of image acquisition devices can be installed at one entrance and exit to more comprehensively obtain video image information, and image acquisition devices can also be installed inside the subway station and along the traffic route corresponding to the entrance and exit to expand the video image acquisition range to obtain more comprehensive and complete people flow statistics or pedestrian tracks.

And 2, extracting pedestrian target coordinate frame information and pedestrian target characteristic information in the monitored image, wherein the pedestrian target characteristic information comprises pedestrian characteristics, a pedestrian in-and-out state, and a pedestrian out-of-station or in-station direction.

The pedestrian target coordinate frame information and the pedestrian target feature information are basic information for identifying and positioning the pedestrian target, and the embodiment extracts two parts of information based on the neural network. The existing target recognition neural networks and feature recognition neural networks are more, and the embodiment does not limit the adopted neural networks. For the convenience of understanding, the SSD object detection network extracts the coordinate frame information of the pedestrian object, and the MobileNet neural network extracts the feature information of the pedestrian object.

As shown in fig. 2, the SSD target detection network is used as a training application process of the target identification algorithm, and the specific steps are as follows:

1. and constructing a pedestrian target data set, wherein the data set comprises image data and marking data, and the marking data marks the region of the pedestrian target in the specified image.

2. And counting the aspect ratio of the pedestrian target in the data set, clustering the aspect ratio data by adopting a clustering method to obtain n clustering centers, namely n aspect ratios, and adopting the n aspect ratios as the proportion of an anchor frame in the target identification network.

3. The image data is enhanced to obtain training data, such as color change, random cropping, image magnification, rotation, and the like.

4. Training data is input into an SSD target detection network for target identification, and a neural network outputs coordinate frame information of pedestrians in an image.

5. And removing repeated frames by adopting a non-maximum suppression method (NMS) for the coordinate frames output by the SSD target detection network to obtain the final output coordinate frames.

6. And selecting the frames which intersect with the labeling coordinate frame and are larger than a larger threshold value as positive samples and selecting the frames which intersect with the labeling coordinate frame and are smaller than a smaller threshold value as negative samples in the output coordinate frames. And randomly selecting a certain proportion and quantity of samples from the positive and negative samples as training samples to train the neural network.

7. And calculating a loss function according to the final training sample coordinate frame and the labeled data, and adjusting the neural network parameters by adopting a back propagation method.

8. And obtaining a final SSD target detection network after certain training.

In the application of the SSD target detection network, the trained SSD target detection network receives image input, outputs the image as a pedestrian coordinate frame, performs non-maximum value suppression operation on the coordinate frame, deletes a repeated coordinate frame, and finally sets a certain threshold value, wherein the image is used as output pedestrian target coordinate frame information when the credibility of the coordinate frame is greater than the threshold value.

As shown in fig. 3, the application process of training the MobileNet neural network as the pedestrian re-recognition algorithm includes the following specific steps:

1. and constructing a pedestrian re-identification data set, wherein the data set comprises image data and id data of pedestrians corresponding to the images.

2. And 3 pictures, namely two different pictures of the pedestrian A, one picture of other pedestrians and three pictures are selected in one training and are respectively input into the MobileNet neural network after image enhancement, and pedestrian characteristics are output.

3. And calculating a ternary loss function according to the output of the neural network, wherein the more similar the image characteristics of the pedestrians of the same person are, the more dissimilar the image characteristics of the pedestrians of different persons are, and therefore, the loss function can be used for training a pedestrian re-recognition algorithm. After the loss function is calculated, the neural network is trained using back propagation.

When the target detection network is deployed, in order to operate the network more quickly, the distillation operation needs to be performed on the network neural network, as shown in fig. 4 in particular. Teacher's model is usually the neural network model of the comparison big of good training, and this kind of model accuracy rate is generally high, but the parameter of network is many, and operating time is slow. The student model is generally a model with less parameter quantity, the model is difficult to train if the model is directly trained by adopting the labeled data, and the neural network distills to enable the student model to learn from the labeled data and the teacher model at the same time, so that a better result can be obtained.

In the application of the MobileNet neural network, the final MobileNet neural network obtained through training receives image data and id data of pedestrians, and outputs pedestrian target characteristic information corresponding to each pedestrian.

The target recognition neural network, the feature recognition neural network and the corresponding training application method provided by the embodiment can completely, comprehensively and accurately extract corresponding data in the monitoring image, and improve high-quality basic information for people flow statistics.

Since the real trajectory of a pedestrian cannot be accurately identified solely according to the state of the pedestrian in one monitored image, trajectory tracking of the pedestrian is required. As shown in fig. 5, the multi-target tracking method in this embodiment is divided into two parts, one of which is a multi-factor fusion pedestrian target tracking method under the monitoring picture of the same image acquisition device, and the method generates the pedestrian trajectory under the monitoring picture of the image acquisition device. The other is a target tracking method based on pedestrian characteristics and crossing image acquisition equipment, and the method is used for matching the pedestrian track of the same pedestrian under different image acquisition equipment. The pedestrian target tracking method of the cross-image acquisition equipment directly adopts the characteristic information of the pedestrian target to calculate the similarity, the similarity is larger than a certain threshold value, the same pedestrian is judged, and the related tracks are associated. The two methods are combined to obtain the pedestrian trajectory data of the cross-region, so that complete pedestrian trajectory tracking is performed, and the accuracy of pedestrian flow statistics is improved.

And 3, based on the monitoring image of the same image acquisition device, carrying out similarity calculation according to the pedestrian target coordinate frame information and the pedestrian target characteristic information to obtain the pedestrian track for the same pedestrian. Namely, the method for tracking the multi-factor fusion pedestrian target is as follows as shown in fig. 6:

and 3.1, acquiring the pedestrian target coordinate frame information and the pedestrian target characteristic information of the current image acquisition equipment in the monitored image.

Step 3.2, judging whether a tracking track set corresponding to the image acquisition equipment is empty, wherein the tracking track set is used for storing the pedestrian track of the pedestrian, and if the tracking track set is not empty, executing step 3.3; and otherwise, directly adding the pedestrian target coordinate frame information and the pedestrian target characteristic information acquired this time into the tracking track set and finishing.

And 3.3, obtaining the information of the estimated target coordinate frame by adopting unscented Kalman filtering based on the pedestrian track in the tracking track set.

Unscented Kalman filtering is developed on the basis of Kalman filtering and transformation, and the unscented Kalman filtering under the linear assumption is applied to a nonlinear system by means of lossless transformation. The unscented kalman filter is used to estimate the position of each existing pedestrian trajectory at the current time.

And 3.4, calculating the coordinate frame similarity between the current pedestrian target and the stored pedestrian target one by one according to the pedestrian target coordinate frame information and the estimated target coordinate frame information, calculating the feature similarity between the current pedestrian target (namely the pedestrian target obtained at the time) and the stored pedestrian target one by one on the basis of the pedestrian target feature information of the pedestrian track and the pedestrian target feature information in the monitored image at the time, and obtaining the similarity between the current pedestrian target and the stored pedestrian target one by the basis of the weighted summation of the coordinate frame similarity and the feature similarity.

Calculating IOU (input output Unit) of the pedestrian target, the distance of the central point of the pedestrian target, the size difference of the pedestrian target and the like based on the pedestrian target estimation frame in the track set predicted by unscented Kalman filtering and the pedestrian target frame obtained by current detection, and carrying out weighted summation on the indexes and the feature similarity (such as cosin similarity of features and the like) of the pedestrian target feature information to construct the similarity between the existing track and the pedestrian to be matched.

The similarity between the current pedestrian target and the stored pedestrian target finally obtained by the embodiment integrates the coordinate frame similarity and the feature similarity, and the pedestrian target is matched from multiple directions, so that the accuracy of the pedestrian track is obviously improved. It should be noted that the calculation of the coordinate frame similarity and the feature similarity is a mature technology in the field of pedestrian trajectory tracking, and is not repeated in this embodiment. And the weighted sum weight value is set according to the emphasis point in actual use.

And in order to visually express the similarity between every two pedestrian targets, a similarity matrix can be adopted for storage, the stored pedestrian targets are longitudinally in the similarity matrix, the current pedestrian target is transversely in the similarity matrix, and the corresponding values in the matrix are the similarity between the corresponding stored pedestrian targets and the current pedestrian target.

And 3.5, matching the pedestrian tracks in the tracking track set and the pedestrian targets obtained at this time by adopting a Hungarian matching algorithm based on the similarity between the current pedestrian target and the stored pedestrian targets.

When the pedestrian tracks under the same image acquisition equipment are generated, the pedestrian track of each pedestrian is updated in real time, and when a pedestrian is newly added in the monitoring range, the newly added pedestrian is confirmed through continuous and multiple successful matching, so that the false detection condition is avoided; and after the pedestrian track is not successfully matched for a plurality of times continuously, the pedestrian track is deleted, so that the storage pressure and the matching pressure are reduced, and the matching speed is increased.

And 4, carrying out similarity matching on the pedestrian target characteristic information based on the pedestrian tracks corresponding to different image acquisition devices, combining the successfully matched pedestrian tracks, and updating the pedestrian tracks corresponding to the pedestrians. Namely, the target tracking method based on the image features of the pedestrians is specifically as follows, as shown in fig. 7:

and 4.1, taking a tracking track set corresponding to the image acquisition equipment, and carrying out similarity calculation on the pedestrian tracks marked as the new tracks in the tracking track set and the pedestrian tracks marked as the leaving tracks in the tracking track sets corresponding to the other image acquisition equipment one by one.

Because the pedestrian can not appear in two control pictures simultaneously under normal conditions, therefore this embodiment only takes the corresponding orbit that leaves of other image acquisition equipment to match to guarantee that the matching result accords with normal pedestrian's action of moving, also reduced characteristic matching pressure, promoted cross regional matching speed simultaneously.

And 4.2, if the similarity is greater than a preset threshold value, the two pedestrian tracks are successfully matched. In this embodiment, the calculation similarity is calculated based on the pedestrian target feature information carried by the two pedestrian tracks, and the calculated pedestrian target feature information may be the pedestrian target feature information in the latest monitored image in the pedestrian track or an average value of the pedestrian target feature information in the latest several frames of monitored images. The similarity may be a cosine similarity and consists in matching in the hungarian algorithm.

In this embodiment, two sections of successfully matched pedestrian trajectories are combined, preferably, the two sections of pedestrian trajectories are spliced according to a time sequence to obtain a pedestrian trajectory that conforms to a real moving path of a pedestrian, and the combined cross-region pedestrian trajectory is moved to a tracking trajectory set where a new trajectory is located, that is, the departure trajectory is also moved from an original tracking trajectory set to the tracking trajectory set where the new trajectory is located, so that the combined management of the pedestrian trajectories is realized.

And 5, acquiring subway lines, subway stations, entrances and exits of the stations in the designated area and overground traffic lines corresponding to the entrances and exits, and fusing and constructing the subway traffic network map in the designated area.

In this embodiment, the traffic route on the ground corresponding to each entrance is understood as the ground road where the entrance is located. The ground road where the entrance is located is taken out of course to be the basic operation of the fusion of the ground and underground traffic networks, and the same principle as the installation of the image acquisition equipment in the step 1 is adopted, if the image acquisition equipment is not only installed at the entrance and the exit, but also is expanded to all ground roads within the preset range of the subway station, the corresponding ground traffic route can also be other ground roads which are related to the image acquisition equipment and are extended from the ground road where the entrance and the exit are located. The finally constructed subway traffic network diagram is shown in fig. 8, points represent subway stations, solid lines represent ground roads, dotted lines represent subway lines, namely subway stations are taken as points, subway lines and traffic lines are taken as sides, and connecting points of the traffic lines and the subway stations are used for indicating entrances and exits. Of course, because the invention mainly counts the pedestrian flow at the entrance and the exit of the subway station and the pedestrian flow in the ground road, the formed subway traffic network map can also be a network map which only contains the subway station as a point and the ground road as a side.

And 6, counting the total incoming and outgoing pedestrian flow of each station and the incoming and outgoing pedestrian flow on the traffic route corresponding to each entrance and exit in each station according to the latest pedestrian track in the preset time period.

The pedestrian track at least comprises a pedestrian moving path in the same image acquisition equipment, and the moving path has a direction, so that the pedestrian entering state or the exiting state can be identified according to the pedestrian track, and the total entering and exiting pedestrian flow in the specified time period can be obtained through statistics correspondingly.

And because the monitoring range of the image acquisition equipment comprises a ground traffic route, the pedestrian track comprises the process of entering the picture from a certain direction of the traffic route to enter the picture or moving the picture from the station to a certain direction of the traffic route in the monitoring picture, namely the corresponding incoming and outgoing pedestrian flow on the traffic route can be obtained, and the incoming and outgoing pedestrian flow on the traffic route comprises the pedestrian exiting or entering directions (taking the example that the exiting only comprises a left turn and a right turn, and the outgoing pedestrian flow on the traffic route comprises the pedestrian flow entering the traffic route after the left turn and the pedestrian flow entering the traffic route after the right turn).

The finally obtained people flow mobile network diagram shows the in-and-out people flow of each entrance and exit of each subway station, the in-and-out people flow comprises the in-station people flow, the out-station people flow and the flow direction of the out-station people flow in the traffic route (for example, the traffic route corresponding to the entrance and exit extends in the north and south directions, the people flow going in the south direction after exiting the station and the people flow going in the north direction after exiting the station), and the in-station people flow comes from the traffic route (for example, the traffic route corresponding to the entrance and exit extends in the north and south directions, the people flow going in the south direction and the people flow going in the north direction are reached by the route).

The pedestrian flow direction or the coming direction can be obtained by superposing a plurality of monitoring images according to the time sequence according to the pedestrian in-out state and the pedestrian out-out or in-in direction of each frame of monitoring image of the pedestrian, namely generating pedestrian tracks. The traffic early warning system is convenient for analyzing the conditions of traffic congestion and the like possibly existing in the ground traffic route based on subway people flow so as to take measures of evacuation, early warning and the like in time, and has important help for traffic early warning in scenic spots, urban arterial roads and nearby urban activity places.

In another embodiment, a people flow prediction method is further provided, which performs people flow prediction based on fusion of subway and ground people flow to assist traffic early warning, and includes:

and obtaining a pedestrian flow mobile network diagram in a specified time period by using a subway pedestrian flow network fusion method based on video pedestrian identification.

And predicting the total incoming and outgoing predicted pedestrian volume of each station in a specified time period in the future by using the graph neural network based on the pedestrian flow mobile network diagram.

And obtaining the average value of the incoming and outgoing proportion of the traffic route corresponding to each entrance and exit of each station based on the incoming and outgoing pedestrian flow on the traffic route corresponding to each entrance and exit of each station in the pedestrian flow mobile network diagram.

In another embodiment, the obtaining of the people flow mobile network map in a specified time period by using the subway people flow network fusion method based on video pedestrian recognition includes:

In another embodiment, the obtaining of the pedestrian trajectory for the same pedestrian based on the monitored image of the same image capturing device by performing similarity calculation according to the pedestrian target coordinate frame information and the pedestrian target feature information includes:

And 3.4, calculating the coordinate frame similarity between the current pedestrian target and the stored pedestrian target one by one according to the pedestrian target coordinate frame information and the estimated target coordinate frame information, calculating the feature similarity between the current pedestrian target and the stored pedestrian target one by one based on the pedestrian target feature information of the pedestrian track and the pedestrian target feature information in the monitored image, and obtaining the similarity between the current pedestrian target and the stored pedestrian target based on the weighted summation of the coordinate frame similarity and the feature similarity.

In another embodiment, the performing similarity matching on the pedestrian target feature information based on the pedestrian trajectories corresponding to different image acquisition devices, combining the pedestrian trajectories successfully matched, and updating the pedestrian trajectories of the corresponding pedestrians includes:

For specific limitations of obtaining the pedestrian flow mobile network map within the specified time period by using the subway pedestrian flow network fusion method based on video pedestrian recognition in this embodiment, reference may be made to the above specific limitations of the subway pedestrian flow network fusion method based on video pedestrian recognition, and details are not repeated here. In this embodiment, the underground and overground pedestrian flow network diagram is constructed by representing a subway network of the whole city as a diagram, where a subway station is a vertex, and a subway line and a traffic line on the ground connecting each entrance and exit of the subway station are sides (the traffic line on the ground connecting each entrance and exit of the subway station may be only used as a side, and if the side includes a subway line, the pedestrian flow of the subway line is data-free, or the pedestrian flow movement inside the subway line is obtained by using the existing subway station card swiping information and other manners based on the present invention). Each vertex has a feature vector consisting of the traffic of the video statistics, and an adjacency matrix can be defined to encode the pairwise dependencies between the vertices. Therefore, the subway network does not need to use grid to represent subway stations and CNN to capture features, but can be described by a general network diagram, and irregular space-time dependence on the subway network layer rather than the grid layer can be effectively captured by using a diagram neural network (GCN).

The problem modeling is mathematical modeling applied to the prediction of the pedestrian flow of the underground and overground pedestrian flow statistical network at each side, and the pedestrian flow of the network in the next days is predicted by using the historical pedestrian flow numerical value of the network. The problem can be modeled specifically by using the spatio-temporal sequence shown in fig. 9, and we define a metro network in the city range on a graph and focus on the structured time-series passenger flow. The model for constructing the people flow mobile network diagram is as follows:

G_t＝(V_t，ε，W)

G_tthe figure of the people flow mobile network at the time t is a figure consisting of a plurality of nodes, V_tIs a finite set of nodes, representing vertices in the graph, that monitor the traffic of each node, i.e., V_tIs a vector consisting of the pedestrian volumes of all nodes, ε represents the set of edges between vertices, and W, which shows the connectivity between vertices, is the weight of the adjacency matrix.

When the pedestrian volume of each vertex is predicted, historical data of the previous t moments are given to predict one or more moments in the future. The method predicts a people flow value, gives historical data from t-M +1 to t, predicts the people flow from t +1 to t + H, and constructs a people flow prediction target model as follows:

in the formula (I), the compound is shown in the specification,

is the predicted eigenvector (total incoming/outgoing pedestrian volume) from time t +1 to time t + H_t-M+1，...，v_tA feature vector (total incoming/outgoing traffic) representing the incoming-M +1 time to t time. In addition, v is_t+1，...，v_t+HAlso the predicted eigenvectors from time t +1 to time t + H,

is a way mathematically used to distinguish between predictor variables.

It is easy to understand that if the input data is the total inbound people flow rate of a certain station at the time from t-M +1 to t, the obtained predicted feature vector is also the total inbound people flow rate in a specified time period in the future; similarly, if the input data is the total outbound pedestrian volume of a certain station at the time from t-M +1 to t, the obtained predicted feature vector is also the total outbound pedestrian volume in the specified time period in the future.

For the graph neural network model, the graph structure data is directly used for carrying out high-order feature extraction in a spatial domain, Chebyshev polynomial approximation is used, and the convolution formula of the Chebyshev graph is as follows:

the model framework used is the framework of the STGCN and is composed of a plurality of space-time convolution modules, each module is structured as a sandwich (as shown in fig. 10), and has two gate sequence convolution layers and a space diagram convolution module in the middle. The Temporal Gated-Conv is used to capture Temporal correlations and consists of a 1-D Conv and a Gated linear unit GLU; the Spatial Graph-Conv is used to capture Spatial correlation and is mainly composed of the Chebyshev Graph convolution module.

And after the total inlet and outlet predicted pedestrian volume of each station is obtained, distributing the total predicted pedestrian volume according to the proportional mean value corresponding to each inlet and outlet of the station. The proportional mean value of the inbound people flow of each entrance and exit of a station is obtained by calculation according to the inbound people flow of each entrance and exit in the corresponding time period; similarly, the proportional mean of the outbound pedestrian flow of each entrance and exit of a station is calculated according to the outbound pedestrian flow of each entrance and exit in the corresponding time period. When the total predicted pedestrian volume distribution is carried out, the distribution is carried out based on the inlet proportion average value (namely the proportion average value of the inlet pedestrian volume) and the distribution is carried out based on the outlet proportion average value (namely the proportion average value of the outlet pedestrian volume), so that a prediction result with traceability is obtained, and the prediction result is guaranteed to have practical application value.

The technical features of the embodiments described above may be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the embodiments described above are not described, but should be considered as being within the scope of the present specification as long as there is no contradiction between the combinations of the technical features.

The above-mentioned embodiments only express several embodiments of the present invention, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the inventive concept, which falls within the scope of the present invention. Therefore, the protection scope of the present patent shall be subject to the appended claims.

Claims

1. A fusion method of subway people flow network based on video pedestrian recognition is used for realizing fusion statistics of subway and ground people flow to assist traffic early warning, and is characterized in that the fusion method of subway people flow network based on video pedestrian recognition comprises the following steps:

2. A method as claimed in claim 1, wherein the video pedestrian recognition-based fusion method of a subway pedestrian flow network, wherein the step of performing similarity calculation on the monitored images based on the same image acquisition device according to the information of the coordinate frame of the pedestrian target and the characteristic information of the pedestrian target to obtain the pedestrian trajectory for the same pedestrian, comprises:

3. A method as claimed in claim 2, wherein the method for fusing pedestrian flow networks of subways based on video pedestrian recognition is characterized in that the method for matching the similarity of pedestrian target characteristic information based on pedestrian trajectories corresponding to different image acquisition devices combines the pedestrian trajectories successfully matched with each other to update the pedestrian trajectories of corresponding pedestrians comprises:

4. A people flow prediction method is used for carrying out people flow prediction to assist traffic early warning based on fusion of subway and ground people flow, and is characterized by comprising the following steps:

5. The people flow prediction method according to claim 4, wherein the obtaining of the people flow mobile network map in the specified time period by using the subway people flow network fusion method based on video pedestrian recognition comprises:

6. The pedestrian flow prediction method according to claim 5, wherein the obtaining of the pedestrian trajectory for the same pedestrian based on the monitored image of the same image acquisition device by performing similarity calculation according to the pedestrian target coordinate frame information and the pedestrian target feature information comprises:

7. The pedestrian flow prediction method according to claim 6, wherein the similarity matching of the pedestrian target feature information is performed based on the pedestrian trajectories corresponding to different image acquisition devices, and the pedestrian trajectories successfully matched are combined to update the pedestrian trajectories of the corresponding pedestrians, and the method comprises the following steps:

8. The people flow prediction method of claim 4, wherein the predicting the total incoming and outgoing predicted people flow of each station in a specified time period in the future by using a graph neural network based on the people flow mobile network diagram comprises the following steps:

G_t＝(V_t，ε，W)

in the formula (I), the compound is shown in the specification,

for the predicted eigenvectors, v, from time t +1 to time t + H_t-M+1,…,v_tRepresenting the input feature vector from-M +1 time to t time;