CN114067271A - Safety risk early warning method based on AIOT and personnel trajectory analysis - Google Patents
Safety risk early warning method based on AIOT and personnel trajectory analysis Download PDFInfo
- Publication number
- CN114067271A CN114067271A CN202111380850.5A CN202111380850A CN114067271A CN 114067271 A CN114067271 A CN 114067271A CN 202111380850 A CN202111380850 A CN 202111380850A CN 114067271 A CN114067271 A CN 114067271A
- Authority
- CN
- China
- Prior art keywords
- key point
- point
- track
- personnel
- key
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/22—Matching criteria, e.g. proximity measures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Evolutionary Computation (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computational Linguistics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Evolutionary Biology (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Image Analysis (AREA)
Abstract
The invention provides a safety risk early warning method based on AIOT and personnel trajectory analysis, which comprises the following steps: acquiring personnel images under a production operation scene in real time by using a camera, acquiring a first key point heat map of personnel based on the personnel images, processing the first key point heat map by using a trained key point presumption network, and presuming to obtain a second key point heat map of the personnel; the first key point is a head central point, and the second key point is a two-foot connecting line central point; acquiring personnel track information according to the heat map of the second key points of the multiple frames; the production operation scene comprises a production operation area, and when the number of personnel in the production operation area reaches a preset number threshold value and people are detected to be close to the production operation area based on the personnel track information, risk early warning is carried out. According to the invention, the accurate positioning of the foot key points is carried out on the basis of the head key points of the personnel in consideration of the oblique overlook imaging characteristic of the camera, so that the positioning accuracy is improved, and the shielding problem is solved.
Description
Technical Field
The invention relates to the field of artificial intelligence, in particular to a safety risk early warning method based on AIOT and personnel trajectory analysis.
Background
The existing personnel positioning method usually adopts a method of enclosing a frame or human key points, and the method is difficult to accurately position under the condition of serious positioning deviation existing in two-dimensional image processing, particularly under the factory safety production scene, the situation that the personnel positioning by using the existing method is easy to have wrong identification is easy to occur, for example, when the personnel positioning is carried out according to the extracted human key points, the detected key points are lost due to personnel shielding, further, multi-frame key points are difficult to match, tracks are difficult to analyze, and effective early warning cannot be carried out.
Disclosure of Invention
In order to solve the problems, the invention provides a safety risk early warning method based on AIOT and personnel trajectory analysis, which comprises the following steps:
acquiring personnel images under a production operation scene in real time by using a camera, acquiring a first key point heat map of personnel based on the personnel images, processing the first key point heat map by using a trained key point presumption network, and presuming to obtain a second key point heat map of the personnel; the first key point is a head central point, and the second key point is a two-foot connecting line central point;
acquiring personnel track information according to the heat map of the second key points of the multiple frames; the production operation scene comprises a production operation area, and when the number of personnel in the production operation area reaches a preset number threshold value, risk early warning is carried out when people are detected to approach the production operation area based on personnel track information;
the training process of the key point presumption network comprises the following steps:
obtaining effective key point matching pairs in each frame of head and foot key point heat map of an image set, wherein one effective key point matching pair comprises a first key point and a second key point; wherein the heat map of the key points of the head and the feet comprises a first key point and a second key point;
forming a training image set by using a first key point heat map corresponding to each frame of head and foot key point heat maps acquired based on effective key point matching pairs, and forming a label image set by using a second key point heat map; and training the key point inference network by using the training image set and the label image set.
Furthermore, an included angle between a direction in which the first key point in the effective key point matching pair points to the second key point and a reference direction of the effective key point matching pair is the smallest, wherein the reference direction of the effective key point matching pair is a direction in which the first key point points to a convergence point, and the convergence point is an intersection point in the body extension direction of all the persons in the person image.
Further, the method for acquiring the convergent point comprises the following steps:
performing rough matching of the first key point and the second key point in the heat map of the key points of the head and the feet to obtain rough matching pairs of the key points;
and generating a hypothetical image along the lower edge direction of the personnel image, connecting the heat map central point of the head and foot key points with the central point of the hypothetical image to obtain a first straight line, connecting the first key point and the second key point in the rough key point matching pair to obtain a plurality of second straight lines, clustering the intersection points of the plurality of second straight lines and the first straight lines, wherein the clustering central point is a convergent point.
Further, the coarse matching process is as follows:
and determining second key points matched with each first key point based on an included angle between the first vector and the second vector to obtain a rough key point matching pair.
Further, acquiring moving starting points and moving termination points of all the persons, and acquiring trajectory lines of the persons based on the multi-frame second key point heat map obtained by inference;
taking the moving starting point as a searching starting point and the moving ending point as a searching ending point, searching continuous pixel points on the trajectory line of the personnel to obtain all possible moving trajectories to form a trajectory set;
and inputting the track set into a track acquisition network to obtain the moving track of each person.
Further, supervising the loss of the trajectory acquisition network training process comprises:
and forgetting and superposing the track points of each moving track output by the track acquisition network according to the preset moving speed of the personnel to acquire the simulated heat value of the track points, and acquiring the simulated heat value of the track points on each moving track output by the network and the thermal value loss acquired by the actual heat value of the track points on the track segment in the corresponding personnel track line according to the track.
Further, monitoring the loss of the trajectory acquisition network training process further comprises: and obtaining distance loss according to the track length of each moving track output by the track obtaining network and the shortest connecting line length between the moving starting point and the moving ending point of the moving track.
Further, monitoring the loss of the trajectory acquisition network training process further comprises: and for each output moving track, acquiring the minimum position difference between the output moving track and each moving track in the track set, and acquiring the position loss according to the minimum position difference corresponding to all the output moving tracks.
The invention has the beneficial effects that:
1. according to the invention, the accurate positioning of the foot key points is carried out on the basis of the head key points of the personnel in consideration of the oblique overlook imaging characteristic of the camera, so that the positioning accuracy is improved, and the shielding problem is solved.
2. The invention effectively separates the complex track lines by using the neural network, avoids the situation of misjudgment when the tracks are superposed and improves the accuracy of track identification.
Drawings
FIG. 1 is a flow chart of the method of the present invention.
Detailed Description
In order that those skilled in the art will better understand the present invention, the following detailed description will be given with reference to the accompanying examples. In the following description, different "one embodiment" or "another embodiment" refers to not necessarily the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.
The specific application scenarios aimed by the invention are as follows: the method comprises the following steps of (1) a factory production operation scene, wherein the scene comprises a production operation area and a non-production operation area, and the production operation area is limited by the number of production operators.
The image acquisition equipment is arranged in a factory production operation scene to acquire RGB images of personnel, in order to cover the whole factory production operation scene, a plurality of image acquisition equipment with the same specification can be arranged according to actual conditions, but all the image acquisition equipment are required to be ensured to be in oblique overlooking visual angles, and certain superposition parts exist in imaging ranges of all the image acquisition equipment so as to facilitate image splicing operation.
The first embodiment is as follows:
the embodiment provides a safety risk early warning method based on AIOT and personnel trajectory analysis, and the implementation flow of the method is shown in FIG. 1, specifically, the method includes:
training the key point inference network:
1) acquiring personnel images collected by multiple cameras under a production operation scene in a top view angle in real time, detecting a first key point and a second key point of the multiple personnel images, and forming an image set by the obtained multiple head and foot key point heat maps; the first key point is a head central point, the second key point is a two-foot connecting line central point, and the head-foot key point heat map comprises the first key point and the second key point.
Obtaining effective key point matching pairs in each frame of head and foot key point heat map of an image set, wherein one effective key point matching pair comprises a first key point and a second key point; the included angle between the direction of the first key point in the effective key point matching pair pointing to the second key point and the reference direction of the effective matching pair is minimum, wherein the reference direction of the effective matching pair is the direction of the first key point pointing to the convergent point, and the convergent point is the intersection point of all the human body extension directions in the human image; specifically, the method for acquiring the convergent point comprises the following steps:
a) and performing rough matching of the first key point and the second key point in the heat map of the key points of the head and the feet to obtain rough matching pairs of the key points.
Generating a virtual image along the lower edge direction of the personnel image, namely generating the virtual image below the personnel image; the method comprises the steps of pointing a first key point in a head and foot key point heat map to a central point of a hypothetical image to generate a first vector, pointing the first key point in the head and foot key point heat map to a second key point to generate a second vector, determining the second key point matched with each first key point based on an included angle between the first vector and the second vector to obtain a rough key point matching pair, specifically, for each first key point, obtaining a first vector and a plurality of second vectors corresponding to the first key point, respectively calculating cosine values of the included angle between the first vector corresponding to the first key point and each second vector, and forming the rough key point matching pair by the second key point corresponding to the second vector and the first key point when the cosine value is maximum.
When a second key point respectively forms a rough key point matching pair with a plurality of first key points, screening the first key points based on the cosine value of the included angle between the vectors; and performing subsequent steps until each first key point in all the rough key point matching pairs respectively corresponds to different second key points.
b) Connecting the central point of the heat map of the key points of the head and the foot with the central point of the virtual image to obtain a first straight line, connecting the first key point and the second key point in each key point matching pair to obtain a plurality of second straight lines, clustering the intersection points of the plurality of second straight lines and the first straight lines, wherein the clustering central point is a convergent point, and specifically, in the embodiment, a K-means clustering algorithm is adopted, and the clustering set is 1.
The obtained convergence point is often a ground projection point of the camera.
After the convergent point is obtained, the first key points in the heat map of the key points of the head and the feet point to the convergent point to generate a third vector, a second key point matched with each first key point is determined based on an included angle between the second vector and the third vector to obtain an effective key point matching pair, and preferably, when the cosine value of the included angle between the second vector and the third vector is more than or equal to 0.8, the first key points and the second key points corresponding to the second vector form the effective key point matching pair.
2) Forming a training image set based on a first key point heat map corresponding to each frame of head and foot key point heat map acquired by effective key point matching pairs, and forming a label image set by a second key point heat map, specifically, forming a first key point heat map by the first key point in the effective key point matching pairs of each frame of head and foot key point heat map in the image set, and forming a second key point heat map by the second key point; the obtained multiple frames of first key point heat maps form a training image set, and the multiple frames of second key point heat maps form a tag image set;
3) and training the key point guess network by utilizing the training image set and the label image set, and adopting a mean square error loss function during training.
It should be noted that the keypoint estimation network is installed at the local end of the image capture device, so that the image capture devices at different positions need to be trained on the keypoint estimation network, and the insufficiency of generalization capability of the keypoint estimation network is avoided.
Preferably, the key point inference network is an Encoder-Decoder network structure.
Applying the key point guessing network:
the method comprises the steps of acquiring a first key point heat map of a person in a production operation scene in real time, processing the first key point heat map by using a trained key point presumption network, and presuming to obtain a second key point heat map of the person.
The reason for presuming the second key point according to the first key point is as follows: the first key point is easy to detect, and usually, the shielding condition does not exist, but the positioning is inaccurate; the second key point can well represent the two-dimensional information of the personnel relative to the ground, and compared with a surrounding frame or a human body central point, the positioning deviation caused by the height difference of the personnel key point and the ground is avoided, and the positioning is accurate but is easy to block; therefore, the mapping relation of the second key point can be obtained according to the first key point, so that the positioning accuracy is improved.
Because the ReiD function can not be realized in the scene, the situation of track overlapping can exist, for the area with simple track, the track is analyzed according to the thermal value to extract the complete track line, but for the area with frequent personnel transaction, the track is difficult to be directly separated through the thermal value information of the track, therefore, the invention provides a method for acquiring the personnel track information according to the multiframe second key point thermal map, and specifically:
acquiring the movement starting points and the movement termination points of all the personnel, and determining the movement starting points and the movement termination points of the personnel by an implementer according to the change condition of the thermal value of the points on the trajectory line; acquiring a person trajectory based on the multi-frame second key point heat map obtained by inference; taking the moving starting point as a searching starting point and the moving ending point as a searching ending point, and searching continuous pixel points on the staff trajectory line to obtain all possible moving trajectories to form a trajectory set; and inputting the track set into a track acquisition network to obtain the moving track of each person.
The method comprises the steps of obtaining a track line of a person by overlapping a plurality of inferred second key point heat maps based on a forgetting coefficient, or determining the track line of the person according to an IOU (input output) of an external rectangle of a second key point in the plurality of inferred second key point heat maps.
It should be noted that the lengths of all the movement tracks in the track set are the same, and specifically, the purpose of the same length of each movement track can be achieved by a complementary value.
Preferably, supervising the loss of the trajectory acquisition network training process comprises:
respectively forgetting and overlapping the track points of each moving track output by the track acquisition network according to the preset moving speed of the personnel to acquire the simulated heat force value of the track points on each output moving track, and acquiring the simulated heat force value of the track points on each output moving track and the corresponding personnel according to the simulated heat force value of the track points on each output moving trackSpecifically, the simulated heat value of the trace point on each output moving trace and the actual heat value of the trace point on the trace segment in the corresponding personnel trace are subjected to mean square error to obtain the Loss of heat value LossH;
Acquiring distance Loss according to the track length of each moving track output by the track acquisition network and the shortest connecting line length between the moving starting point and the moving ending point of the moving track, wherein specifically, the difference between the track length of each moving track output by the track acquisition network and the shortest connecting line length between the moving starting point and the moving ending point of the moving track is distance LossD;
For each moving track output by the track acquisition network, acquiring the minimum position difference between the output moving track and each moving track in the track set, and acquiring the position loss according to the minimum position difference corresponding to all the output moving tracks; specifically, the mean square error between the coordinates of the trace points on the output moving track and the coordinates of the trace points on each moving track in the track set is calculated respectively, the minimum value of the mean square error represents the minimum position difference corresponding to the output moving track, and the sum of the minimum position differences corresponding to all the output moving tracks is the position LossR。
Preferably, in the embodiment, the Loss function used when the trajectory acquisition network is trained is Loss ═ LossH+LossD+LossR。
The production operation scene comprises a production operation area, and when the number of people in the production operation area reaches a preset number threshold value and then people approach the production operation area, risk early warning is carried out; specifically, after the number of the persons in the production operation area reaches a preset number threshold, the infrared sensor is called to verify when the persons move to the production operation area and the distance between the persons and the production operation area is smaller than a preset distance threshold according to the movement track of the persons, when the distance is judged to be smaller than the preset distance threshold by the sensor, the safety risk is confirmed to exist, risk early warning information is generated, the form of the risk early warning information is not limited, and an implementer can select short message prompt, alarm prompt, light prompt or other early warning forms.
The foregoing is intended to provide those skilled in the art with a better understanding of the invention, and is not intended to limit the invention to the particular forms disclosed, since modifications and variations can be made without departing from the spirit and scope of the invention.
Claims (8)
1. A safety risk early warning method based on AIOT and personnel trajectory analysis is characterized by comprising the following steps:
acquiring personnel images under a production operation scene in real time by using a camera, acquiring a first key point heat map of personnel based on the personnel images, processing the first key point heat map by using a trained key point presumption network, and presuming to obtain a second key point heat map of the personnel; the first key point is a head central point, and the second key point is a two-foot connecting line central point;
acquiring personnel track information according to the heat map of the second key points of the multiple frames; the production operation scene comprises a production operation area, and when the number of personnel in the production operation area reaches a preset number threshold value, risk early warning is carried out when people are detected to approach the production operation area based on personnel track information;
the training process of the key point presumption network comprises the following steps:
obtaining effective key point matching pairs in each frame of head and foot key point heat map of an image set, wherein one effective key point matching pair comprises a first key point and a second key point; wherein the heat map of the key points of the head and the feet comprises a first key point and a second key point;
forming a training image set by using a first key point heat map corresponding to each frame of head and foot key point heat maps acquired based on effective key point matching pairs, and forming a label image set by using a second key point heat map; and training the key point inference network by using the training image set and the label image set.
2. The method of claim 1, wherein the direction of the first keypoint in the active keypoint matching pair pointing to the second keypoint is at a minimum angle to a reference direction of the active matching pair, wherein the reference direction of the active matching pair is the direction of the first keypoint pointing to a convergence point, and the convergence point is an intersection point of all human body extension directions in the human image.
3. The method of claim 2, wherein the method for acquiring the aggregation point comprises:
performing rough matching of the first key point and the second key point in the heat map of the key points of the head and the feet to obtain rough matching pairs of the key points;
and generating a hypothetical image along the lower edge direction of the personnel image, connecting the heat map central point of the head and foot key points with the central point of the hypothetical image to obtain a first straight line, connecting the first key point and the second key point in the rough key point matching pair to obtain a plurality of second straight lines, clustering the intersection points of the plurality of second straight lines and the first straight lines, wherein the clustering central point is a convergent point.
4. The method of claim 3, wherein the course of the coarse matching is:
and determining second key points matched with each first key point based on an included angle between the first vector and the second vector to obtain a rough key point matching pair.
5. The method according to claim 1, wherein a moving start point and a moving end point of all the persons are acquired, and trajectory lines of the persons are acquired based on the supposedly obtained multiframe second key point heat map;
taking the moving starting point as a searching starting point and the moving ending point as a searching ending point, searching continuous pixel points on the trajectory line of the personnel to obtain all possible moving trajectories to form a trajectory set;
and inputting the track set into a track acquisition network to obtain the moving track of each person.
6. The method of claim 4, wherein supervising the loss of the trajectory acquisition network training process comprises:
and forgetting and superposing the track points of each moving track output by the track acquisition network according to the preset moving speed of the personnel to acquire the simulated heat value of the track points, and acquiring the simulated heat value of the track points on each moving track output by the network and the thermal value loss acquired by the actual heat value of the track points on the track segment in the corresponding personnel track line according to the track.
7. The method of claim 6, wherein supervising the loss of the trajectory acquisition network training process further comprises: and obtaining distance loss according to the track length of each moving track output by the track obtaining network and the shortest connecting line length between the moving starting point and the moving ending point of the moving track.
8. The method of claim 7, wherein supervising the loss of the trajectory acquisition network training process further comprises: and for each output moving track, acquiring the minimum position difference between the output moving track and each moving track in the track set, and acquiring the position loss according to the minimum position difference corresponding to all the output moving tracks.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202111380850.5A CN114067271A (en) | 2021-11-20 | 2021-11-20 | Safety risk early warning method based on AIOT and personnel trajectory analysis |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202111380850.5A CN114067271A (en) | 2021-11-20 | 2021-11-20 | Safety risk early warning method based on AIOT and personnel trajectory analysis |
Publications (1)
Publication Number | Publication Date |
---|---|
CN114067271A true CN114067271A (en) | 2022-02-18 |
Family
ID=80278619
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202111380850.5A Pending CN114067271A (en) | 2021-11-20 | 2021-11-20 | Safety risk early warning method based on AIOT and personnel trajectory analysis |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN114067271A (en) |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN114707856A (en) * | 2022-04-02 | 2022-07-05 | 河南鑫安利安全科技股份有限公司 | Risk identification analysis and early warning system based on computer vision |
CN116652396A (en) * | 2023-08-01 | 2023-08-29 | 南通大学 | Safety early warning method and system for laser inner carving machine |
-
2021
- 2021-11-20 CN CN202111380850.5A patent/CN114067271A/en active Pending
Cited By (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN114707856A (en) * | 2022-04-02 | 2022-07-05 | 河南鑫安利安全科技股份有限公司 | Risk identification analysis and early warning system based on computer vision |
CN116652396A (en) * | 2023-08-01 | 2023-08-29 | 南通大学 | Safety early warning method and system for laser inner carving machine |
CN116652396B (en) * | 2023-08-01 | 2023-10-10 | 南通大学 | Safety early warning method and system for laser inner carving machine |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
JP7266106B2 (en) | Image coordinate system transformation method and its device, equipment and computer program | |
CN108399367B (en) | Hand motion recognition method and device, computer equipment and readable storage medium | |
Aghaei et al. | Single image human proxemics estimation for visual social distancing | |
CN114067271A (en) | Safety risk early warning method based on AIOT and personnel trajectory analysis | |
CN104361327A (en) | Pedestrian detection method and system | |
EP4137901A1 (en) | Deep-learning-based real-time process monitoring system, and method therefor | |
CN109598242A (en) | A kind of novel biopsy method | |
CN104036522A (en) | Water flowing speed monitoring method based on moving target identification in videos | |
KR101023951B1 (en) | motion recognition system and method | |
CN110956067B (en) | Construction method and device for eyelid curve of human eye | |
CN112989889B (en) | Gait recognition method based on gesture guidance | |
CN114894337B (en) | Temperature measurement method and device for outdoor face recognition | |
Devanne et al. | Learning shape variations of motion trajectories for gait analysis | |
JP2021067469A (en) | Distance estimation device and method | |
CN113688817A (en) | Instrument identification method and system for automatic inspection | |
Dong et al. | Vector detection network: An application study on robots reading analog meters in the wild | |
JP7518609B2 (en) | Image processing device, image processing method, and program | |
CN112800918A (en) | Identity recognition method and device for illegal moving target | |
CN109993116B (en) | Pedestrian re-identification method based on mutual learning of human bones | |
Chen et al. | Vision-based skeleton motion phase to evaluate working behavior: case study of ladder climbing safety | |
CN116659518B (en) | Autonomous navigation method, device, terminal and medium for intelligent wheelchair | |
JP6786098B2 (en) | Gait analyzer | |
CN106056080B (en) | A kind of visual biometric information acquisition device and method | |
KR101313879B1 (en) | Detecting and Tracing System of Human Using Gradient Histogram and Method of The Same | |
CN114639168B (en) | Method and system for recognizing running gesture |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination |