CN116524573A

CN116524573A - Abnormal article and mask detection system

Info

Publication number: CN116524573A
Application number: CN202310569935.0A
Authority: CN
Inventors: 刘世达; 李清怡; 吉鸿海; 范金凤; 于邱洋
Original assignee: Beijing Hongzhi Ruilong Education Technology Co ltd
Current assignee: Beijing Hongzhi Ruilong Education Technology Co ltd
Priority date: 2023-05-19
Filing date: 2023-05-19
Publication date: 2023-08-01

Abstract

The application provides an unusual article and gauze mask detecting system for solve current unusual article and gauze mask and detect untimely technical problem that leads to danger coefficient high. The abnormal article and mask detection system is characterized in that the first camera module with the acquisition range covering the entrance is used for acquiring images of facial features and physical features of a target object when the target object enters a bus, so that the timeliness of detection is improved. By identifying the shielding state of the facial features of the target object and the carrying object of the body features, abnormal behaviors and suspicious objects in the driving process are monitored, the risk coefficient is reduced, and the driving safety is improved.

Description

Abnormal article and mask detection system

Technical Field

The application relates to the technical field of image recognition, in particular to a detection system for abnormal articles and masks.

Background

The bus is used as a core traffic tool of a city, and has the characteristics of dense personnel, wide dispersion range, long points and multiple lines and the like.

In implementing the prior art, the inventors found that:

the flow of people in the bus is large, and the environment is closed. Because of the defect of vehicle-mounted video monitoring, passengers are reminded to wear the mask in the bus at present through a driver, so that not only is the labor consumed, but also the safety of other people on the bus is not guaranteed.

Therefore, it is necessary to provide a detecting system for abnormal articles and masks, so as to solve the technical problem that the dangerous coefficient is high due to the fact that the existing abnormal articles and masks are not detected in time.

Disclosure of Invention

The embodiment of the application needs to provide an abnormal article and mask detection system for solving the technical problem that the detection of the existing abnormal article and mask is not timely caused to have high risk coefficient.

Specifically, an unusual article and gauze mask detecting system includes:

the first camera module is used for collecting images comprising facial features and physical features of a target object;

the identification module is used for identifying the shielding state of the facial features of the target object; the portable object is also used for identifying the physical characteristics of the target object;

the prompting module is used for outputting prompting information when the shielding state of the facial features of the target object is not shielded; or the portable object is used for outputting prompt information when the portable object with the physical characteristics of the target object is an abnormal object.

Further, the identification module is configured to identify an occlusion state of a facial feature of the target object, and specifically is configured to:

inputting an image comprising facial features and physical features of a target object to a pre-training yolo_v5s detection model, and detecting whether the facial features of the target object in the image are blocked by a blocking object;

and when the facial features of the target object in the image are blocked by the blocking object, identifying the image semantics of the blocking object.

Further, the prompting module is configured to output prompting information when the shielding state of the facial features of the target object is not shielded, and is specifically configured to:

when the image semantic of the shielding object is not a mask, judging that the shielding state of the facial features of the target object is not shielded, and outputting prompt information.

Further, the identification module is configured to identify a carrying object of the physical feature of the target object, and specifically is configured to:

inputting an image comprising facial features and physical features of a target object to a pre-training yolo_v5s detection model, and detecting whether the physical features of the target object in the image are related to a carrying object or not;

and when the physical characteristics of the target object in the image are associated with the carrying object, identifying the image semantics of the carrying object.

Further, the prompting module is configured to output prompting information when the portable object with the physical characteristics of the target object is an abnormal object, and is specifically configured to:

calculating the size ratio coefficient of the carrying object;

when the size ratio coefficient of the carrying object is larger than a preset threshold, judging that the carrying object is an abnormal object, and outputting prompt information;

the formula for calculating the size ratio coefficient of the carrying object is expressed as follows:，

in the method, in the process of the invention,representing the width of the image as a whole, +.>Representing the height of the image as a whole, +.>Representing the width of the carrying object, < >>Representing the height of the carrying object.

Further, the pre-trained yolo_v5s detection model is obtained through training by the following steps:

acquiring a training image at least comprising facial features and physical features of a target object, carrying a shelter image semantic tag, and detecting model configuration information;

inputting training images, shelter image semantic tags, object image semantic tags, and detection model configuration information to an original yolo_v5s detection model to obtain training results;

calculating sample loss of the training result;

and inputting the sample loss to the original yolo_v5s detection model to perform back propagation training until the sample loss converges, and obtaining the pre-training yolo_v5s detection model.

Further, the first camera module at least includes:

a camera;

and the image acquisition card is connected with the camera through a CVBS interface.

Further, the identification module is a Jetson Xavier NX edge embedded device connected with the image acquisition card through a PCI interface, and includes:

the bottom layer V4L2 driving library is used for acquiring image frames from the image acquisition card;

and the storage unit is used for storing the image frames in a FIFO queue.

Further, the Jetson Xavier NX edge embedded device is configured to discard image frames greater than a preset frame rate by adopting a storage policy of frame-separation extraction, so as to ensure that the queue does not overflow.

Further, the abnormal article and mask detection system further includes:

a second camera module for tracking facial features of the target object to prompt the target object to unocclude facial features at a non-entry location; the acquisition range of the second camera module is not overlapped with the acquisition range of the first camera module.

The technical scheme provided by the embodiment of the application has at least the following beneficial effects:

through the first camera module of the coverage entry of collection scope, gather target object facial feature, physical feature's image when target object gets into the bus, improved the timeliness of detection. By identifying the shielding state of the facial features of the target object and the carrying object of the body features, abnormal behaviors and suspicious objects in the driving process are monitored, the risk coefficient is reduced, and the driving safety is improved.

Drawings

The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the application and do not constitute an undue limitation to the application. In the drawings:

fig. 1 is a schematic structural diagram of an abnormal article and mask detection system according to an embodiment of the present application.

100. Abnormal article and mask detection system

11. First camera module

12. Identification module

13. Prompt module

14. And a second camera module.

Detailed Description

For the purposes, technical solutions and advantages of the present application, the technical solutions of the present application will be clearly and completely described below with reference to specific embodiments of the present application and corresponding drawings. It will be apparent that the described embodiments are only some, but not all, of the embodiments of the present application. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the present disclosure, are within the scope of the present disclosure.

Referring to fig. 1, a system 100 for detecting an abnormal article and mask provided in the present application includes:

a first camera module 11 with a collection range covering the entrance is used for collecting images including facial features and physical features of the target object.

An identification module 12 for identifying an occlusion state of the facial feature of the target object; and also for identifying the carrying object of the physical characteristics of the target object.

A prompt module 13, configured to output prompt information when the occlusion state of the facial feature of the target object appears as non-occlusion; or the portable object is used for outputting prompt information when the portable object with the physical characteristics of the target object is an abnormal object.

It will be appreciated that the acquisition range of the first camera module 11 covers the entrance. The entry is in a specific application scenario, i.e. the area corresponding to a bus, for example a bus door. Therefore, when the target object enters the carriage space, the risk coefficient of the target object can be detected at the first time, and the timeliness of detection is improved.

Of course, the acquisition range of the first camera module 11 can also be adaptively adjusted. In a preferred embodiment provided in the present application, the cabin space may be modeled first according to the shape, size, space, and other regional information of the bus cabin. For the cabin space, the interior of the vehicle may be divided into a plurality of areas, such as a driver operation area, a passenger boarding area, a passenger alighting area, a cabin front portion, a cabin rear portion, and the like. The first camera module 11 may be disposed in any one of the driver's operation area, the passenger boarding area, the passenger alighting area, the front compartment area, and the rear compartment area. In other words, the acquisition range of the first camera module 11 may cover any area of the driver operation area, the passenger boarding area, the passenger alighting area, the front compartment area, and the rear compartment area.

Further, in a specific embodiment provided in the present application, the first image capturing module 11 includes at least:

a camera;

The facial features and the body feature images of the target object can be acquired through a camera. Specifically, images of facial features and physical features of a target object, which are shot by a camera in real time, are acquired. The camera may be a monocular infrared visible light camera. And then the video stream data is stored in the image acquisition card through the CVBS interface.

The identifying module 12 is used for identifying the shielding state of the facial features of the target object.

Under the mask detection scene, the shielding state of the facial features of the identification target object can be understood as the condition that the face of the person on the vehicle is worn by the person on the vehicle.

Further, in another preferred embodiment provided in the present application, the identifying module is configured to identify an occlusion state of a facial feature of the target object, specifically configured to:

It will be appreciated that when the facial features of the target object are occluded by the occlusion object, an image of the occluded facial features is acquired, and the image cannot be directly applied to facial recognition or key point detection. If the face is not blocked by the blocking object, an image with the face characteristics not blocked can be obtained, and the image can be directly applied to the recognition of the face characteristics or the detection of key points. It should be noted that, the specific determination requirement of whether the mask masks the facial features needs to be determined according to the actual application requirement. For example, when facial feature recognition is performed, all areas of the face in the acquired facial features must not be blocked by a blocking object. At this time, in the view angle range of image acquisition, it can be determined that the facial features of the target object are blocked by the blocking object when wearing a mask, a sunglasses, or the like. However, when the facial feature local region of the target object is recognized, the local region is not blocked by the blocking object. However, when the facial features of the target are recognized, the mouth and nose wearing mask does not affect the recognition and detection of the facial features. At this time, it cannot be determined that the facial features of the target object are blocked by the blocking object. Therefore, after the facial features with the target object are acquired, it is necessary to detect whether the facial features of the target object in the image are occluded according to the set determination-occlusion requirement. When detecting whether the facial features of the target object are blocked, the detection and classification of the images can be performed manually or by a related detection model. For example, when the facial features of the target object are detected to be blocked by the blocking object, the facial features of the corresponding target object are images of a first type; when the facial features of the target object are detected to be not blocked by the blocking object, the facial features of the corresponding target object are the second type image. It will be appreciated that the specific representation of the facial feature classification of the target object described herein is clearly not limiting on the specific scope of protection of the present application.

Specifically, an image including facial features and physical features of a target object is input to a pre-training yolo_v5s detection model, and whether the facial features of the target object in the image are blocked by a blocking object is detected.

The yolo_v5s detection model can be understood as a detection classification algorithm, and a user can select a calculation method of the backbone network according to actual requirements. Therefore, a yolo_v5s detection model is selected to detect whether the facial features of the target object are occluded by an occlusion object. It can be understood that the specific calculation method of the backbone network selected by the yolo_v5s detection model described herein obviously does not limit the specific protection scope of the present application. In a specific facial feature recognition scene, video data collected by cameras at the boarding position of a target passenger and before and after a carriage are input into a cover detection network. And for each frame of input video data, on the basis of a general target detection method, detecting and positioning the face in the image by using a one-stage mask detection frame, and judging whether a passenger wears a mask. Specifically, based on yolo_v5s algorithm, training the data set of personnel with the mask worn and the mask not worn, and detecting the real-time input video stream by using an abnormal behavior detection algorithm to obtain the abnormal condition detected by the mask not worn.

The identification module 12 is further configured to identify the carrying object of the physical feature of the target object.

It can be understood that in the abnormal object detection scenario, the object carrying the body feature of the identification target object can be understood as identifying whether the object carried on the target object on the vehicle is abnormal. The physical features are understood to be body parts such as hands, neck, waist, etc. The object to be carried can be understood as an article carried by the target object, including articles such as a backpack, a mobile phone and the like.

Further, in still another preferred embodiment provided in the present application, the identifying module is configured to identify a carrying object of a physical feature of the target object, specifically configured to:

The image of the facial features, body features of the target object can be understood as a dataset. The carrying object may include a large-sized case or the like.

In a specific embodiment of the application, a pre-training yolo_v5s detection model is used for training a large-scale box target detection algorithm aiming at a data set, and detection and identification are carried out on an actually acquired abnormal object video in a bus through the pre-training yolo_v5s detection model, so that real-time large-scale box identification, position information and time information are obtained. After the pictures of various forbidden articles are collected through the first camera module 11, labeling and labeling are carried out on the pictures, the processed pictures are input into a yolo_v5s detection model for training, and proper model parameters are obtained. When the method is applied, when the body features of the target object in the image are associated with the carrying object, the image semantics of the carrying object are identified, namely the position information of the suspicious object in each frame of picture and the probability of the type of the suspicious object are identified, namely the type confidence.

Further, in yet another preferred embodiment provided herein, the pre-trained yolo_v5s detection model is obtained by training the following steps:

calculating sample loss of the training result;

It can be understood that the yolo_v5s detection model has a clear structure and excellent real-time performance, and a user can select a calculation method of the yolo_v5s backbone network according to actual use requirements.

Obtaining training images at least comprising facial features and physical features of a target object, carrying object image semantic tags and detecting model configuration information.

The training image can be understood as an image including the facial features and the physical features of the target object required for pre-training the model. The mask image semantic tag can be understood as a class of face masks of a target person, such as mask tags, glasses tags and the like. The object image semantic tag can be understood as a category of object personnel carrying objects, such as a box tag, a handbag tag and the like.

After the training image is obtained, the training image, the shelter image semantic label, the object image semantic label and the detection model configuration information are sent to an original yolo_v5s detection model, and a training result is obtained.

Sample loss of training results is calculated. The sample loss may be understood as a loss value between a predicted value and a true value of the model, used to evaluate the degree of variability between the predicted value and the true value of the model. The smaller the sample loss value, i.e. the better the performance of the model. Sample loss for different models is also different in different application scenarios. And inputting the sample loss to the original yolo_v5s detection model to perform back propagation training until the sample loss converges, and obtaining the pre-training yolo_v5s detection model. The sample loss convergence can be understood as the gradual reduction or even coincidence of the loss between the predicted value and the true value during the training process, so as to obtain a pre-training yolo_v5s detection model.

Further, in a preferred embodiment provided in the present application, the identification module is a Jetson Xavier NX edge embedded device connected to the image capture card through a PCI interface, including:

and the storage unit is used for storing the image frames in a FIFO queue.

Further, in another preferred embodiment provided in the present application, the Jetson Xavier NX edge embedded device is configured to discard image frames greater than a preset frame rate by adopting a storage policy of frame-separation extraction, so as to ensure that the queue does not overflow.

The PCI (Peripheral Component Interconnect, peripheral component interconnect standard) interface can be understood as a PCI slot.

Video data acquired by the cameras are respectively input into a Jetson Xavier NX edge embedded device of the vehicle-mounted terminal for processing, and video streams are respectively input into corresponding abnormal behavior detection networks to acquire an abnormal detection result. Optionally, because the processing of the video data at the edge end requires a strong computing power, the vehicle-mounted terminal selects an artificial intelligent supercomputer with a smaller appearance, such as Jetson Xavier NX, and the application of the device in the embedded system and the edge system greatly improves the speed of processing the video data. Meanwhile, the equipment has high-capacity storage capacity, can realize local storage of detection data, has wireless communication and emergency communication capacity, and can ensure real-time transmission of alarm signals and cloud instructions

For a single frame image, the embedded device reads the video stream from the image acquisition card by calling the bottom layer V4L2 driving library. The algorithm design constructs a temporary FIFO queue for storing each frame of image of the video stream from the perspective of memory allocation. The FIFO (First Input First Output) queue is a first-in first-out queue. It will be appreciated that N data elements are always maintained in the FIFO queue, the data elements in the queue containing the most recent data and N-1 older data.

In a specific embodiment, after the bottom layer V4L2 driving library acquires the image frames from the image acquisition card, the image frames are subjected to the steps of marking, scaling, converting the size, converting the frame format and the like, and then stored in the FIFO queue to wait for the reading of the detection network. Since the processing speed (12 frames per second) of the detection network is smaller than the frame rate (30 frames per second) of the video stream, the method of frame-separated extraction is adopted to properly discard the redundant frames so as to ensure that the queue cannot overflow.

The method comprises the steps of acquiring a video segment data set containing abnormal objects and people (including wearing masks and unworn masks) from an actual bus infrared camera, and detecting a head target by adopting a yolo_v5s detection model. In the actual training process, MAP0.5 may reach 0.90. Under the real complex bus internal scene, the recognition accuracy is improved.

And the prompting module 13 is used for outputting prompting information when the shielding state of the facial features of the target object is not shielded.

It can be understood that when the blocking state of the facial features of the target object appears as non-blocking, an early warning mechanism is designed, and warning measures are timely made.

Further, in another preferred embodiment provided in the present application, the prompting module 13 is configured to output a prompting message when the occlusion state of the facial feature of the target object is not occluded, specifically configured to:

It can be understood that when the image semantic meaning of the shielding object is detected to be not the mask, the target object is judged not to wear the mask at the moment, and prompt information is output to give an alarm. Correspondingly, when the image semantic of the detected shielding object is the mask, the target object is judged to wear the mask, and the real-time video frame detection is continuously carried out on the target object.

And the prompt module 13 is used for outputting prompt information when the carrying object with the physical characteristics of the target object is an abnormal object.

Further, in another preferred embodiment provided in the present application, the prompting module is configured to output prompting information when the portable object of the physical feature of the target object is an abnormal object, and is specifically configured to:

calculating the size ratio coefficient of the carrying object;

the formula for calculating the size ratio coefficient of the carrying object is expressed as follows:in which, in the process,representing the width of the image as a whole, +.>Representing the height of the image as a whole, +.>Representing the width of the carrying object, < >>Representing the height of the carrying object.

It can be understood that the portable object with the physical characteristics of the target object is an abnormal object, and the output prompt information first needs to preset a threshold value to determine whether the portable object is not an abnormal object. In the abnormal article identification, if the abnormal article or the class confidence degree of the cutter and the gun with high risk is judged to be larger than a preset threshold value, an alarm is directly sent out.

Specifically calculating the size ratio of the portable objectThe formula for the coefficients is as follows:wherein->Indicating the detection of the width of the suspicious object, +.>High, indicative of detection of suspicious object, +.>Representing the width of the whole picture,/->High representing the whole picture, +.>Representing the set ranking ratio threshold. When->>/>At this time, an abnormality alarm is given. In a specific bus detection foreign object embodiment, a larger volume bin may be identified, with the threshold set at 0.03. After abnormal behaviors of the bus are detected, the cloud platform can send wireless instructions to the vehicle through the interaction server.

Further, in still another preferred embodiment provided in the present application, the abnormal article and mask detection system further includes:

a second camera module 14 for tracking facial features of the target object to prompt the target object to unocclude facial features in a non-portal position; the acquisition range of the second camera module 14 does not overlap with the acquisition range of the first camera module 11.

The second camera module 14 may be understood as a tracking camera provided for supplementing the full-scale facial features of the target object.

This is considered in that, after the target object successfully enters the cabin space, a case of taking off the mask may occur. This increases the safety factor in the cabin space. The multi-target tracking system based on the single camera inevitably has the problems that the camera has limited visual field, can not track the target in the whole course, is difficult to solve the target shielding and the like due to the limitation of the multi-target tracking system. Therefore, at least the second camera module 14 needs to be provided to track the monitoring target object.

It can be distinguished that the first camera module 11 is configured to acquire an image including facial features and physical features of the target object covering the entrance. The second camera module 14 is configured to acquire non-occlusion facial features of the target object at a non-entrance position.

In the specific facial feature recognition scenario of the present application, the second camera module 14 may be disposed in any area of the driver's operation area, the passenger getting-off area, the front of the cabin, and the rear area of the cabin. In other words, the acquisition range of the second camera module 14 may cover any of the driver operation area, the passenger getting-off area, the front cabin area, and the rear cabin area.

Of course, the camera modules can be deployed in the above-mentioned areas, so that the installation positions and angles of a plurality of cameras in the vehicle can be designed according to the pre-established compartment space model. And then calibrating by a camera, and jointly detecting people and objects in the carriage by adopting a mode of combining multiple cameras. And sending the collected video data to the vehicle-mounted terminal, and analyzing and identifying by utilizing the calculation and processing functions of the embedded equipment.

In a specific image acquisition scenario, multiple cameras are distributed at various locations of the car to acquire a more comprehensive image. After the video streams of the facial features and the body features of the target object are acquired through the camera, data in a plurality of video streams are fused in the vehicle-mounted terminal. By simultaneously tracking the movement of the same object in the carriage by using a plurality of cameras, whether the behavior of the object is abnormal or not can be judged in a multi-angle manner.

After the angles of the cameras are adjusted for a plurality of times, each camera is required to be calibrated respectively in order to determine the correlation between the three-dimensional geometric position of a certain point on the surface of the object in the carriage and the corresponding point in the image, the camera is calibrated respectively by using a Zhang Zhengyou calibration method, a geometric model of camera imaging is established, internal and external parameters and distortion coefficients of the camera are obtained, and therefore, the conversion relation between the world coordinate system and the image coordinate system is established.

The abnormal article and mask detection system based on the multiple cameras can better solve the problems by utilizing the advantages of the multiple cameras. In the multi-camera collaborative tracking stage, targets among different cameras are mapped by adopting a target consistency calibration method based on plane homography, polar geometry constraint and camera overlapping area constraint, so that multi-camera fusion and collaborative tracking are conveniently realized. In addition, by means of the target detection network and a typical personnel database, targets in multiple cameras are matched, and the re-identification accuracy can be greatly improved.

In summary, in the abnormal article and mask detection system provided by the application, the first camera module 11 installed in the bus is used for video monitoring, optimizing and combining to design the existing target detection and identification algorithm, and deploying the algorithm into the embedded algorithm board. The method is characterized in that the behaviors of a mask and abnormal objects which are not worn correctly by passengers when riding are identified and judged, the real-time frame capturing and alarm information output of the identified abnormal behavior picture by a local monitoring interface are realized, and the information real-time early warning of a remote platform is realized.

It should be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, the statement "comprises" or "comprising" an element defined by … … does not exclude the presence of other identical elements in a process, method, article or apparatus that comprises the element.

The foregoing is merely exemplary of the present application and is not intended to limit the present application. Various modifications and changes may be made to the present application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc. which are within the spirit and principles of the present application are intended to be included within the scope of the claims of the present application.

Claims

1. An abnormal article and mask detection system, comprising:

2. The abnormal article and mask detection system of claim 1, wherein the identification module is configured to identify an occlusion state of a facial feature of the target object, in particular:

3. The abnormal article and mask detection system according to claim 2, wherein the prompting module is configured to output a prompting message when the shielding state of the facial feature of the target object is not shielded, specifically configured to:

4. The abnormal article and mask detection system of claim 1, wherein the identification module is configured to identify a carrying object of a physical feature of the target object, in particular:

5. The abnormal article and mask detection system according to claim 4, wherein the prompting module is configured to output prompting information when the object carrying the physical feature of the target object is an abnormal article, and is specifically configured to:

calculating the size ratio coefficient of the carrying object;

6. The abnormal article and mask detection system according to any one of claims 2 or 4, wherein the pre-trained yolo_v5s detection model is trained by:

calculating sample loss of the training result;

7. The abnormal article and mask detection system of claim 1, wherein the first camera module comprises at least:

a camera;

8. The abnormal article and mask detection system of claim 7, wherein the identification module is a Jetson Xavier NX edge embedded device connected to the image capture card through a PCI interface, comprising:

and the storage unit is used for storing the image frames in a FIFO queue.

9. The abnormal article and mask detection system of claim 8, wherein the Jetson Xavier NX edge embedded device is configured to discard image frames greater than a predetermined frame rate using a storage policy of frame-by-frame extraction to ensure that the queue does not overflow.

10. The abnormal article and mask detection system of claim 1, further comprising: