CN113470009A

CN113470009A - Illegal umbrella opening detection and identification method and device, electronic equipment and storage medium

Info

Publication number: CN113470009A
Application number: CN202110843093.4A
Authority: CN
Inventors: 余正法; 唐圣; 章合群; 傅凯; 周祥明
Original assignee: Zhejiang Dahua Technology Co Ltd
Current assignee: Zhejiang Dahua Technology Co Ltd
Priority date: 2021-07-26
Filing date: 2021-07-26
Publication date: 2021-10-01
Anticipated expiration: 2041-07-26
Also published as: CN113470009B

Abstract

The invention discloses a method and a device for detecting and identifying illegal umbrella opening, electronic equipment and a storage medium, wherein the method comprises the following steps: acquiring a video to be detected, identifying images in the video to be detected, and determining each umbrella detection area, human body detection area and non-motor vehicle detection area in the images; aiming at each umbrella detection area, respectively determining the intersection ratio of the umbrella detection area and each human body detection area, and determining the intersection ratio of the umbrella detection area and each non-motor vehicle detection area; and determining a target umbrella detection area with illegal umbrella opening behavior from each umbrella detection area according to the intersection ratio of each umbrella detection area to each human body detection area and the intersection ratio of each umbrella detection area to each non-motor vehicle detection area. Therefore, the misdetection of normal umbrella opening behaviors such as pedestrian umbrella opening, non-motor vehicle umbrella opening and the like as illegal umbrella opening behaviors is avoided. Therefore, the illegal umbrella opening detection scheme provided by the embodiment of the invention has better accuracy.

Description

Illegal umbrella opening detection and identification method and device, electronic equipment and storage medium

Technical Field

The invention relates to the technical field of target detection, in particular to a method and a device for detecting and identifying illegal umbrella opening, electronic equipment and a storage medium.

Background

Along with the increasing development of urbanization, the construction of urban public security and urban appearance is more and more emphasized. The illegal umbrella opening detection is an important link of smart city construction, the attractiveness of the city is seriously affected, and the illegal umbrella opening detection mainly comes from a store door, a booth side, a mobile vendor, a street stall and the like at present. The traditional treatment for illegal umbrella opening detection is usually urban management field inspection or the specific position of illegal umbrella opening is counted manually by observing videos, and specific urban management personnel are dispatched to the field to enforce law. The traditional method needs to consume a large amount of city management manpower and material resources, the implementation effect is general, the management is often low in efficiency, timely feedback cannot be achieved, and forensics and law enforcement are not convenient.

With the rapid development of artificial intelligence technology, a plurality of cities begin to increase investment and adopt AI technology to construct smart cities, and illegal umbrella opening behavior in videos detected by a deep learning method appears. After the detection model is trained, the acquired image is input into the detection model, and a detection result of whether illegal umbrella opening exists is output. However, because of the variety of types of umbrellas, including round, square, and varying sizes, other similar shapes to umbrellas include window sunshades, non-motor sunshades, and the like. The prior art scheme is easy to generate false detection, for example, the false detection that the pedestrian opens the umbrella is the illegal opening of the umbrella, and the like.

Therefore, the prior art has the problem of poor detection accuracy of illegal umbrella opening.

Disclosure of Invention

The embodiment of the invention provides a method and a device for detecting and identifying illegal umbrella opening, electronic equipment and a storage medium, which are used for solving the problem of poor accuracy of illegal umbrella opening detection and identification in the prior art.

The embodiment of the invention provides a method for detecting and identifying illegal umbrella opening, which comprises the following steps:

acquiring a video to be detected, identifying images in the video to be detected, and determining each umbrella detection area, human body detection area and non-motor vehicle detection area in the images;

aiming at each umbrella detection area, respectively determining the intersection ratio of the umbrella detection area and each human body detection area, and determining the intersection ratio of the umbrella detection area and each non-motor vehicle detection area;

and determining a target umbrella detection area with illegal umbrella opening behavior from each umbrella detection area according to the intersection ratio of each umbrella detection area to each human body detection area and the intersection ratio of each umbrella detection area to each non-motor vehicle detection area.

Further, the identifying the image in the video to be detected and the determining each umbrella detection area, human body detection area and non-motor vehicle detection area in the image comprises:

inputting images in the video to be detected into a pre-trained target detection model, and determining each umbrella detection area, human body detection area and non-motor vehicle detection area in the images based on the target detection model; the target detection model is obtained by training each sample image in the first training set according to the sample image and the corresponding labeled image, wherein the labeled image is labeled with position information of each umbrella detection area, the human body detection area and the non-motor vehicle detection area in the corresponding sample image.

Further, after determining each umbrella detection area in the image, before determining, for each umbrella detection area, an intersection ratio between the umbrella detection area and each human body detection area, respectively, the method further includes:

and inputting the umbrella detection area into a pre-trained check model aiming at each umbrella detection area, determining the type information of the umbrella detection area based on the check model, if the type information of the umbrella detection area is an umbrella, reserving the umbrella detection area, and otherwise, filtering the umbrella detection area.

Further, the training process of the verification model comprises:

inputting the detection area image and corresponding label information into the verification model aiming at each detection area image in a second training set, and training the verification model by adopting a fine-grained classification algorithm of counterstudy; wherein the tag information includes type information of the corresponding detection area image.

Further, according to the intersection ratio of each umbrella detection area to each human body detection area and the intersection ratio of each umbrella detection area to each non-motor vehicle detection area, determining a target umbrella detection area with an illegal umbrella opening behavior from each umbrella detection area comprises:

for each umbrella detection area, determining whether the umbrella detection area is a fixed position area or a mobile position area according to the position information of the umbrella detection area and the position information of the corresponding umbrella detection area in the image with the preset frame number away from the image;

if the umbrella detection area is a fixed position area, judging whether the maximum intersection ratio of the umbrella detection area and a human body detection area and the maximum intersection ratio of the umbrella detection area and a non-motor vehicle detection area are both smaller than a preset first intersection ratio threshold value, if so, determining that the umbrella detection area is a target umbrella detection area with illegal umbrella opening behavior, and if not, determining that the umbrella detection area is not the target umbrella detection area;

if the umbrella detection area is a mobile position area, judging whether the maximum intersection ratio of the umbrella detection area and a preset type of non-motor vehicle detection area is larger than a preset second intersection ratio threshold value, if so, determining that the umbrella detection area is a target umbrella detection area with illegal umbrella opening behaviors, and if not, determining that the umbrella detection area is not the target umbrella detection area.

Further, if the umbrella detection area is a fixed position area, after the maximum intersection ratio of the umbrella detection area and the human body detection area and the maximum intersection ratio of the umbrella detection area and the non-motor vehicle detection area are both smaller than a preset first intersection ratio threshold value, the umbrella detection area is determined to be before a target umbrella detection area with illegal umbrella opening behavior, and the method further comprises the following steps:

judging whether the umbrella detection area is located within a preset regular area range, and if so, determining that the umbrella detection area is a target umbrella detection area with illegal umbrella opening behavior;

if the umbrella detection area is a mobile position area, after the maximum intersection ratio of the umbrella detection area and a preset type of non-motor vehicle detection area is judged to be larger than a preset second intersection ratio threshold value, the umbrella detection area is determined to be in front of a target umbrella detection area with illegal umbrella opening behaviors, and the method further comprises the following steps:

and judging whether the umbrella detection areas are all located in a preset regular area range in the image with the preset frame number, and if so, determining that the umbrella detection area is a target umbrella detection area with illegal umbrella opening behavior.

Further, after determining that there is a target umbrella detection area for illegal umbrella opening behavior, the method further includes:

and outputting alarm information for prompting the existence of illegal umbrella opening behavior, and carrying the position information of the target umbrella detection area in the alarm information.

On the other hand, the embodiment of the invention provides a device for detecting and identifying illegal umbrella opening, which comprises the following components:

the system comprises a first determining module, a second determining module and a third determining module, wherein the first determining module is used for acquiring a video to be detected, identifying images in the video to be detected and determining each umbrella detection area, human body detection area and non-motor vehicle detection area in the images;

the second determining module is used for respectively determining the intersection ratio of the umbrella detection area and each human body detection area aiming at each umbrella detection area and determining the intersection ratio of the umbrella detection area and each non-motor vehicle detection area;

and the detection module is used for determining a target umbrella detection area with illegal umbrella opening behavior from each umbrella detection area according to the intersection ratio of each umbrella detection area to each human body detection area and the intersection ratio of each umbrella detection area to each non-motor vehicle detection area.

Further, the first determining module is specifically configured to input an image in the video to be detected into a pre-trained target detection model, and determine each umbrella detection area, human body detection area and non-motor vehicle detection area in the image based on the target detection model; the target detection model is obtained by training each sample image in the first training set according to the sample image and the corresponding labeled image, wherein the labeled image is labeled with position information of each umbrella detection area, the human body detection area and the non-motor vehicle detection area in the corresponding sample image.

Further, the apparatus further comprises:

and the screening module is used for inputting the umbrella detection area into a pre-trained check model aiming at each umbrella detection area, determining the type information of the umbrella detection area based on the check model, reserving the umbrella detection area if the type information of the umbrella detection area is an umbrella, and filtering the umbrella detection area if the type information of the umbrella detection area is not the umbrella.

Further, the apparatus further comprises:

the first training module is used for inputting the sample image and the corresponding labeled image into the target detection model aiming at each sample image in the first training set and training the target detection model; and the label image is marked with the position information of each umbrella detection area, the human body detection area and the non-motor vehicle detection area in the corresponding sample image.

Further, the apparatus further comprises:

the second training module is used for inputting the detection area image and corresponding label information into the verification model aiming at each detection area image in a second training set, and training the verification model by adopting a fine-grained classification algorithm of counterstudy; wherein the tag information includes type information of the corresponding detection area image.

Further, the detection module is specifically configured to determine, for each umbrella detection area, whether the umbrella detection area is a fixed position area or a mobile position area according to the position information of the umbrella detection area and the position information of an umbrella detection area corresponding to an image spaced by a preset number of frames from the image; if the umbrella detection area is a fixed position area, judging whether the maximum intersection ratio of the umbrella detection area and a human body detection area and the maximum intersection ratio of the umbrella detection area and a non-motor vehicle detection area are both smaller than a preset first intersection ratio threshold value, if so, determining that the umbrella detection area is a target umbrella detection area with illegal umbrella opening behavior, and if not, determining that the umbrella detection area is not the target umbrella detection area; if the umbrella detection area is a mobile position area, judging whether the maximum intersection ratio of the umbrella detection area and a preset type of non-motor vehicle detection area is larger than a preset second intersection ratio threshold value, if so, determining that the umbrella detection area is a target umbrella detection area with illegal umbrella opening behaviors, and if not, determining that the umbrella detection area is not the target umbrella detection area.

Further, the apparatus further comprises:

the judging module is used for judging the maximum intersection ratio of the umbrella detection area and the human body detection area and the maximum intersection ratio of the umbrella detection area and the non-motor vehicle detection area which are both smaller than a preset first intersection ratio threshold value if the umbrella detection area is a fixed position area, judging whether the umbrella detection area is located in a preset regular area range, and if so, determining that the umbrella detection area is a target umbrella detection area with illegal umbrella opening behavior;

the judging module is further configured to judge whether the maximum intersection-to-parallel ratio of the umbrella detection area to the preset type non-motor vehicle detection area is greater than a preset second intersection-to-parallel ratio threshold value if the umbrella detection area is a mobile position area, judge whether the umbrella detection areas are all located within a preset regular area range in the image with the preset number of frames, and determine that the umbrella detection area is a target umbrella detection area with illegal umbrella opening behavior if the umbrella detection area is the mobile position area.

Further, the apparatus further comprises:

and the alarm module is used for outputting alarm information for prompting the existence of illegal umbrella opening behavior and carrying the position information of the target umbrella detection area in the alarm information.

On the other hand, the embodiment of the invention provides electronic equipment, which comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete mutual communication through the communication bus;

a memory for storing a computer program;

a processor for implementing any of the above method steps when executing a program stored in the memory.

In another aspect, an embodiment of the present invention provides a computer-readable storage medium, in which a computer program is stored, and the computer program, when executed by a processor, implements the method steps of any one of the above.

The embodiment of the invention provides a method and a device for detecting and identifying an illegal umbrella opening, electronic equipment and a storage medium, wherein the method comprises the following steps: acquiring a video to be detected, identifying images in the video to be detected, and determining each umbrella detection area, human body detection area and non-motor vehicle detection area in the images; aiming at each umbrella detection area, respectively determining the intersection ratio of the umbrella detection area and each human body detection area, and determining the intersection ratio of the umbrella detection area and each non-motor vehicle detection area; and determining a target umbrella detection area with illegal umbrella opening behavior from each umbrella detection area according to the intersection ratio of each umbrella detection area to each human body detection area and the intersection ratio of each umbrella detection area to each non-motor vehicle detection area.

The technical scheme has the following advantages or beneficial effects:

in the embodiment of the invention, after the image in the video to be detected is identified, and each umbrella detection area, human body detection area and non-motor vehicle detection area in the image are determined, aiming at each umbrella detection area, the intersection and parallel ratio of the umbrella detection area and each human body detection area is respectively determined, and the intersection and parallel ratio of the umbrella detection area and each non-motor vehicle detection area is determined. And screening each umbrella detection area according to the intersection ratio of each umbrella detection area to each human body detection area and the intersection ratio of each umbrella detection area to each non-motor vehicle detection area, so as to filter out the pedestrian umbrella opening and the normal umbrella opening behavior on the non-motor vehicles, and finally determining the target umbrella detection area with the illegal umbrella opening behavior through screening. Therefore, the misdetection of the normal umbrella opening behavior of pedestrians and non-motor vehicles as the illegal umbrella opening behavior is avoided. Therefore, the illegal umbrella opening detection and identification scheme provided by the embodiment of the invention has better accuracy.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.

Fig. 1 is a schematic diagram of a detection and identification process of an illegal umbrella opening provided in embodiment 1 of the present invention;

FIG. 2 is a flow chart of umbrella screening provided in example 3 of the present invention;

fig. 3 is a flowchart of detection of illegal umbrella opening according to embodiment 4 of the present invention;

fig. 4 is a flowchart of detection of illegal umbrella opening according to embodiment 5 of the present invention;

fig. 5 is a flowchart of another illegal umbrella opening detection provided in embodiment 5 of the present invention;

fig. 6 is an overall flowchart of detection of illegal umbrella opening according to embodiment 5 of the present invention;

fig. 7 is a schematic structural diagram of a device for detecting and identifying an illegal umbrella opening according to embodiment 6 of the present invention;

fig. 8 is a schematic structural diagram of an electronic device according to embodiment 7 of the present invention.

Detailed Description

The present invention will be described in further detail with reference to the attached drawings, and it should be understood that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

Example 1:

fig. 1 is a schematic diagram of a detection and identification process of an illegal umbrella opening provided by an embodiment of the present invention, where the process includes the following steps:

s101: the method comprises the steps of obtaining a video to be detected, identifying images in the video to be detected, and determining each umbrella detection area, human body detection area and non-motor vehicle detection area in the images.

S102: and aiming at each umbrella detection area, respectively determining the intersection ratio of the umbrella detection area and each human body detection area, and determining the intersection ratio of the umbrella detection area and each non-motor vehicle detection area.

S103: and determining a target umbrella detection area with illegal umbrella opening behavior from each umbrella detection area according to the intersection ratio of each umbrella detection area to each human body detection area and the intersection ratio of each umbrella detection area to each non-motor vehicle detection area.

The detection and identification method for the illegal umbrella opening provided by the embodiment of the invention is applied to electronic equipment, and the electronic equipment can be equipment such as a PC (personal computer), a tablet personal computer and the like, and can also be intelligent image acquisition equipment. If the electronic equipment is intelligent image acquisition equipment, the intelligent image acquisition equipment directly performs subsequent detection and identification processes of illegal umbrella opening after acquiring the video to be detected. If the electronic equipment is a PC, a tablet personal computer and the like, after the intelligent image acquisition equipment acquires the video to be detected, the video to be detected is firstly sent to the electronic equipment, and then the electronic equipment carries out the subsequent detection and identification processes of the illegal umbrella opening.

In the embodiment of the invention, after the electronic equipment acquires the video to be detected, the image in the video to be detected is identified, and each umbrella detection area, human body detection area and non-motor vehicle detection area in the image are determined. Specifically, the image in the video to be detected may be identified based on a deep neural network model, may also be identified based on a fractal feature method, and may also be identified based on a wavelet moment method. Thereby determining each umbrella detection area, human detection area and non-motor vehicle detection area in the image. And determining each umbrella detection area, human body detection area and non-motor vehicle detection area as areas contained in the external connection of each umbrella, human body and non-motor vehicle in the image.

And aiming at each umbrella detection area, respectively determining the intersection ratio of the umbrella detection area and each human body detection area, and determining the intersection ratio of the umbrella detection area and each non-motor vehicle detection area. And then screening each umbrella detection area according to the intersection ratio of each umbrella detection area to each human body detection area and the intersection ratio of each umbrella detection area to each non-motor vehicle detection area, and determining a target umbrella detection area with illegal umbrella opening behaviors.

For example, a preset intersection ratio threshold is saved in the electronic device, and for each umbrella detection area, if the maximum intersection ratio of the umbrella detection area and the human body detection area is greater than the preset intersection ratio threshold, the umbrella detection area is considered as the detection area for pedestrian umbrella opening, and at this time, the umbrella detection area is not taken as the target umbrella detection area with illegal umbrella opening behavior. Or if the maximum intersection ratio of the umbrella detection area to the non-motor vehicle detection area is greater than a preset intersection ratio threshold, the umbrella detection area is considered as the umbrella detection area on the non-motor vehicle, and the umbrella detection area is not taken as the target umbrella detection area with illegal umbrella opening behavior. And if the maximum intersection ratio of the umbrella detection area to the human body detection area is not more than a preset intersection ratio threshold value, and the maximum intersection ratio of the umbrella detection area to the non-motor vehicle detection area is not more than a preset intersection ratio threshold value, taking the umbrella detection area as a target umbrella detection area with illegal umbrella opening behaviors.

Example 2:

in order to determine each umbrella detection area, human detection area and non-motor vehicle detection area in an image, on the basis of the above embodiments, in an embodiment of the present invention, the identifying an image in the video to be detected, and determining each umbrella detection area, human detection area and non-motor vehicle detection area in the image includes:

In the embodiment of the invention, the electronic equipment stores a pre-trained target detection model which is trained according to the sample images in the training set and the corresponding labeled images.

Specifically, the training process of the target detection model includes:

for each sample image in a first training set, inputting the sample image and a corresponding label image into the target detection model, and training the target detection model; and the label image is marked with the position information of each umbrella detection area, the human body detection area and the non-motor vehicle detection area in the corresponding sample image.

The electronic equipment stores a first training set, wherein the first training set comprises a large number of sample images, and each sample image is provided with a corresponding labeled image. And the label image is marked with the position information of each umbrella detection area, the human body detection area and the non-motor vehicle detection area in the corresponding sample image. And inputting the sample image and the corresponding label image into a target detection model aiming at each sample image in the first training set, wherein the target detection model can output a training image corresponding to the sample image, the training image comprises position information of each umbrella detection area, human body detection area and non-motor vehicle detection area, and then, a loss value is calculated according to the position information of each umbrella detection area, human body detection area and non-motor vehicle detection area in the training image and the position information of each umbrella detection area, human body detection area and non-motor vehicle detection area in the label image corresponding to the sample image. And adjusting the parameter values of all parameters in the target detection model according to the loss values. When the convergence condition is reached, the target detection model training is completed.

And inputting the image in the video to be detected into a pre-trained target detection model, and outputting each umbrella detection area, human body detection area and non-motor vehicle detection area in the image by the target detection model.

Example 3:

in order to make identification of an illegal umbrella opening more accurate, on the basis of the above embodiments, in an embodiment of the present invention, after determining each umbrella detection area in the image, for each umbrella detection area, before determining an intersection ratio between the umbrella detection area and each human body detection area, the method further includes:

In the embodiment of the invention, the electronic equipment stores a pre-trained verification model, and the verification model is used for verifying and screening each umbrella detection area in the identified image.

Specifically, after the electronic device identifies each umbrella detection area in the image, the electronic device inputs the pre-trained verification model into the umbrella detection area for each umbrella detection area, and determines the type information of the umbrella detection area based on the verification model. If the type information of the umbrella detection area is the umbrella, the umbrella detection area is reserved, otherwise, the umbrella detection area is filtered.

The training process of the verification model comprises the following steps:

The electronic equipment stores a second training set, the second training set comprises a large number of detection area images, and each detection area image has corresponding label information. Wherein the tag information includes type information of the corresponding detection area image. And inputting the detection area image and the corresponding label information into a verification model aiming at each detection area image in the second training set, outputting training label information corresponding to the detection area image by the verification model, and calculating a loss value according to the training label information and the label information corresponding to the detection area image. And adjusting the parameter values of all parameters in the verification model according to the loss values. When the convergence condition is reached, the calibration model training is completed.

And inputting the umbrella detection area into a pre-trained verification model aiming at each umbrella detection area, and outputting the type information of the umbrella detection area by the verification model.

Specifically, the check model is used for improving the detection precision of the umbrella and eliminating a target frame similar to the umbrella. The verification model is a fine-grained umbrella classification model, the input of which is 128 × 128 pixels in size, and the model is very small. A countercheck learning training strategy is introduced in model training, the confusion of regions with countercheck pictures of 8 × 8 and 16 × 16 is carried out, the reconstruction of the confusion regions is carried out when the pictures are sampled at 8 times and 16 times, and the model can learn the information of fine differences between different classes. The verification model loss comprises three parts: classification loss, counterlearning loss, and region reconstruction loss. The overall overview of the quadratic model verification is as follows: firstly, each identified umbrella detection area is obtained, then each umbrella detection area is cut on the original image, then the image is zoomed to 128 × 128 pixels, and then the image is sent to the check model. And the verification model outputs the class information, when the output class information is consistent with the detection class information, the output class information is judged to be the umbrella type, the umbrella detection area is reserved, and otherwise, the umbrella detection area is abandoned.

Fig. 2 is a flow chart of umbrella screening provided in the embodiment of the present invention, which includes the following steps:

s201: inputting images in a video to be detected into a pre-trained target detection model, and determining each umbrella detection area in the images based on the target detection model.

S202: and inputting the umbrella detection area into a pre-trained check model aiming at each umbrella detection area, determining the type information of the umbrella detection area based on the check model, if the type information of the umbrella detection area is an umbrella, reserving the umbrella detection area, and otherwise, filtering the umbrella detection area.

Example 4:

in order to make detection of illegal umbrella opening more accurate, on the basis of the above embodiments, in an embodiment of the present invention, according to an intersection ratio of each umbrella detection area to each human body detection area and an intersection ratio of each umbrella detection area to each non-motor vehicle detection area, determining, from each umbrella detection area, a target umbrella detection area where illegal umbrella opening behavior exists includes:

In the embodiment of the invention, different strategies are adopted to detect whether the illegal umbrella opening is carried out on the umbrella with a moving position and the umbrella with a fixed position in the video, so that the illegal umbrella opening detection is more accurate.

First, for each umbrella detection area, it is determined whether the umbrella detection area is a fixed position area or a moving position area based on the position information of the umbrella detection area and the position information of the corresponding umbrella detection area in the image spaced by a preset number of frames from the image.

The electronic equipment acquires a video to be detected, identifies each frame of image in the video to be detected, and determines each umbrella detection area in each frame of image and the ID corresponding to each umbrella detection area. And determining two umbrella detection areas with the same ID in the two images with the preset frame number. And determining whether the umbrella detection area is a fixed position area or a moving position area according to the position information of the two umbrella detection areas with the same ID in the image. The electronic device may store a preset distance threshold, determine, for two umbrella detection areas with the same ID, a distance between the two umbrella detection areas with the same ID according to position information of the two umbrella detection areas with the same ID in the image, and if the distance is greater than the preset distance threshold, determine that the umbrella detection area is a mobile position area, otherwise determine that the umbrella detection area is a fixed position area.

If the umbrella detection area is a fixed position area, judging whether the maximum intersection ratio of the umbrella detection area and the human body detection area and the maximum intersection ratio of the umbrella detection area and the non-motor vehicle detection area are both smaller than a preset first intersection ratio threshold value, if so, determining that the umbrella detection area is a target umbrella detection area with illegal umbrella opening behaviors, and if not, determining that the umbrella detection area is not the target umbrella detection area. Therefore, the umbrella detection area of the sunshade on the pedestrian umbrella or the non-motor vehicle in the static state is filtered, and the target umbrella detection area for illegally opening the umbrella is reserved.

If the umbrella detection area is a mobile position area, judging whether the maximum intersection ratio of the umbrella detection area and a preset type of non-motor vehicle detection area is larger than a preset second intersection ratio threshold value, if so, determining that the umbrella detection area is a target umbrella detection area with illegal umbrella opening behaviors, and if not, determining that the umbrella detection area is not the target umbrella detection area. The preset category of non-motor vehicles are non-motor vehicles commonly used by statistical mobile vendors, such as electric tricycles. The electronic equipment acquires a video to be detected, identifies each frame of image in the video to be detected, and determines each non-motor vehicle detection area in each frame of image and non-motor vehicle type information corresponding to each non-motor vehicle detection area, wherein the non-motor vehicle type information is, for example, electric bicycles, electric tricycles and the like. And for a moving umbrella detection area, if the maximum intersection ratio of the umbrella detection area and a preset type of non-motor vehicle detection area is greater than a preset second intersection ratio threshold value, the umbrella detection area is an umbrella detection area for opening an umbrella by a mobile dealer, and the umbrella detection area is determined to be a target umbrella detection area with illegal umbrella opening behaviors. Therefore, the target umbrella detection area with the illegal umbrella opening behavior is effectively and accurately determined.

Fig. 3 is a flowchart of detection for illegal umbrella opening according to an embodiment of the present invention, where the process includes the following steps:

s301: the method comprises the steps of obtaining a video to be detected, identifying images in the video to be detected, and determining each umbrella detection area, human body detection area and non-motor vehicle detection area in the images.

S302: and aiming at each umbrella detection area, respectively determining the intersection ratio of the umbrella detection area and each human body detection area, and determining the intersection ratio of the umbrella detection area and each non-motor vehicle detection area.

S303: and for each umbrella detection area, determining whether the umbrella detection area is a fixed position area or a mobile position area according to the position information of the umbrella detection area and the position information of the corresponding umbrella detection area in the image with the preset number of frames away from the image, and if the umbrella detection area is determined to be the fixed position area, performing S304, and if the umbrella detection area is determined to be the mobile position area, performing S305.

S304: and judging whether the maximum intersection ratio of the umbrella detection area to the human body detection area and the maximum intersection ratio of the umbrella detection area to the non-motor vehicle detection area are both smaller than a preset first intersection ratio threshold value, if so, determining that the umbrella detection area is a target umbrella detection area with illegal umbrella opening behaviors, and if not, determining that the umbrella detection area is not the target umbrella detection area.

S305: and judging whether the maximum intersection ratio of the umbrella detection area to the non-motor vehicle detection area of the preset type is larger than a preset second intersection ratio threshold value, if so, determining that the umbrella detection area is a target umbrella detection area with illegal umbrella opening behaviors, and if not, determining that the umbrella detection area is not the target umbrella detection area.

The fact shows that the result output by the target detection model cannot be directly used as the basis of illegal umbrella opening because the real events comprise too many non-illegal umbrella opening events. The embodiment of the invention mainly eliminates the determined non-illegal umbrella opening event from the detection results, such as pedestrian umbrella opening, sunshade of an electric vehicle or a motorcycle, and the other detection results need to be filtered for the parking event judgment of targets, such as movement of illegal umbrella on a mobile vendor vehicle (usually a tricycle) or pedestrian umbrella moving, and the like. In the embodiment of the invention, the detection target information output by the target detection model is acquired, wherein the detection target information comprises category information, position information and target unique informationid. An umbrella detection area, a human detection area, and a non-motor vehicle detection area. And respectively calculating the intersection ratio IOU of the umbrella detection area and the human body detection area, and calculating the intersection ratio IOU of the umbrella detection area and the non-motor vehicle detection area. And tracking the id by adopting a multi-target tracking algorithm, and keeping an id pair of the IOU of the pedestrians, the non-motor vehicles and the umbrella, which is larger than a threshold value alpha, and an id pair of the maximum IOU for the specific position of the target id of each frame of the umbrella. Let the distance threshold be beta, every k frames later, if

(m represents the mth frame, n represents the target with id n, and d represents the target position information) then represents that the umbrella position is a fixed position, otherwise the umbrella is in a moving state. For the umbrella in a fixed position state, the umbrella id larger than alpha in the fixed position state is removed, the umbrella is a pedestrian umbrella in a static state or a sunshade on a motorcycle, and a correct illegal umbrella in the static state is reserved. For the umbrella in the moving state, the id pair of the tricycle and the illegal umbrella is reserved, and the judgment is also made for the fact that a new umbrella target enters in the follow-up process. The non-violating umbrellas identified so far can be completely located.

The parameters alpha and beta are parameters for adjusting the sensitivity of the algorithm, beta is used for measuring external environment interference factors, beta can be adjusted to be larger when the external environment changes to be larger, alpha reflects the density degree of the detected target, and alpha can be properly improved when the target aggregation is serious, and a group of parameters alpha and beta can be generated through the number of pairs of the target and the IOU.

Example 5:

in order to enable detection of illegal umbrella opening to be more accurate, on the basis of the above embodiments, in an embodiment of the present invention, if the umbrella detection area is a fixed position area, after determining that the maximum intersection ratio between the umbrella detection area and a human body detection area and the maximum intersection ratio between the umbrella detection area and a non-motor vehicle detection area are both less than a preset first intersection ratio threshold, it is determined that the umbrella detection area is a target umbrella detection area where illegal umbrella opening behavior exists, the method further includes:

Considering that the illegal umbrella opening event mainly occurs in areas such as the front of a shop door, two sides of a traffic road and the like, in the embodiment of the invention, the electronic device stores the preset regular area range. The preset regular area range is, for example, an area range in front of a shop door, at both sides of a traffic road, and the like in the screen.

Fig. 4 is a flowchart of detection for illegal umbrella opening according to an embodiment of the present invention, which includes the following steps:

s401: and if the umbrella detection area is a fixed position area, determining the maximum intersection ratio of the umbrella detection area and the human body detection area, and the maximum intersection ratio of the umbrella detection area and the non-motor vehicle detection area are both smaller than a preset first intersection ratio threshold value.

S402: and judging whether the umbrella detection area is located in a preset regular area range, if so, performing S403, and if not, performing S404.

S403: and determining that the umbrella detection area is a target umbrella detection area with illegal umbrella opening behavior.

S404: and determining that the umbrella detection area is not the target umbrella detection area with the illegal umbrella opening behavior.

Fig. 5 is a flowchart of detection for illegal umbrella opening according to an embodiment of the present invention, which includes the following steps:

s501: and if the umbrella detection area is a mobile position area, determining that the maximum intersection ratio of the umbrella detection area and the non-motor vehicle detection area of the preset type is greater than a preset second intersection ratio threshold value.

S502: and judging whether the umbrella detection areas are all located in the range of a preset regular area in the image with the preset frame number, if so, performing S403, and if not, performing S404.

S503: and determining that the umbrella detection area is a target umbrella detection area with illegal umbrella opening behavior.

S504: and determining that the umbrella detection area is not the target umbrella detection area with the illegal umbrella opening behavior.

In the embodiment of the present invention, the preset regular area range may be a partial area range in the image or a full screen range. The accuracy rate of positive events is high and the false alarm is low, but the umbrella opening event is a random event, the umbrella opening position is usually not fixed, the common central point of the illegal umbrella cover is higher, the alarm cannot be triggered, the full-screen detection recall rate is high, and the false alarm can be increased. For the illegal umbrella opening event, the regular areas are mainly arranged in front of the shop door and at two sides of the traffic road, and the regular areas can be arranged to be high enough.

In the embodiment of the invention, whether the umbrella at the fixed position is in the regular area or not is judged, and if the umbrella is in the regular area, the target umbrella detection area with illegal umbrella opening behavior is determined. And tracking the id of the moving umbrella, and if the umbrella detection areas are all located in the range of the preset regular area in the image with the preset frame number, determining the target umbrella detection area with the illegal umbrella opening behavior.

In this embodiment of the present invention, after determining that there is a target umbrella detection area with an illegal umbrella opening behavior, the method further includes:

In order to facilitate reminding personnel of illegal umbrella opening behaviors, and enable related personnel to timely process illegal umbrella opening events, in the embodiment of the invention, after a target umbrella detection area with the illegal umbrella opening behaviors is determined, alarm information used for prompting the existence of the illegal umbrella opening behaviors is output, and the position information of the target umbrella detection area is carried in the alarm information. The alarm information may be character alarm information, for example, a video picture and alarm content are displayed on a display screen of the electronic device, and the alarm content includes position information of a target umbrella detection area. Preferably, the alarm content can be displayed in a manner of being superposed on the position of the target umbrella detection area in the video picture. The alarm information may also be voice alarm information, for example, an alarm voice is output through a speaker of the electronic device, wherein the alarm voice carries the position information of the target umbrella detection area.

The embodiment of the invention is based on target detection and identification of deep learning, and adds a secondary verification module, an event filtering module and an illegal target rule alarming module on the basis of detection, wherein the former two modules mainly improve the detection precision of a target, and the latter gives an alarming rule for the illegal event. The whole process of the illegal umbrella opening detection scheme is as follows: the training data set of the collected umbrellas is used for training an illegal umbrella opening target detection model, attention needs to be paid to a training strategy different from other methods, the umbrella data comprise common umbrella covering and illegal large umbrellas for pedestrians, the illegal large umbrellas comprise circular umbrellas and four-corner umbrellas, and the umbrellas carried by the pedestrians are usually circular. Compared with a four-corner umbrella, the round umbrella is easier to detect by mistake, most of negative samples and difficult samples are round umbrellas supported by pedestrians and non-motor vehicles, the difficulty in the scheme is the distinction of the positive samples and the negative samples, and the key point is to remove the false detection.

Fig. 6 is an overall flowchart of detection of illegal umbrella opening according to the embodiment of the present invention. And outputting the video to be detected to an umbrella detection area, a human body detection area and a non-motor vehicle detection area through a target detection model. And sending the umbrella detection area into a verification model, and screening the umbrella detection area. Different strategies are adopted to detect whether the umbrella with the moving position and the umbrella with the fixed position in the video are illegally opened. And performing final illegal umbrella opening detection according to a preset regular area range.

The overall scheme is summarized as follows: the video stream to be detected passes through a target detection model and is output to an umbrella detection area, a human body detection area and a non-motor vehicle detection area, then the umbrella detection area is sent to a verification model, and False Positives (FP) targets similar to umbrellas, such as a shop lintel, a rain shelter of a window and the like, can be removed through the verification model. And filtering the umbrellas supported by the pedestrians and the non-motor vehicles by the umbrella filtering rule module in the remaining umbrella detection area, human body detection area and non-motor vehicle detection area. And sending the filtered target frame to an alarm rule module for illegal umbrella opening, and finally judging whether the umbrella opening is illegal or not through detailed logic. The umbrella filtering rule module detects whether the umbrella opening is illegal according to different strategies according to the umbrella with the movable position and the umbrella with the fixed position in the video. And the alarm rule module performs a final illegal umbrella opening detection process according to a preset rule area range.

Example 6:

fig. 7 is a schematic structural diagram of a device for detecting and identifying an illegal umbrella opening, provided by an embodiment of the present invention, where the device includes:

the first determining module 71 is configured to acquire a video to be detected, identify an image in the video to be detected, and determine each umbrella detection area, human body detection area and non-motor vehicle detection area in the image;

a second determining module 72, configured to determine, for each umbrella detection area, an intersection ratio between the umbrella detection area and each human detection area, and determine an intersection ratio between the umbrella detection area and each non-motor vehicle detection area;

and the detection module 73 is used for determining a target umbrella detection area with illegal umbrella opening behavior from each umbrella detection area according to the intersection ratio of each umbrella detection area to each human body detection area and the intersection ratio of each umbrella detection area to each non-motor vehicle detection area.

The first determining module 71 is specifically configured to input an image in the video to be detected into a pre-trained target detection model, and determine each umbrella detection area, human body detection area and non-motor vehicle detection area in the image based on the target detection model; the target detection model is obtained by training each sample image in the first training set according to the sample image and the corresponding labeled image, wherein the labeled image is labeled with position information of each umbrella detection area, the human body detection area and the non-motor vehicle detection area in the corresponding sample image.

The device further comprises:

and the screening module 74 is configured to input the umbrella detection area into a pre-trained verification model for each umbrella detection area, determine type information of the umbrella detection area based on the verification model, reserve the umbrella detection area if the type information of the umbrella detection area is an umbrella, and filter the umbrella detection area if the type information of the umbrella detection area is an umbrella.

The device further comprises:

a first training module 75, configured to input, for each sample image in the first training set, the sample image and the corresponding labeled image into the target detection model, and train the target detection model; and the label image is marked with the position information of each umbrella detection area, the human body detection area and the non-motor vehicle detection area in the corresponding sample image.

The device further comprises:

a second training module 76, configured to input, for each detection area image in the second training set, the detection area image and corresponding tag information into the verification model, and train the verification model by using a fine-grained classification algorithm for counterstudy; wherein the tag information includes type information of the corresponding detection area image.

The detection module 73 is specifically configured to determine, for each umbrella detection area, whether the umbrella detection area is a fixed position area or a mobile position area according to position information of the umbrella detection area and position information of an umbrella detection area corresponding to an image that is spaced by a preset number of frames from the image; if the umbrella detection area is a fixed position area, judging whether the maximum intersection ratio of the umbrella detection area and a human body detection area and the maximum intersection ratio of the umbrella detection area and a non-motor vehicle detection area are both smaller than a preset first intersection ratio threshold value, if so, determining that the umbrella detection area is a target umbrella detection area with illegal umbrella opening behavior, and if not, determining that the umbrella detection area is not the target umbrella detection area; if the umbrella detection area is a mobile position area, judging whether the maximum intersection ratio of the umbrella detection area and a preset type of non-motor vehicle detection area is larger than a preset second intersection ratio threshold value, if so, determining that the umbrella detection area is a target umbrella detection area with illegal umbrella opening behaviors, and if not, determining that the umbrella detection area is not the target umbrella detection area.

The device further comprises:

a judging module 77, configured to, if the umbrella detection area is a fixed position area, judge a maximum intersection ratio between the umbrella detection area and a human body detection area, and the maximum intersection ratio between the umbrella detection area and a non-motor vehicle detection area is smaller than a preset first intersection ratio threshold, judge whether the umbrella detection area is within a preset regular area range, and if so, determine that the umbrella detection area is a target umbrella detection area where an illegal umbrella opening behavior exists;

the determining module 77 is further configured to determine, if the umbrella detection area is a mobile position area, that a maximum intersection ratio between the umbrella detection area and a preset type of non-motor vehicle detection area is greater than a preset second intersection ratio threshold, determine whether the umbrella detection areas are all located within a preset regular area range in an image with a preset number of frames, and if so, determine that the umbrella detection area is a target umbrella detection area with an illegal umbrella opening behavior.

The device further comprises:

and the alarm module 78 is used for outputting alarm information for prompting that an illegal umbrella opening behavior exists, and carrying the position information of the target umbrella detection area in the alarm information.

Example 7:

on the basis of the foregoing embodiments, an embodiment of the present invention further provides an electronic device, as shown in fig. 8, including: the system comprises a processor 801, a communication interface 802, a memory 803 and a communication bus 804, wherein the processor 801, the communication interface 802 and the memory 803 complete mutual communication through the communication bus 804;

the memory 803 has stored therein a computer program which, when executed by the processor 801, causes the processor 801 to perform the steps of:

Based on the same inventive concept, the embodiment of the invention also provides an electronic device, and as the principle of solving the problems of the electronic device is similar to the detection and identification method of the illegal umbrella opening, the implementation of the electronic device can refer to the implementation of the method, and repeated details are omitted.

The electronic device provided by the embodiment of the invention can be a desktop computer, a portable computer, a smart phone, a tablet computer, a Personal Digital Assistant (PDA), a network side device and the like.

The communication bus mentioned in the electronic device may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The communication bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown, but this does not mean that there is only one bus or one type of bus.

The communication interface 802 is used for communication between the above-described electronic apparatus and other apparatuses.

The Memory may include a Random Access Memory (RAM) or a Non-Volatile Memory (NVM), such as at least one disk Memory. Alternatively, the memory may be at least one memory device located remotely from the processor.

The Processor may be a general-purpose Processor, including a central processing unit, a Network Processor (NP), and the like; but may also be a Digital Signal Processor (DSP), an application specific integrated circuit, a field programmable gate array or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or the like.

When the processor executes the program stored in the memory, the video to be detected is acquired, the image in the video to be detected is identified, and each umbrella detection area, human body detection area and non-motor vehicle detection area in the image are determined; aiming at each umbrella detection area, respectively determining the intersection ratio of the umbrella detection area and each human body detection area, and determining the intersection ratio of the umbrella detection area and each non-motor vehicle detection area; and determining a target umbrella detection area with illegal umbrella opening behavior from each umbrella detection area according to the intersection ratio of each umbrella detection area to each human body detection area and the intersection ratio of each umbrella detection area to each non-motor vehicle detection area. In the embodiment of the invention, after the image in the video to be detected is identified, and each umbrella detection area, human body detection area and non-motor vehicle detection area in the image are determined, aiming at each umbrella detection area, the intersection and parallel ratio of the umbrella detection area and each human body detection area is respectively determined, and the intersection and parallel ratio of the umbrella detection area and each non-motor vehicle detection area is determined. And screening each umbrella detection area according to the intersection ratio of each umbrella detection area to each human body detection area and the intersection ratio of each umbrella detection area to each non-motor vehicle detection area, so as to filter out the pedestrian umbrella opening and the normal umbrella opening behavior on the non-motor vehicles, and finally determining the target umbrella detection area with the illegal umbrella opening behavior through screening. Therefore, the misdetection of the normal umbrella opening behavior of pedestrians and non-motor vehicles as the illegal umbrella opening behavior is avoided. Therefore, the illegal umbrella opening detection and identification scheme provided by the embodiment of the invention has better accuracy.

Example 8:

on the basis of the foregoing embodiments, an embodiment of the present invention further provides a computer storage readable storage medium, in which a computer program executable by an electronic device is stored, and when the program is run on the electronic device, the electronic device is caused to execute the following steps:

Based on the same inventive concept, embodiments of the present invention further provide a computer-readable storage medium, and since a principle of solving a problem when a processor executes a computer program stored on the computer-readable storage medium is similar to the method for detecting and identifying an illegal umbrella opening, the method may be used for implementing the computer program stored on the computer-readable storage medium by the processor, and repeated details are not repeated.

The computer readable storage medium may be any available medium or data storage device that can be accessed by a processor in an electronic device, including but not limited to magnetic memory such as floppy disks, hard disks, magnetic tape, magneto-optical disks (MOs), etc., optical memory such as CDs, DVDs, BDs, HVDs, etc., and semiconductor memory such as ROMs, EPROMs, EEPROMs, non-volatile memory (NAND FLASH), Solid State Disks (SSDs), etc.

The computer program is stored in a computer readable storage medium provided by the embodiment of the invention, and when being executed by a processor, the computer program realizes the acquisition of a video to be detected, the identification of an image in the video to be detected and the determination of each umbrella detection area, human body detection area and non-motor vehicle detection area in the image; aiming at each umbrella detection area, respectively determining the intersection ratio of the umbrella detection area and each human body detection area, and determining the intersection ratio of the umbrella detection area and each non-motor vehicle detection area; and determining a target umbrella detection area with illegal umbrella opening behavior from each umbrella detection area according to the intersection ratio of each umbrella detection area to each human body detection area and the intersection ratio of each umbrella detection area to each non-motor vehicle detection area. In the embodiment of the invention, after the image in the video to be detected is identified, and each umbrella detection area, human body detection area and non-motor vehicle detection area in the image are determined, aiming at each umbrella detection area, the intersection and parallel ratio of the umbrella detection area and each human body detection area is respectively determined, and the intersection and parallel ratio of the umbrella detection area and each non-motor vehicle detection area is determined. And screening each umbrella detection area according to the intersection ratio of each umbrella detection area to each human body detection area and the intersection ratio of each umbrella detection area to each non-motor vehicle detection area, so as to filter out the pedestrian umbrella opening and the normal umbrella opening behavior on the non-motor vehicles, and finally determining the target umbrella detection area with the illegal umbrella opening behavior through screening. Therefore, the misdetection of the normal umbrella opening behavior of pedestrians and non-motor vehicles as the illegal umbrella opening behavior is avoided. Therefore, the illegal umbrella opening detection and identification scheme provided by the embodiment of the invention has better accuracy.

The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

While preferred embodiments of the present invention have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. Therefore, it is intended that the appended claims be interpreted as including preferred embodiments and all such alterations and modifications as fall within the scope of the invention.

It will be apparent to those skilled in the art that various changes and modifications may be made in the present invention without departing from the spirit and scope of the invention. Thus, if such modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include such modifications and variations.

Claims

1. A method for detecting and identifying an illegal umbrella opening, the method comprising:

2. The method of claim 1, wherein the identifying the image in the video to be detected and the determining each of the umbrella detection area, the human detection area, and the non-motor vehicle detection area in the image comprises:

3. The method of claim 1, wherein after determining each umbrella detection area in the image, before determining, for each umbrella detection area, a union ratio of the umbrella detection area to each human detection area, respectively, the method further comprises:

4. The method of claim 3, wherein the training process of the verification model comprises:

5. The method of claim 1, wherein the determining a target umbrella detection area from each umbrella detection area for which an illegal umbrella opening behavior exists according to the intersection ratio of each umbrella detection area to each human detection area and the intersection ratio of each umbrella detection area to each non-motor vehicle detection area comprises:

6. The method according to claim 5, wherein if the umbrella detection area is a fixed position area, after the maximum intersection ratio of the umbrella detection area and the human body detection area is judged and the maximum intersection ratio of the umbrella detection area and the non-motor vehicle detection area is smaller than a preset first intersection ratio threshold value, before the umbrella detection area is determined to be a target umbrella detection area with illegal umbrella opening behavior, the method further comprises:

7. The method of claim 1, wherein after determining that a target umbrella detection area for illegal umbrella opening behavior exists, the method further comprises:

8. A device for detecting and identifying an offending umbrella support, the device comprising:

9. An electronic device is characterized by comprising a processor, a communication interface, a memory and a communication bus, wherein the processor and the communication interface are used for realizing mutual communication by the memory through the communication bus;

a memory for storing a computer program;

a processor for implementing the method steps of any one of claims 1 to 7 when executing a program stored in the memory.

10. A computer-readable storage medium, characterized in that a computer program is stored in the computer-readable storage medium, which computer program, when being executed by a processor, carries out the method steps of any one of claims 1 to 7.