CN116994121A

CN116994121A - Method and device for detecting illegal coded articles, electronic equipment and storage medium

Info

Publication number: CN116994121A
Application number: CN202210422794.5A
Authority: CN
Inventors: 程啸
Original assignee: SF Technology Co Ltd
Current assignee: SF Technology Co Ltd
Priority date: 2022-04-21
Filing date: 2022-04-21
Publication date: 2023-11-03

Abstract

The application provides a method and a device for detecting illegal goods, electronic equipment and a computer readable storage medium. The method for detecting the illegal coded articles comprises the following steps: detecting based on an image to be identified of an area to be detected to obtain an illegal coding prediction area in the image to be identified; detecting based on the image to be identified to obtain an interference prediction area in the image to be identified; acquiring a first overlapping degree between an illegal coding prediction area in the image to be identified and an interference prediction area in the image to be identified; and if the first overlapping degree is smaller than a first preset degree threshold value, determining that illegal goods are placed in the area to be detected. The application can improve the detection accuracy of illegal goods.

Description

Method and device for detecting illegal coded articles, electronic equipment and storage medium

Technical Field

The application relates to the technical field of computer vision, in particular to a method and a device for detecting illegal goods, electronic equipment and a computer readable storage medium.

Background

In recent years, with rapid development of computer vision technology, automation capability of various industries has been increased, wherein whether articles are placed illegally (for example, in order to pursue timeliness, an article sorting person may occur illegal goods during the process of sorting the articles) can also be detected by the computer vision technology. Therefore, in the prior art, a regression classification is performed on a target area in a monitored image or video through a monitored image or video detection technology, so as to determine whether an article in the monitored image or video has illegal codes.

However, the inventor finds that, in the actual research and development process, because of the larger interference in the image or video, the accuracy of the detection technology is lower, and the service requirement cannot be met.

Disclosure of Invention

The application provides a method, a device, electronic equipment and a computer readable storage medium for detecting illegal goods, which can improve the detection accuracy of illegal goods.

In a first aspect, the present application provides a method for detecting an offending article, the method comprising:

detecting based on an image to be identified of an area to be detected to obtain an illegal coding prediction area in the image to be identified;

detecting based on the image to be identified to obtain an interference prediction area in the image to be identified;

acquiring a first overlapping degree between an illegal coding prediction area in the image to be identified and an interference prediction area in the image to be identified;

and if the first overlapping degree is smaller than a first preset degree threshold value, determining that illegal goods are placed in the area to be detected.

In a second aspect, the present application provides a detection apparatus for illicitly placing an item, the detection apparatus for illicitly placing an item comprising:

The detection unit is used for detecting based on an image to be identified of the area to be detected, and obtaining an illegal coding prediction area in the image to be identified;

the detection unit is further used for detecting based on the image to be identified to obtain an interference prediction area in the image to be identified;

the acquisition unit is used for acquiring a first overlapping degree between the illegal coding prediction area in the image to be identified and the interference prediction area in the image to be identified;

and the judging unit is used for determining that illegal goods are placed in the area to be detected if the first overlapping degree is smaller than a first preset degree threshold value.

In some embodiments of the present application, the determining unit is specifically configured to:

detecting an illegal coding prediction area in a j-th image of the area to be detected and an interference prediction area in the j-th image, wherein the j-th image is each image of K images acquired after the image to be identified, and the interval duration between the acquisition time of the image to be identified and the acquisition time of the j-th image is within a preset duration range;

if the illegal coding prediction area in the jth image and the interference prediction area in the jth image are overlapped to a second degree;

And if the first overlapping degree is smaller than a first preset degree threshold value and the second overlapping degree is smaller than the first preset degree threshold value, determining that illegal goods exist in the to-be-detected area.

In some embodiments of the present application, the detection device for illegal items further includes an early warning unit, where the early warning unit is specifically configured to:

determining a first recording area of the currently detected illegal goods in the to-be-detected area based on the illegal goods placement prediction area in the to-be-identified image and the illegal goods placement prediction area in the j-th image;

acquiring a second recording area of the illegal goods in the area to be detected, which is detected last time;

and if the third overlapping degree between the first recording area and the second recording area is smaller than a second preset degree threshold, outputting early warning information of illegal goods in the area to be detected.

In some embodiments of the present application, the early warning unit is specifically configured to:

if the third overlapping degree is greater than or equal to a second preset degree threshold, detecting a target time interval between the occurrence time of the illegal article in the to-be-detected area detected last time and the occurrence time of the illegal article in the to-be-detected area detected this time;

And if the target time interval is larger than the preset time interval, outputting early warning information of illegal goods in the to-be-detected area.

In some embodiments of the present application, the detection unit is specifically configured to:

extracting features of an image to be identified of a region to be detected to obtain image features of the image to be identified;

predicting based on the image characteristics of the image to be identified, and obtaining the central point prediction coordinates of the bounding box of the illegal coding region in the image to be identified and the angular point prediction coordinates of the bounding box;

and determining an illegal coding prediction area in the image to be identified based on the central point prediction coordinates and the angular point prediction coordinates.

if the illegal coding prediction area exists in the image to be identified and the interference area does not exist, determining that illegal coding objects exist in the area to be detected;

or if the illegal coding prediction area does not exist in the image to be identified, determining that illegal coding objects do not exist in the area to be detected.

In some embodiments of the present application, the acquiring unit is specifically configured to:

And acquiring the intersection ratio between the illegal coding prediction area in the image to be identified and the interference prediction area in the image to be identified as the first overlapping degree.

In a third aspect, the present application also provides an electronic device, where the electronic device includes a processor and a memory, where the memory stores a computer program, and when the processor invokes the computer program in the memory, the processor executes steps in any of the detection methods for illegal coded articles provided in the present application.

In a fourth aspect, the present application also provides a computer readable storage medium having stored thereon a computer program, the computer program being loaded by a processor to perform the steps of the method of detecting a offending article.

When the first overlapping degree between the illegal coding prediction area in the image to be identified and the interference prediction area in the image to be identified is smaller than a first preset degree threshold value, determining that illegal coding objects exist in the area to be detected; when the first overlapping degree between the illegal coding prediction area in the image to be identified and the interference prediction area in the image to be identified is smaller, the influence of the illegal coding prediction area in the image to be identified on the interference area is proved to be smaller, namely the detection accuracy of the illegal coding prediction area in the image to be identified is higher, so that the detection accuracy of illegal coding objects can be improved.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the description of the embodiments will be briefly described below, it being obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

FIG. 1 is a schematic view of a system for detecting offensive items according to an embodiment of the present application;

FIG. 2 is a schematic flow chart of a method for detecting illegal items according to an embodiment of the present application;

FIG. 3 is a schematic diagram of a network structure of a single-stage object detection algorithm according to an embodiment of the present application;

FIG. 4 is a schematic diagram of another network architecture of a single-stage object detection algorithm provided in an embodiment of the present application;

FIG. 5 is a schematic illustration of bounding boxes of a region of offending code in an embodiment of the present application;

FIG. 6 is a schematic illustration of a first overlap determination provided in an embodiment of the present application;

FIG. 7 is a schematic illustration of an item violation detection process provided by an embodiment of the present application;

FIG. 8 is a schematic structural view of an embodiment of a detection device for offensive items provided in an embodiment of the present application;

Fig. 9 is a schematic structural diagram of an embodiment of an electronic device provided in an embodiment of the present application.

Detailed Description

The following description of the embodiments of the present application will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present application, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to fall within the scope of the application.

In describing embodiments of the present application, it should be understood that the terms "first," "second," and "second" are used for descriptive purposes only and are not to be construed as indicating or implying a relative importance or number of features indicated. Thus, a feature defining "a first" or "a second" may explicitly or implicitly include one or more of the described features. In the description of the embodiments of the present application, the meaning of "plurality" is two or more, unless explicitly defined otherwise.

The following description is presented to enable any person skilled in the art to make and use the application. In the following description, details are set forth for purposes of explanation. It will be apparent to one of ordinary skill in the art that the present application may be practiced without these specific details. In other instances, well-known processes have not been described in detail in order to avoid unnecessarily obscuring the description of the embodiments of the application. Thus, the present application is not intended to be limited to the embodiments shown, but is to be accorded the widest scope consistent with the principles and features disclosed herein.

The embodiment of the application provides a method and a device for detecting illegal goods, electronic equipment and a computer readable storage medium. The detection device for the illegal article can be integrated in electronic equipment, and the electronic equipment can be a server or a terminal and other equipment.

The execution main body of the method for detecting the illegal article in the embodiment of the application can be the device for detecting the illegal article provided by the embodiment of the application, or different types of electronic Equipment such as a server device, a physical host or User Equipment (UE) and the like integrated with the device for detecting the illegal article, wherein the device for detecting the illegal article can be realized in a hardware or software mode, and the UE can be a terminal device such as a smart phone, a tablet computer, a notebook computer, a palm computer, a desktop computer or a personal digital assistant (Personal Digital Assistant, PDA) and the like.

The electronic device may be operated in a single operation mode, or may also be operated in a device cluster mode.

Referring to fig. 1, fig. 1 is a schematic view of a scene of a detection system for illegal storage of articles according to an embodiment of the present application. The system for detecting the illegal article may include an electronic device 100, and a detecting device for the illegal article is integrated in the electronic device 100. For example, the electronic device may detect based on an image to be identified of a region to be detected, to obtain an illegal coding prediction region in the image to be identified; detecting based on the image to be identified to obtain an interference prediction area in the image to be identified; acquiring a first overlapping degree between an illegal coding prediction area in the image to be identified and an interference prediction area in the image to be identified; and if the first overlapping degree is smaller than a first preset degree threshold value, determining that illegal goods are placed in the area to be detected.

In addition, as shown in fig. 1, the system for detecting the offending article may further include a memory 200 for storing data, such as image data, video data.

It should be noted that, the schematic view of the scenario of the detecting system for the illegal article shown in fig. 1 is only an example, and the detecting system for the illegal article and the scenario described in the embodiment of the present application are for more clearly describing the technical solution of the embodiment of the present application, and do not constitute a limitation to the technical solution provided by the embodiment of the present application, and as a person of ordinary skill in the art can know that the technical solution provided by the embodiment of the present application is equally applicable to similar technical problems as the evolution of the detecting system for the illegal article and the occurrence of new service scenarios occur.

Next, a method for detecting an offensive article provided by an embodiment of the present application will be described, where in the embodiment of the present application, an electronic device is used as an execution body, and in order to simplify and facilitate description, the execution body will be omitted in the subsequent method embodiments.

Referring to fig. 2, fig. 2 is a schematic flow chart of a method for detecting an offensive article according to an embodiment of the present application. It should be noted that although a logical order is depicted in the flowchart of FIG. 2 or other figures, in some cases the steps shown or described may be performed in a different order than shown or described herein; for example, step 201 and step 202 may be performed simultaneously, or step 201 may be performed first and then step 202 may be performed, or step 202 may be performed first and then step 201 may be performed. The method for detecting the illegal coded articles comprises the following steps 201 to 204, wherein:

201. And detecting based on the image to be identified of the area to be detected, and obtaining the illegal coding prediction area in the image to be identified.

The illegal goods can be goods in a logistics sorting scene, finished products or semi-finished products on a production line, commodities on a supermarket shelf and the like.

The to-be-detected area refers to a scene area where whether illegal goods are placed or not to be detected, for example, a field area for placing goods in logistics sorting; as another example, a shelf area in a supermarket for stacking merchandise.

The illegal coding prediction area is a pixel area which is obtained by prediction based on the image and has illegal coding objects in the image.

The illegal coding prediction area of the image to be identified is a pixel area which is obtained by predicting the image to be identified and in which illegal coding objects exist in the image to be identified.

The image to be identified is an image for identifying whether the illegal article is placed in the area to be detected. There are various ways to acquire the image to be identified, and exemplary ways include:

(1) In practical application, the electronic device may integrate a camera on hardware, and obtain a video frame or image of the area to be detected through real-time shooting by the camera, so as to be used as the image to be identified.

(2) The camera can be arranged above the area to be detected, the video frame or the image of the area to be detected can be obtained through real-time shooting of the camera above the area to be detected, and the electronic equipment is connected with the camera above the area to be detected in a network. And according to the network connection, acquiring a video frame or an image of the to-be-detected area, which is obtained by shooting by the camera above the to-be-detected area, from the camera above the to-be-detected area on line to serve as an image to be recognized.

(3) The electronic device may also read the image of the area to be detected obtained by shooting the camera (including the camera integrated by the electronic device or the camera above the area to be detected) from a relevant storage medium storing the image of the area to be detected obtained by shooting the camera, as the image to be identified.

(4) And reading a video frame or an image of the region to be detected, which is acquired in advance and stored in the electronic equipment, as an image to be identified.

The camera may take an image according to a preset shooting mode, for example, a shooting height, a shooting direction or a shooting distance may be set, and a specific shooting mode thereof may be adjusted according to the camera itself, which is not limited herein.

The image to be identified is only taken as an example, but not limited to.

In step 201, the detection is performed based on the image to be detected to obtain the illegal coding prediction area in the image to be identified in various manners, which illustratively includes:

1) By a dual-stage or multi-stage target detection algorithm, such as the dual-stage target detection algorithm, the fast RCNN algorithm. At this time, step 201 may specifically include the following steps 2011A to 2013A, where:

2011A, extracting the characteristics of the image to be identified of the area to be detected through a characteristic extraction module in a double-stage or multi-stage target detection algorithm, and obtaining the image characteristics of the image to be identified.

2012A, determining each illegal coding candidate region in the image to be identified based on the image characteristics of the image to be identified by a prediction module in a dual-stage or multi-stage target detection algorithm.

2013A, determining each illegal coding candidate area in the image to be identified from the image characteristics of the image to be identified through a prediction module in a dual-stage or multi-stage target detection algorithm, and determining the illegal coding candidate area with the probability of being the illegal coding area larger than a preset probability threshold value as the illegal coding prediction area in the image to be identified.

2) And detecting by a single-stage target detection algorithm, such as a CenterNet detection algorithm, so as to obtain the illegal coding prediction area in the image to be identified. At this time, step 201 may specifically include the following steps 2011B to 2013B, where:

2011B, extracting features of the image to be identified of the area to be detected, and obtaining image features of the image to be identified.

For better understanding, the principle functions of the single-stage object detection algorithm adopted in the embodiment of the present application are described first, as shown in fig. 3 and fig. 4, where fig. 3 is a schematic diagram of a network structure of the single-stage object detection algorithm provided in the embodiment of the present application, and fig. 4 is a schematic diagram of another network structure of the single-stage object detection algorithm provided in the embodiment of the present application. The single-stage target detection algorithm specifically can comprise a feature extraction module and a prediction module, wherein the prediction module can further comprise a first sub-prediction module and a second sub-prediction module.

The feature extraction module is used for extracting features of the image to obtain image features, and takes the image as input and the image features as output. For example, the image to be identified is input into the feature extraction module, and the feature extraction module performs feature extraction on the image to be identified to obtain and output image features of the image to be identified. There are various implementations of the network structure of the feature extraction module for extracting image features, and illustratively, the implementation includes:

(1) In some embodiments, the feature extraction module may be specifically configured by using a convolutional neural network (Convolutional Neural Network, CNN), for example, a backbone network in a central net detection algorithm may be used as the feature extraction module in the embodiment of the present application, and the backbone network in the central net detection algorithm is extracted to be used as the feature extraction module for extracting image features in the embodiment of the present application.

(2) In some embodiments, the feature extraction module may also be constructed using a Swin-transducer network. The Swin-transform network performs feature extraction in a blocking, dimension reduction and other processing modes, so that the calculated amount of feature extraction is greatly reduced, and therefore, the application adopts the Swin-transform network to construct the feature extraction module for extracting the image features, so that the data processing amount of feature extraction can be reduced, the feature extraction speed can be improved, the detection speed of illegal objects can be further improved, and the detection requirement of illegal objects based on real-time images can be met to a certain extent. Furthermore, a BiFPN (weighted bi-directional feature pyramid network, multi-scale feature pyramid) layer can be added after the Swin-transform network as a feature extraction module, so that the feature extraction capability and generalization of the feature extraction module are improved, and the detection accuracy of illegal goods is further improved.

The specific network structure of the feature extraction module is merely exemplary, and the feature extraction module may be virtually any network structure that can be used to extract image features, such as the CNN network, the Swin-Transformer network, and other network structures that can be used to extract image features, or network structures that can be used to extract image features that occur in the future.

The first sub-prediction module is used for predicting according to image features to obtain center point coordinates of surrounding frames of illegal coding areas in the images and corner point coordinates of the surrounding frames; and calculating according to the center point coordinates of the bounding box of the illegal coding region and the corner point coordinates of the bounding box to obtain the illegal coding prediction region in the image. For example, a classification regression layer in a single-stage detection algorithm may be employed as the first sub-prediction module, for example, a classification regression layer in a centrnet detection algorithm may be employed as the first sub-prediction module.

The second sub-prediction module is used for predicting according to the image characteristics to obtain the center point coordinates of the bounding box of the interference area in the image and the corner point coordinates of the bounding box; and calculating according to the center point coordinates of the bounding box of the interference area and the corner point coordinates of the bounding box to obtain the interference prediction area in the image. Similarly, a classification regression layer in a single-stage detection algorithm may also be employed as the second sub-prediction module, for example, a classification regression layer in a centrnet detection algorithm may be employed as the second sub-prediction module.

In the embodiment of the application, the illegal coding prediction area in the image to be identified is needed to be detected, and the interference prediction area in the image to be identified is needed to be detected, so that the feature extraction module can extract the image features for detecting the illegal coding prediction area in the image and for detecting the interference prediction area in the image.

Specifically, as shown in fig. 3, the feature extraction module may perform feature extraction once, which is used to detect the illegal coding prediction area in the image and detect the image features of the interference prediction area in the image. The feature extraction module takes the image as input to perform feature extraction to obtain image features, and then the image features are respectively input into a first sub-prediction module for detecting illegal coding prediction areas in the image and a second sub-prediction module for detecting interference prediction areas in the image.

Alternatively, as shown in fig. 4, the feature extraction module may further include a first sub-extraction module and a second sub-extraction module, where the first sub-extraction module takes the image as an input and outputs the image feature to be input into a first sub-prediction module for detecting the illegal coding prediction area in the image. The second sub-extraction module takes the image as input and outputs the image characteristics to be input into a second sub-prediction module for detecting the interference prediction area in the image.

Based on the difference in the structure of the feature extraction module, the manner of extracting the image features of the image to be identified in step 2011B is also different, for example:

(1) in some embodiments, the feature extraction module is of a CNN network structure, and at this time, the feature extraction of the image to be identified can be achieved by performing operations such as convolution and pooling on the image to be identified based on the feature extraction module of the CNN network structure, so as to obtain the image feature of the image to be identified.

(2) In some embodiments, the feature extraction module is a Swin-transform network, and the feature extraction module based on the Swin-transform network structure performs operations such as blocking, dimension reduction, linear transformation and the like on the image to be identified, so that feature extraction is performed on the image to be identified, and image features of the image to be identified are obtained.

2012B, predicting based on the image characteristics of the image to be identified, to obtain the central point prediction coordinates of the bounding box of the illegal coding region in the image to be identified and the angular point prediction coordinates of the bounding box.

The bounding box of the illegal coding area in the image to be identified is used for indicating the area of the article to be identified which is illegal in coding. Illustratively, the bounding box may be shaped as a rectangle or square, the particular shape of the bounding box being not limited herein.

The central point prediction coordinates refer to central point coordinates of bounding boxes of illegal coding areas in the image, which are obtained by prediction based on image features of the image.

The central point prediction coordinates of the bounding box of the illegal coding region in the image to be recognized refer to central point coordinates of the bounding box of the illegal coding region in the image to be recognized, which are obtained by prediction based on the image characteristics of the image to be recognized.

The corner prediction coordinates refer to corner coordinates of bounding boxes of illegal coding areas in images, which are obtained by prediction based on image features of the images. Specifically, when the bounding box is rectangular, the corner coordinates may be coordinates of all the corners of the bounding box, coordinates of a pair of diagonal corners of the bounding box, coordinates of two adjacent corners of the bounding box, or coordinates of one corner of the bounding box.

The corner prediction coordinates of the bounding box of the illegal coding region in the image to be recognized refer to the corner coordinates of the bounding box of the illegal coding region in the image to be recognized, which are obtained by prediction based on the image characteristics of the image to be recognized.

Referring to fig. 3, fig. 4, and fig. 5, fig. 5 is a schematic illustration of a bounding box of a illegal cargo area in an embodiment of the present application, and by way of example, a first sub-prediction module in a single-stage object detection algorithm may predict, based on image features of an image to be identified, to obtain coordinates of a center point (e.g., O point in fig. 5) of a bounding box of the illegal cargo area in the image to be identified (e.g., dashed line box in fig. 5) and coordinates of an angular point (e.g., A, B point in fig. 5) of the bounding box of the illegal cargo area in the image to be identified.

2013B, determining an illegal coding prediction area in the image to be identified based on the prediction center point coordinates and the prediction corner point coordinates.

As shown in fig. 5, for example, when the predicted center point coordinates of the bounding box of the offending area in the image to be identified are the O-point coordinates (20, 20), the predicted corner point coordinates of the bounding box of the offending area in the image to be identified are the coordinates of the M-point (15, 15), and the coordinates of the N-point (25, 25), it is possible to determine that the offending predicted area in the image to be identified is the pixel point area whose abscissa is in the [15,25] section and whose ordinate is in the [15,25] section.

Because the single-stage target detection algorithm directly completes the prediction from the characteristics to classification and regression, the two-stage or multi-stage processes such as candidate region acquisition based on the characteristics, classification and regression based on the candidate region are not needed; namely: according to the embodiment of the application, the illegal coding prediction area in the image to be identified is determined by directly predicting and based on the central point prediction coordinates of the bounding box of the illegal coding area in the image to be identified and the angular point prediction coordinates of the bounding box, and the prediction from the characteristics to classification and regression can be directly completed, so that the detection speed of the illegal coding prediction area in the image to be identified can be improved to a certain extent, and the real-time detection requirement of illegal coding objects can be met.

202. And detecting based on the image to be identified to obtain an interference prediction area in the image to be identified.

In the detection of goods placed illegally in a logistics scene, for example, the detection accuracy of the goods placed illegally is generally affected by the goods placed illegally on a goods conveyor belt, an elastic frame, a cage car and the like, so that the detection of the part of the interference area is performed in the embodiment of the application.

The interference prediction area is a pixel area which is obtained by prediction based on an image and has interference in the image.

The interference prediction area in the image to be identified is a pixel area which is obtained by predicting the image to be identified and has interference in the image to be identified.

Illustratively, there are also various ways to detect the interference prediction area in the image to be identified in step 202, for example:

(1) as shown in fig. 3, the interference prediction region in the image to be identified may be obtained by predicting based on the image features of the image to be identified obtained by performing feature extraction in step 2011B through the second sub-prediction module in the single-stage object detection algorithm of the present embodiment.

(2) As shown in fig. 4, first, feature extraction is performed on an image to be identified in a region to be detected by a second sub-extraction module in the single-stage object detection algorithm of the present embodiment, so as to obtain image features of the image to be identified. Then, through a second sub-prediction module in the single-stage target detection algorithm of the embodiment, the image features of the image to be identified, which are extracted by the second sub-extraction module, are predicted, and an interference prediction area in the image to be identified is obtained.

The second sub-extraction module performs feature extraction on the image to be identified of the area to be detected to obtain the image feature of the image to be identified, which is similar to the feature extraction on the image to be identified of the area to be detected in the above step 2011B, and specific reference may be made to the above related description, so that description is omitted herein for simplicity.

The method of predicting the second sub-prediction model based on the image features of the image to be identified to obtain the interference prediction area in the image to be identified is similar to the method of predicting the illegal coding prediction area in the image to be identified in the above steps 2011B-2013B, and specific reference may be made to the above related description, which is not repeated herein for simplifying the description.

203. And acquiring a first overlapping degree between the illegal coding prediction area in the image to be identified and the interference prediction area in the image to be identified.

The first overlapping degree refers to the overlapping degree between the illegal coding prediction area in the image to be identified and the interference prediction area in the image to be identified.

In step 203, a plurality of manners of determining the first overlapping degree may be used, which illustratively includes:

(1) And taking the sum of the area of the overlapping area of the illegal coding prediction area in the image to be identified and the interference prediction area in the image to be identified and the area of the illegal coding prediction area in the image to be identified and the area of the interference prediction area in the image to be identified as a first overlapping degree. At this time, step 203 may specifically include: acquiring the area of an overlapped area of an illegal coding prediction area in an image to be identified and an interference prediction area in the image to be identified; acquiring the sum of areas of an illegal coding prediction area in an image to be identified and an interference prediction area in the image to be identified; and obtaining the sum of the area of the overlapping area and the area as a first overlapping degree.

(2) And taking the intersection ratio between the illegal coding prediction area in the image to be identified and the interference prediction area in the image to be identified as a first overlapping degree. At this time, step 203 may specifically include: and acquiring the intersection ratio between the illegal coding prediction area in the image to be identified and the interference prediction area in the image to be identified as the first overlapping degree.

Specifically, firstly, acquiring the area of an overlapped area between an illegal coding prediction area in an image to be identified and an interference prediction area in the image to be identified, the first area of the illegal coding prediction area in the image to be identified and the second area of the interference prediction area in the image to be identified; then, the cross-over ratio between the offending code prediction region in the image to be recognized and the interference prediction region in the image to be recognized is determined as the first degree of overlap according to a cross-over ratio calculation formula shown in the following formula (1).

In the formula (1), IOU represents the intersection ratio of two areas (specifically, the intersection ratio between the illegal coding prediction area in the image to be identified and the interference prediction area in the image to be identified), C represents the area of the overlapped area between the two areas (specifically, the area of the overlapped area between the illegal coding prediction area in the image to be identified and the interference prediction area in the image to be identified), and A, B represents the areas of the two areas (specifically, the area of the illegal coding prediction area in the image to be identified and the area of the interference prediction area in the image to be identified).

For example, as shown in fig. 6, the region a in fig. 6 represents the illegal coding prediction region in the image to be recognized, the region B represents the interference prediction region in the image to be recognized, and the region C represents the illegal coding prediction region in the image to be recognizedThe overlapping area of the domain a and the interference prediction area B in the image to be identified can determine that the cross-over ratio between the illegal coding prediction area in the image to be identified and the interference prediction area in the image to be identified is:

204. and if the first overlapping degree is smaller than a first preset degree threshold value, determining that illegal goods are placed in the area to be detected.

The specific value of the first preset degree threshold may be set according to the actual service scene requirement, where the specific value of the first preset degree threshold is not limited.

Specifically, if the first overlapping degree is smaller than the first preset degree threshold, it may be determined that there is a illegal article in the image to be identified, that is, the determination result of the image to be identified may be: there are illegal goods placed in the area to be detected. If the first overlapping degree is greater than or equal to the first preset degree threshold, the illegal coding prediction area in the image to be identified can be considered as misidentification, and it can be determined that no illegal coding object exists in the image to be identified, namely, the determination result of the image to be identified can be determined as follows: no illegal items are placed in the area to be detected.

The above description is made taking, as an example, a case where there is one illegal coded prediction area and one interference area in the image to be identified, and whether there is an illegal coded article in the area to be detected based on the image to be identified.

However, since there may be multiple illegal coding prediction areas or multiple interference prediction areas in the image to be identified, it may be sequentially calculated whether the first degree of overlap between each illegal coding prediction area and each interference prediction area of the image to be identified is greater than a first preset degree threshold, and if the first degree of overlap between one illegal coding prediction and any one interference prediction area is less than the first preset degree threshold, it may be determined that an illegal coding object exists in the area to be detected.

Referring to fig. 7, for easy understanding, a specific example is described for describing the determining process of the presence of the illegal article in the area to be detected in steps 203 to 204, for example, the detection result set (denoted as P) for the image to be identified includes: in step 201, M illegal coding prediction areas in the image to be identified may be obtained, and in step 202, N interference prediction areas in the image to be identified may be obtained, where the process may specifically be as follows:

(1) Traversing M illegal coding prediction areas in the detection result set, and acquiring an ith illegal coding prediction area in the M illegal coding prediction areas; the ith illegal coding prediction area represents the current traversed illegal coding prediction area, and i is more than or equal to 1 and less than or equal to M;

(2) Calculating the overlapping degree between the ith illegal coding prediction area and each interference prediction area in the N interference prediction areas respectively to serve as a first overlapping degree;

(3) If the first overlapping degree of the ith illegal coding prediction area and the h interference prediction area in the N interference prediction areas is larger than or equal to a first preset degree threshold value, eliminating the ith illegal coding prediction area from the detection result set to obtain an updated detection result set;

(4) If the first overlapping degree of the ith illegal coding prediction area and one of the N interference prediction areas is smaller than a first preset degree threshold value, determining that an illegal coding object exists in the ith illegal coding prediction area, and reserving the ith illegal coding prediction area in a detection result set;

(5) Repeating the steps (1) - (4) until i=m, and obtaining an updated detection result set (denoted as P') of the image to be identified.

(6) If the updated detection result in the step (5) further comprises a illegal placement prediction area, it can be determined that illegal placement objects exist in the area to be detected, and it can be determined that illegal placement objects exist in the position corresponding to the illegal placement prediction area in the updated detection result set obtained in the step (5).

The steps (1) to (6) illustrate the detection mode of the illegal article with the condition that the illegal article prediction area and the interference prediction area exist in the image to be identified at the same time (specifically, when the illegal article prediction area and the interference prediction area exist in the image to be identified at the same time, if the first overlapping degree between the illegal article prediction area and the interference prediction area in the image to be identified is smaller than a first preset degree threshold value, the illegal article exists in the area to be detected based on the image to be identified, and if the first overlapping degree between the illegal article prediction area and the interference prediction area in the image to be identified is larger than or equal to the first preset degree threshold value, the illegal article does not exist in the area to be detected based on the image to be identified.

As shown in fig. 7, it can be understood that in the practical application process, the detection method of the illegal article also has the possibility of detecting only the illegal article prediction area and not detecting the interference prediction area; or only the interference prediction area is detected, and the illegal coding prediction area is not detected, namely the method for detecting the illegal coding object can further comprise the following steps: if only the illegal coding prediction area and the interference prediction area are detected to exist in the image to be identified, determining that illegal coding objects exist in the area to be detected; if the fact that only an interference prediction area and an illegal coding prediction area exist in the image to be identified are detected, determining that illegal coding objects do not exist in the area to be detected; or if the illegal coding prediction area does not exist in the image to be identified, determining that illegal coding objects do not exist in the area to be detected.

When an illegal coding prediction area exists in the image to be identified and an interference area does not exist, directly judging that illegal coding objects exist in the area to be detected; or when the illegal coding prediction area does not exist in the image to be identified, the fact that the illegal coding objects do not exist in the area to be detected is directly judged, so that whether the illegal coding objects exist in the area to be detected can be rapidly and accurately judged, and the real-time detection of the illegal coding objects is improved.

In the embodiment of the present application, there are various ways of finally determining whether there are illegal goods in the area to be detected, which includes:

mode 1: and determining whether the illegal article is placed in the area to be detected based on a discrimination result of one image. At this time, when it is determined that the illegal items exist in the area to be detected based on the image to be identified, it may be finally determined in step 204 that the illegal items exist in the area to be detected.

Mode 2: and determining whether the illegal article is placed in the area to be detected or not based on the discrimination results of the images. Therefore, when the illegal article is determined to exist in the to-be-detected area based on the image to be recognized and the image acquired in a period of time after the image acquisition time to be recognized (or the illegal article is determined to exist in the to-be-detected area based on the image exceeding the preset proportion), the illegal article is finally determined to exist in the to-be-detected area, so that the problem that erroneous judgment is easy to occur based on a single frame image is avoided, and the detection accuracy of the illegal article is improved. In this case, the step 204 may specifically include the following steps 2041 to 2043:

2041. and detecting an illegal coding prediction area in a j-th image of the area to be detected and an interference prediction area in the j-th image.

The j-th image is each image in the K images acquired after the image to be identified, and the interval duration between the acquisition time of the image to be identified and the acquisition time of the j-th image is in a preset duration range.

The illegal coding prediction area in the jth image is a pixel area which is obtained by prediction based on the jth image and has illegal coding objects in the jth image.

The interference prediction region in the jth image is a pixel region in which interference exists in the jth image, which is predicted based on the jth image.

Specifically, for each image (i.e., the ith image) in the N images acquired after the image to be identified, detection may be performed in the manner described above with reference to steps 201 and 202, so as to obtain the illegal coding prediction area in the jth image and the interference prediction area in the jth image, which are not described herein for simplicity.

2042. If the illegal coding prediction area in the jth image and the interference prediction area in the jth image are overlapped to a second degree;

the second overlapping degree refers to the overlapping degree between the illegal coding prediction area in the jth image and the interference prediction area in the jth image.

The second overlapping degree is determined in a similar manner to the first overlapping degree, and reference may be made to the description about the first overlapping degree in step 203, which is omitted herein for simplicity.

2043. And if the first overlapping degree is smaller than a first preset degree threshold value and the second overlapping degree is smaller than the first preset degree threshold value, determining that illegal goods exist in the to-be-detected area.

Specifically, on the one hand, if the first overlapping degree is smaller than the first preset degree threshold, it may be determined that there is a illegal article in the image to be identified, that is, it may be determined that the determination result of the image to be identified is: there are illegal goods placed in the area to be detected. If the first overlapping degree is greater than or equal to the first preset degree threshold, the illegal coding prediction area in the image to be identified can be considered as misidentification, and it can be determined that no illegal coding object exists in the image to be identified, namely, the determination result of the image to be identified can be determined as follows: no illegal items are placed in the area to be detected.

On the other hand, if the second overlapping degree is smaller than the first preset degree threshold, it may be determined that the illegal article exists in the j-th image, that is, the determination result of the j-th image may be determined as follows: there are illegal goods placed in the area to be detected. If the second overlapping degree is greater than or equal to the first preset degree threshold, the illegal coding prediction area in the jth image can be considered as misrecognition, and it can be determined that no illegal coding object exists in the jth image, that is, the determination result of the jth image can be determined as follows: no illegal items are placed in the area to be detected.

And then, combining the judging result of the image to be identified and the judging result of the jth image (wherein j is more than or equal to 1 and less than or equal to N), and finally determining whether illegal coded articles exist in the area to be detected. For example, if the determination result of the image to be identified and the determination result of the j-th image (i.e., the determination result of each image of the K images acquired after the image to be identified) are both: if the illegal goods are placed in the to-be-detected area, finally determining that the illegal goods are placed in the to-be-detected area; otherwise, if the judgment result of one of the images is: and if the illegal goods do not exist in the to-be-detected area, finally determining that the illegal goods do not exist in the to-be-detected area. For another example, if the image to be identified and the K images acquired after the image to be identified are larger than the preset proportion image, the determination result is: if the illegal goods are placed in the to-be-detected area, finally determining that the illegal goods are placed in the to-be-detected area; otherwise, if the image to be identified and the K images acquired after the image to be identified are smaller than or equal to the preset proportion image, the judgment result is: if the illegal goods are stored in the to-be-detected area, the fact that the illegal goods are not stored in the to-be-detected area is finally determined.

By combining the judging result of the image to be identified and the judging result of the j-th image (namely, each image in the K images acquired after the image to be identified), whether the illegal article is placed in the area to be detected is finally judged, so that the problem that misjudgment is easy to occur based on a single frame image is avoided, and the detecting accuracy of the illegal article is improved.

Further, in order to avoid repeated early warning of the same illegal coding event and unnecessary resource waste, whether to output early warning information can be determined by comparing the overlapping degree of the illegal coding regions detected before and after two times. For example, when the degree of overlapping of the two times of detected illegal coded areas is higher, the method does not output, and after step 204 detects that the illegal coded objects exist in the area to be detected, the method may further include:

determining a first recording area of the currently detected illegal goods in the area to be detected based on the illegal goods in the image to be identified; acquiring a second recording area of the illegal goods in the area to be detected, which is detected last time; and if the third overlapping degree between the first recording area and the second recording area is smaller than a second preset degree threshold, outputting early warning information of illegal goods in the area to be detected.

The first recording area is a position, detected based on the image to be identified, of the illegal article in the area to be detected.

The second recording area is the position of the illegal article in the area to be detected, which is detected based on the image to be identified last time.

The third overlapping degree refers to the overlapping degree of the first recording area and the second recording area. For example, the sum of the area of the overlapping area of the first recording area and the second recording area and the area of the first recording area and the second recording area may be taken as the third overlapping degree. Alternatively, the overlap ratio between the first recording region and the second recording region may be used as the third overlap degree.

The specific value of the second preset degree threshold may be set according to the actual service scene requirement, where the specific value of the second preset degree threshold is not limited.

The method for determining the first recording area of the illegal article in the detected area to be detected is various, and exemplary includes:

(1) and directly taking the illegal coding prediction area in the image to be identified as a first recording area of the illegal coding articles in the detected area to be detected.

(2) And directly taking the illegal coding prediction area in the j-th image as a first recording area of the illegal coding article in the detected area to be detected.

(3) And taking an intersection area between the illegal coding prediction area in the image to be identified and the illegal coding prediction area in the j-th image as a first recording area of the illegal coding object in the detected area to be detected.

(4) And taking a union region between the illegal coding prediction region in the image to be identified and the illegal coding prediction region in the j-th image as a first recording region of the illegal coding object in the detected region to be detected.

For example, step 2043 may be followed by steps A1 to A3, wherein:

a1, determining a first recording area of the illegal goods in the area to be detected, which is detected at the present time, based on the illegal goods placement prediction area in the image to be identified and the illegal goods placement prediction area in the j-th image.

For example, an intersection region between the illegal coding prediction region in the image to be identified and the illegal coding prediction region in the j-th image may be used as the first recording region of the illegal coding object in the detected area to be detected. For another example, a union region between the illegal coding prediction region in the image to be identified and the illegal coding prediction region in the j-th image can be used as the first recording region of the illegal coding object in the detected region to be detected.

A2, acquiring a second recording area of the illegal article placement in the area to be detected, which is detected last time.

For example, each time when detecting that an illegal article exists in the area to be detected, the recording area of the illegal article in the area to be detected may be determined and stored in the preset database by referring to the determination manner of the first recording area, and in step A2, the second recording area may be directly read from the preset database.

A3, outputting early warning information of illegal goods in the to-be-detected area if the third overlapping degree between the first recording area and the second recording area is smaller than a second preset degree threshold value.

Because the third overlapping degree is smaller than the second preset degree threshold value, the fact that the detected illegal coding event and the illegal coding event detected last time are the same illegal coding event can be indicated to a certain degree, and repeated early warning of the same illegal coding event can be avoided by comparing the first recorded area of the illegal coding object in the area to be detected this time with the second overlapping degree between the second recorded area of the illegal coding object in the area to be detected last time, when the third overlapping degree is smaller than the second preset degree threshold value, early warning information is output again, and accordingly waste of resources is reduced.

Further, in order to avoid that the illegal cargo event cannot be processed in time after being detected, when the illegal cargo event detected at this time and the illegal cargo event detected last time are the same illegal cargo event, if the interval duration between the illegal cargo event detected at this time and the illegal cargo event detected last time is longer (if greater than a preset time interval), early warning information can be output again. At this time, after detecting that the illegal article is present in the area to be detected in step 204, the method may further include:

determining a first recording area of the currently detected illegal goods in the area to be detected based on the illegal goods in the image to be identified; acquiring a second recording area of the illegal goods in the area to be detected, which is detected last time; and if the third overlapping degree is greater than or equal to a second preset degree threshold value, and the target time interval between the occurrence time of the illegal article in the to-be-detected area detected last time and the occurrence time of the illegal article in the to-be-detected area detected this time is greater than a preset time interval, outputting early warning information of the illegal article in the to-be-detected area.

The target time interval is the interval duration between the occurrence time of the illegal article in the to-be-detected area detected last time and the occurrence time of the illegal article in the to-be-detected area detected this time.

The preset time interval may be 1 hour, 1 day, or one week, and the specific value of the preset time interval may be set according to the actual service scene requirement, where the specific value of the preset time interval is not limited.

For example, step 2043 may be followed by steps B1 to B4, wherein:

b1, determining a first recording area of the illegal goods in the area to be detected, which is detected at the present time, based on the illegal goods placement prediction area in the image to be identified and the illegal goods placement prediction area in the j-th image.

And B2, acquiring a second recording area of the illegal article placement in the area to be detected last time.

And B3, if the third overlapping degree between the first recording area and the second recording area is larger than or equal to a second preset degree threshold, detecting a target time interval between the occurrence time of the illegal article in the area to be detected, which is detected last time, and the occurrence time of the illegal article in the area to be detected, which is detected this time.

And B4, outputting early warning information of illegal goods in the area to be detected if the target time interval is larger than a preset time interval.

Outputting early warning information of illegal goods in the area to be detected if the target time interval is larger than the preset time interval; if the target time interval is smaller than or equal to the preset time interval, the early warning information of the illegal article in the to-be-detected area is not output, so that resource waste caused by repeated early warning of the same illegal article in a short time is avoided.

From the above, it can be seen that, in the embodiment of the present application, whether an illegal coded object exists in the to-be-detected area is determined by determining whether the first overlapping degree between the illegal coded prediction area in the to-be-identified image and the interference prediction area in the to-be-identified image is smaller than the first preset degree threshold; when the first overlapping degree is smaller than a first preset degree threshold value, determining that illegal goods are placed in the area to be detected; when the first overlapping degree between the illegal coding prediction area in the image to be identified and the interference prediction area in the image to be identified is smaller, the influence of the illegal coding prediction area in the image to be identified on the interference area is proved to be smaller, namely the detection accuracy of the illegal coding prediction area in the image to be identified is higher, so that the detection accuracy of illegal coding objects can be improved.

In order to better implement the method for detecting the illegal objects in the embodiment of the present application, on the basis of the method for detecting the illegal objects, the embodiment of the present application further provides a device for detecting the illegal objects, as shown in fig. 8, which is a schematic structural diagram of an embodiment of the device for detecting the illegal objects in the embodiment of the present application, where the device 800 for detecting the illegal objects includes:

the detection unit 801 is configured to detect based on an image to be identified of a region to be detected, to obtain a prediction region of illegal coding in the image to be identified;

the detection unit 801 is further configured to detect, based on the image to be identified, to obtain an interference prediction area in the image to be identified;

an obtaining unit 802, configured to obtain a first degree of overlap between a violation encoding prediction area in the image to be identified and an interference prediction area in the image to be identified;

and the determining unit 803 is configured to determine that an illegal article is present in the area to be detected if the first overlapping degree is less than a first preset degree threshold.

In some embodiments of the present application, the determining unit 803 is specifically configured to:

In some embodiments of the present application, the detecting device 800 for illegal items further includes an early warning unit (not shown in the figure), and the early warning unit is specifically configured to:

In some embodiments of the present application, the detection unit 801 is specifically configured to:

In some embodiments of the present application, the obtaining unit 802 is specifically configured to:

In the implementation, each unit may be implemented as an independent entity, or may be implemented as the same entity or several entities in any combination, and the implementation of each unit may be referred to the foregoing method embodiment, which is not described herein again.

Since the detecting device for the illegal article can execute the steps in the detecting method for the illegal article according to any embodiment of fig. 1 to 7, the beneficial effects of the detecting method for the illegal article according to any embodiment of fig. 1 to 7 can be achieved, and detailed descriptions are omitted herein.

In addition, in order to better implement the method for detecting the illegal article in the embodiment of the present application, on the basis of the method for detecting the illegal article, the embodiment of the present application further provides an electronic device, referring to fig. 9, fig. 9 shows a schematic structural diagram of the electronic device in the embodiment of the present application, and specifically, the electronic device provided in the embodiment of the present application includes a processor 901, where the processor 901 is configured to implement steps of the method for detecting the illegal article in any embodiment, as shown in fig. 1 to 7, when executing a computer program stored in the memory 902; alternatively, the processor 901 is configured to implement the functions of each unit in the corresponding embodiment as shown in fig. 8 when executing the computer program stored in the memory 902.

By way of example, a computer program may be partitioned into one or more modules/units that are stored in the memory 902 and executed by the processor 901 to accomplish an embodiment of the present application. One or more of the modules/units may be a series of computer program instruction segments capable of performing particular functions to describe the execution of the computer program in a computer device.

Electronic devices may include, but are not limited to, a processor 901, a memory 902. It will be appreciated by those skilled in the art that the illustrations are merely examples of electronic devices, and are not limiting of electronic devices, and may include more or fewer components than shown, or may combine some components, or different components, e.g., electronic devices may further include input and output devices, network access devices, buses, etc., through which the processor 901, memory 902, input and output devices, network access devices, etc., are connected.

The processor 901 may be a central processing unit (Central Processing Unit, CPU), but may also be other general purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), off-the-shelf programmable gate arrays (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like, which is a control center for an electronic device, with various interfaces and lines connecting various parts of the overall electronic device.

The memory 902 may be used to store computer programs and/or modules, and the processor 901 implements various functions of the computer device by running or executing the computer programs and/or modules stored in the memory 902 and invoking data stored in the memory 902. The memory 902 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program (such as a sound playing function, an image playing function, etc.) required for at least one function, and the like; the storage data area may store data (such as audio data, video data, etc.) created according to the use of the electronic device, and the like. In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as a hard disk, memory, plug-in hard disk, smart Media Card (SMC), secure Digital (SD) Card, flash Card (Flash Card), at least one disk storage device, flash memory device, or other volatile solid-state storage device.

It will be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working process of the above-described detecting device for illegal items, electronic device and corresponding units thereof may refer to the description of the detecting method for illegal items in any embodiment corresponding to fig. 1 to 7, and details are not repeated herein.

Those of ordinary skill in the art will appreciate that all or a portion of the steps of the various methods of the above embodiments may be performed by instructions, or by instructions controlling associated hardware, which may be stored in a computer-readable storage medium and loaded and executed by a processor.

To this end, an embodiment of the present application provides a computer readable storage medium, in which a plurality of instructions capable of being loaded by a processor are stored, so as to execute steps in a method for detecting a illegal object in any embodiment, where the steps correspond to fig. 1 to fig. 7, and specific operations may refer to descriptions of the method for detecting a illegal object in any embodiment, where the descriptions are omitted herein.

Wherein the computer-readable storage medium may comprise: read Only Memory (ROM), random access Memory (RAM, random Access Memory), magnetic or optical disk, and the like.

Since the instructions stored in the computer readable storage medium may execute the steps in the method for detecting the illegal items according to any embodiment of the present application as shown in fig. 1 to 7, the beneficial effects that can be achieved by the method for detecting the illegal items according to any embodiment of the present application as shown in fig. 1 to 7 can be achieved, which are detailed in the foregoing description and are not repeated herein.

The above description is made in detail of a method, a device, an electronic device and a computer readable storage medium for detecting illegal coded articles provided by the embodiments of the present application, and specific examples are applied to illustrate the principles and implementations of the present application, and the above description of the embodiments is only for helping to understand the method and core ideas of the present application; meanwhile, as those skilled in the art will have variations in the specific embodiments and application scope in light of the ideas of the present application, the present description should not be construed as limiting the present application.

Claims

1. A method of detecting an offensive article, the method comprising:

2. The method for detecting a specific article according to claim 1, wherein if the first overlapping degree is smaller than a first preset degree threshold, determining that a specific article exists in the to-be-detected area includes:

3. The method of detecting a offending article according to claim 2, further comprising:

4. A method of detecting a offending article according to claim 3, further comprising:

5. The method for detecting the illegal coded object according to claim 1, wherein the detecting based on the image to be identified of the area to be detected, to obtain the illegal coded prediction area in the image to be identified, includes:

6. The method of detecting a offending article according to claim 1, further comprising:

7. The method of any one of claims 1-6, wherein the obtaining a first degree of overlap between the offending coded prediction region in the image to be identified and the interfering prediction region in the image to be identified comprises:

8. A detection device for illegal goods, characterized in that the detection device for illegal goods comprises:

9. An electronic device comprising a processor and a memory, the memory having stored therein a computer program, the processor, when calling the computer program in the memory, performing the method of detecting an offending article according to any of claims 1 to 7.

10. A computer-readable storage medium, having stored thereon a computer program, the computer program being loaded by a processor to perform the steps in the method of detecting a offending article as claimed in any of claims 1 to 7.