CN113095301A

CN113095301A - Road occupation operation monitoring method, system and server

Info

Publication number: CN113095301A
Application number: CN202110554463.2A
Authority: CN
Inventors: 杨帆; 张凯翔; 胡建国
Original assignee: Nanjing Zhenshi Intelligent Technology Co Ltd
Current assignee: Xiaoshi Technology Jiangsu Co ltd
Priority date: 2021-05-21
Filing date: 2021-05-21
Publication date: 2021-07-09
Anticipated expiration: 2041-05-21
Also published as: CN113095301B

Abstract

The invention provides a method, a system and a server for monitoring road occupation operation, which comprises the steps of firstly defining an illegal road occupation operation area, taking 1 frame of picture at intervals of N frames aiming at a monitoring video, detecting a booth in a related area through a target detection algorithm, and extracting bbox coordinate information of a related booth target; then calculating the intersection ratio of all target rectangular frames in the current frame and the previous frame of picture and sequencing; then, acquiring a current detection target, and adding the target picture into a bottom library when the intersection ratio of the detection target is smaller than all thresholds (the first occurrence); and finally, when the intersection ratio of the detection target is greater than the threshold value and is continuously processed for M times, judging that the illegal lane occupation operation is carried out. By the monitoring of the invention, the problems of difficult management, time consumption and labor consumption of the occupied road operation can be solved. Compared with the prior art, the method for monitoring the track occupation operation has the advantages of higher accuracy, lower false detection, strong monitoring real-time performance, convenience in implementation and low deployment cost.

Description

Road occupation operation monitoring method, system and server

Technical Field

The invention relates to the technical field of image processing, in particular to a method, a system and a server for monitoring occupied road operation based on target detection and position constraint.

Background

The lane occupation operation refers to the behavior that an operator occupies public places such as urban roads, bridges and urban squares to buy and sell goods or services in a profitable manner. The behavior affects urban traffic safety, and causes urban environmental pollution, which brings inconvenience to the traveling and daily life of surrounding citizens. Due to the limited manpower resources of city management, the illegal road occupation operation is difficult to be radically controlled only by a manual patrol mode and the efficiency is low. The manual patrol is carried out, at least 3-5 workers are put into each street in the district daily to carry out catering open-air barbecue and patrol work under the road occupying operation condition, and the management mode of staring at and guarding and patrol depending on manpower is low in efficiency and quality and extremely easy to repeat the renovation effect.

In the prior art, an end-to-end-based deep learning method is tried to be adopted to detect the illegal occupation booths in public fields such as a specific road in real time, and the detection result is analyzed and fed back in real time, so that the working intensity of workers can be reduced to a great extent, and the working efficiency is improved.

In the prior art, recognition of the occupied road operation based on a clustering algorithm is proposed, for example, an optimized DBSCAN clustering algorithm is adopted to process collected urban street occupied road operation data so as to finally determine an urban street occupied road operation high-rise area. However, in the scheme, an optimized DBSCAN clustering algorithm is adopted to determine the occupied road operation area of the urban street, but the occupied road operation behavior cannot be found in time, and a city manager is informed to process the occupied road operation event in time.

In the prior art, recognition of the occupied road operation is realized through a target detection technology, but the conditions of high system false alarm rate, low accuracy rate and the like can be caused through simple target recognition, so that frequent alarm is caused, and higher false alarm rate is generated.

Prior art documents:

patent document 1: the patent application with the publication number of CN108304798A and the publication date of 2018, 7 and 20 discloses a street order event video detection method based on deep learning and motion consistency, wherein a detection system for an outdoor business event and a lane occupation management event is designed by jointly judging the event through multiple conditions in a mode of combining a target detection technology in a static video frame and a target behavior analysis technology in a dynamic video in the field of video intelligent analysis, and the detection method adopts multi-model fusion judgment combining target identification and behavior identification, so that the models are relatively complex;

patent document 2: the patent application with the publication number of CN108831158A and the publication date of 2018, 11, 16 discloses an illegal parking monitoring method, device and electronic terminal, which are used for extracting a feature vector in an image of a road-occupied business district based on an SIFT algorithm and determining whether illegal parking exists or not through similarity calculation and judgment. In the scheme, the characteristics of the occupied road operation area are determined only by using a characteristic vector extraction and matching mode, and when the scheme is applied to booth identification, compared with the illegal parking identification of an automobile on a main road, the frequent alarm and false alarm are more easily caused;

patent document 3: patent application publication No. CN111931864A, published as 11/13/2020, discloses a method and system for multiple optimization of target detectors based on vertex distance and intersection ratio, comprising: acquiring an intersection ratio IOU of an anchor and a marking frame; based on the normalized distance coefficient distance between the anchor and the four vertexes corresponding to the labeling frame, correcting the cross-over ratio IOU to obtain a corrected cross-over ratio P-IOU; redefining positive and negative samples based on the modified cross-over ratio P-IOU; and training a detector based on the redefined positive and negative sample classifications. By the method of multiple optimization of the target detector based on the vertex distance and the intersection ratio, the classification performance of the detector is optimized by optimizing a matching mechanism, and false detection is reduced.

Disclosure of Invention

The invention aims to provide a method for monitoring the operation of occupying a road based on target detection and position constraint, which comprises the following steps:

extracting the continuous images of the input video stream according to a preset period to obtain images P according to a time sequence_i，P_iRepresenting the ith frame image, wherein the value of i is a natural number which is more than or equal to 1;

detecting the ith frame image P by adopting a target detector based on central point position constraint_iBooth Q in_i，Q_iRepresents the set of all jth booths in the ith frame image, and extracts the detected booths Q_iBbox coordinate information of (a); the target detector based on the central point position constraint is a target detector trained based on correction of anchor coordinates, vertex coordinates and central point coordinates of a labeling frame;

from the ith frame picture P_iInitially, the detected booth Q is judged_iWhether the bbox coordinate information is in the range of a pre-defined illegal lane occupying operation area Z or not, and if the bbox coordinate information is in the illegal lane occupying operation area Z, entering the next step for identification processing; the identification processing procedure comprises the following steps:

in the i +1 th frame picture P_i+1Detected booth Q_i+1In the middle, based on bbox coordinate information, calculating the stall Q_i+1Each booth in (1) and booth Q_iThe cross-over ratio of each booth in (a); and judging that:

1) if the booth Q_i+1One of the booths Q_i+1,kAs an object of identification, with booth Q_iThe intersection ratio of each booth in the image processing system is less than a set first threshold value, the (i + 1) th frame image P is processed_i+1The booth Q in_i+1,kStoring the data into a target base; each booth stored in the target base library is a target;

2) if the booth Q_i+1One of the booths Q_i+1,jAs an object of identification, with booth Q_iOne of the booths Q_i,lThe intersection ratio of the identification object and the booth identified in the previous frame image exceeds a set second threshold value, in the continuous m frames of images from the i +1 th frame image, the intersection ratio of the identification object and the booth identified in the previous frame image exceeds the second threshold value, the road occupation management is judged to send early warning information, otherwise, the target is abandoned, and no early warning is carried out.

Preferably, the target detector based on the central point position constraint is a pre-trained target detector, and the training process includes:

firstly, calculating the intersection ratio IOU of an anchor and a marking frame;

then, calculating a center point distance D and a diagonal distance c based on the vertex coordinates and the center point coordinates of the anchor and the labeling frame, and correcting the cross parallel comparison IOU according to the center point distance D and the diagonal distance c to obtain a cross parallel comparison D-IOU;

redefining positive and negative samples based on the modified orthogonal sum-of-sums ratio D-IOU; and

training the detector based on the redefined positive and negative sample classification to obtain a target detector based on central point position constraint.

Preferably, the redefining positive and negative samples based on the modified orthogonal union ratio D-IOU includes:

the maximum value of the modified orthogonal sum-of-difference D-IOU corresponding to each anchor and the category of the corresponding labeling frame gt are reserved, if the maximum value of the modified orthogonal sum-of-difference D-IOU is 0, the anchor category is judged to be background, and the anchor is judged to be a negative sample;

comparing the maximum value of the modified cross-comparison D-IOU corresponding to all the anchors with a preset training threshold, keeping the category of a labeling frame gt matched with the anchors when the maximum value of the modified cross-comparison D-IOU corresponding to the anchors is larger than the preset training threshold, and judging the anchors as positive samples; and for the anchor corresponding to the modification sum smaller than or equal to the maximum value of the D-IOU, modifying the class of the labeling frame gt matched with the anchor into the background, and judging that the anchor is a negative sample.

Preferably, the image P in the (i + 1) th frame_i+1Detected booth Q_i+1In the middle, based on bbox coordinate information, calculating the stall Q_i+1Each booth in (1) and booth Q_iThe cross-over ratio for each booth in (a), comprising:

for detected booth Q_i+1In any booth Q_i+1,xRespectively calculate the booth Q_i+1,xAnd the ith frame picture P_iDetected booth Q in (1)_iThe cross-over ratio of each booth in the system is obtained, a plurality of cross-over ratio values are obtained and sorted, and the maximum value of the cross-over ratio values is taken as the Q value of the booth_i+1,xAnd comparing with the booth identified by the previous frame image.

Preferably, when determining that the road occupation management sends the early warning information, the method further comprises the step of identifying the target base library of the identified object, and the method specifically comprises the following steps:

matching the identification object determined in the step 2) with a target base library, if the matching is successful, judging that the identification object is an illegal lane occupation management, and sending early warning information; otherwise, judging that the illegal occupied road operation is newly migrated.

Preferably, the matching the identified object determined in the step 2) with the target base library includes:

extracting the characteristics of the identified object determined in the step 2), respectively matching with the characteristics of the target stored in the target base, calculating the Euclidean distance between the characteristics of the identified object determined in the step 2) and each target characteristic in the target base, and if the distance is smaller than a set threshold value, judging that the two are the same target.

The second aspect of the present invention further provides a lane occupation operation monitoring system based on target detection and location constraint, including:

one or more processors;

a memory storing instructions that are operable, when executed by the one or more processors, to cause the one or more processors to perform operations comprising processes of the aforementioned lane keep monitoring method based on target detection and location constraints.

The third aspect of the present invention further provides a server, including:

one or more processors;

a memory storing instructions that are operable, when executed by the one or more processors, to cause the one or more processors to perform operations comprising the aforementioned processes of the lane operation monitoring method based on target detection and location constraints.

Compared with the prior art, the monitoring method can solve the problems of difficult management, time consumption and labor consumption of the occupied road operation. Compared with the prior art, the method for monitoring the track occupation operation has the advantages of higher accuracy, lower false detection, strong monitoring real-time performance, convenience in implementation and low deployment cost.

It should be understood that all combinations of the foregoing concepts, as well as additional concepts described in greater detail below, may be considered as part of the presently disclosed subject matter, provided such concepts are not mutually inconsistent. In addition, all combinations of claimed subject matter are considered a part of the presently disclosed subject matter.

Drawings

Fig. 1 is a schematic flow diagram of a lane occupancy monitoring method according to an exemplary embodiment of the present invention.

FIG. 2 is a schematic diagram of modifying a cross-correlation based on a center position constraint according to a preferred embodiment of the invention.

Fig. 3-5 are exemplary illustrations of the preemption identification scenario at various time periods.

Detailed Description

In this disclosure, aspects of the present invention are described with reference to the accompanying drawings, in which a number of illustrative embodiments are shown. Embodiments of the present disclosure are not necessarily intended to include all aspects of the invention. It should be appreciated that the various concepts and embodiments described above, as well as those described in greater detail below, may be implemented in any of numerous ways, as the disclosed concepts and embodiments are not limited to any one implementation. In addition, some aspects of the present disclosure may be used alone, or in any suitable combination with other aspects of the present disclosure.

In combination with the method for monitoring the lane occupation operation based on the target detection and the position constraint shown in fig. 1, the method based on the deep learning is adopted to detect the lane occupation operation, whether a booth is located in a designated lane occupation operation area, whether the same booth is continuously occupied, and whether the target of the lane occupation operation is the same booth are analyzed and early warned through the continuous detection of the booth target in the previous frame image and the next frame image, and an accumulation mechanism is used for processing, so that the problems of false detection and frequent warning are avoided and reduced.

In an optional embodiment, the lane occupation operation monitoring method based on target detection and position constraint can pre-define an illegal lane occupation operation area, in the monitoring process, 1 frame of picture is taken for every N frames of a monitoring video, a booth in a related area is detected through a target detection algorithm, and bbox coordinate information of a related booth target is extracted; then calculating the intersection ratio of all target rectangular frames in the current frame and the previous frame of picture and sequencing; then, acquiring a current detection target, and adding the target picture into a bottom library when the intersection ratio of the detection target is smaller than all thresholds (the first occurrence); and when the intersection ratio of the detection target is greater than the threshold value and is continuously processed for M times, judging that the illegal lane occupation operation is carried out. By the monitoring of the invention, the problems of difficult management, time consumption and labor consumption of the occupied road operation can be solved. Compared with the prior art, the method for monitoring the track occupation operation has the advantages of higher accuracy, lower false detection, strong monitoring real-time performance, convenience in implementation and low deployment cost.

The method for monitoring the operation of the occupied road based on the target detection and the position constraint, which is combined with the example shown in FIG. 1, comprises the following steps:

The various steps and aspects of the embodiments described above are described in more detail below with reference to the drawings.

Frame image extraction

In an embodiment of the invention, the images of the input video stream are extracted continuously according to a preset period, for example, an extraction period of 25ms, and the images P in time series are obtained_i，P_iAnd (3) representing the ith frame image, wherein the value of i is a natural number which is more than or equal to 1, so that an image in the subsequent processing process is obtained.

It should be understood that the video stream is continuously input, and the process of image extraction according to the preset period is also continuously performed.

Booth target detection

In the embodiment of the invention, the ith frame image P is detected by adopting a target detector based on central point position constraint_iBooth Q in_i，Q_iRepresents the set of all jth booths in the ith frame image, and extracts the detected booths Q_iBbox coordinate information of.

By way of example, the target detector based on the central point position constraint is a target detector trained based on correction of anchor coordinates, vertex coordinates and central point coordinates of a labeling frame, compared with a traditional target detector, the target detector is subjected to central position constraint in the training process of the detector, so that a prediction frame and the labeling frame are overlapped as much as possible, the prediction frame with the highest degree of overlap is selected as a final prediction result through the central position constraint correction, and the accuracy of model learning and training and the accuracy of prediction are improved.

In an alternative embodiment, the target detector constrained based on the position of the center point is a pre-trained target detector, and the training process includes:

firstly, calculating the intersection ratio IOU of an anchor (anchor) and a marking box (gt);

then, calculating a central point distance D and a diagonal distance c based on the vertex coordinates and the central point coordinates of the anchor (anchor) and the labeling frame (gt), and correcting the cross-correlation IOU according to the central point distance D and the diagonal distance c to obtain a cross-correlation D-IOU;

As shown in fig. 2, the cross-over comparison IOU is corrected according to the center point distance d and the diagonal line distance c, and the specific implementation includes the following steps:

and correcting according to the following formula to obtain a corrected parallel ratio D-IOU:

；

the distance d between the center points represents the distance between the center points of the anchor frame and the marking frame, and the diagonal distance c represents the diagonal distance of the minimum closure area which can simultaneously contain the anchor frame and the marking frame;

representing a hyperparameter for controlling the weight of the latter half ratio, which is equivalent to the original IOU when its value is large;

the original IOU is represented as follows:

。

wherein, the super parameter defaults to 1, and is used for controlling the weight of the latter half ratio, and is equivalent to the original IOU when the value is large.

In fig. 2, the rectangular boxes formed by two solid line portions represent the anchor and the label box gt, respectively. The purpose is to make two frames coincide, but there is usually a little deviation and dislocation in the actual prediction, and the purpose of the center distance corrected D-IOU proposed is to select the prediction frame with the highest coincidence degree as the final prediction result.

On the basis, redefining positive and negative samples for target detector training based on the modified orthogonal union ratio D-IOU, and the redefining comprises the following steps:

Therefore, the redefined positive sample and the redefined negative sample are used as training sets, the detectors are classified and trained on the basis of the existing target detection algorithm, and the precision and the accuracy of the classified detectors are improved, so that the recognition precision is improved under the complex scene of the track occupation management of the invention, and as shown in fig. 3-5, typical representations of recognition scenes in different time periods are shown.

Booth target location determination

In this embodiment, the bbox coordinate information of the booth object identified from the image is obtained, and it can be determined whether the booth object is in the predefined illegal lane-occupying business area Z according to the bbox coordinate information of the booth object.

In an alternative embodiment, we use the cross-over ratio calculation to realize from the ith frame image P_iInitially, the detected booth is judgedQ_iWhether the bbox coordinate information is in the range of a pre-defined illegal lane occupation operating area Z or not specifically comprises the following steps:

calculating the detected booth Q_iAnd comparing the bbox coordinate information with the preset violation lane occupying operation area Z, if the value of the cross-over ratio exceeds a preset value, judging that the bbox coordinate information is positioned in the violation lane occupying operation area Z, and otherwise, judging that the bbox coordinate information is not positioned in the violation lane occupying operation area Z.

Booth target road occupation operation identification processing

In this embodiment, if it is determined that the booth target extracted from the image is in the pre-defined illegal lane-occupation management area Z, the lane-occupation management recognition process is further performed, and the i +1 th frame image P is processed_i+1Detected booth Q_i+1And the image P of the i-th frame which is the previous frame_iDetected booth Q_iThe booth in the system is compared to determine whether the booth belongs to road occupation operation or temporary parking, and the method specifically comprises the following steps:

in the i +1 th frame picture P_i+1Detected booth Q_i+1In the middle, based on bbox coordinate information, calculating the stall Q_i+1Each booth in (1) and the i-th frame image P_iDetected booth Q_iThe cross-over ratio of each booth in (a); and judging that:

Wherein, the booth Q_i+1,jAnd booth Q_i+1,kThe subscripts j and k are used for distinguishing purposes only and have no special meaning, i.e., the booth Q_i+1To a booth.

In the scene shown in fig. 3-5, there is an overlapping phenomenon of booth detection frames, so it is preferable that we calculate the intersection ratio of booths in two adjacent frames of images, and calculate one by one and take the maximum value, that is:

in the i +1 th frame picture P_i+1Detected booth Q_i+1In the middle, based on bbox coordinate information, calculating the stall Q_i+1Each booth in (1) and booth Q_iThe cross-over ratio for each booth in (a), comprising:

Wherein, the booth Q_i+1,jThe subscript x, used for distinction only, has no special meaning, i.e. booth Q_i+1To a booth.

For example, for the i +1 th frame image P of the current frame_i+1Detected booth Q_i+1A booth object A in the system, and the previous frame image P_iDetected booth Q in (1)_iThe intersection ratio of each booth is calculated, and the vertex coordinates and bbox coordinates of the detection frame of the detected booth are determined, so that the intersection ratio Q of each booth can be calculated_iThe value of the cross-over ratio of each booth is sorted, and the maximum value of the cross-over ratio is taken as the cross-over ratio of the booth target A and the booth identified by the previous frame image.

Wherein if calculated with booth Q_iThe value of the cross-over ratio of each booth in the system is less than a preset first threshold value, the booth target is determined as a new booth targetAnd adding the target, namely the target does not appear in the previous frame of image, into the target base library.

If, with the booth Q_i+1,xAnd comparing the intersection ratio of the booth identified by the previous frame image with a preset second threshold value, exceeding the set second threshold value, and in m frames of continuous images from the i +1 frame image, indicating that the booth identified by the identification object is an occupied booth target and giving an alarm, wherein the intersection ratio of the identification object and the booth identified by the previous frame image exceeds the second threshold value.

Same target recognition decision

When the occupied road management is judged to send the early warning information, the method also comprises the step of carrying out target base identification on the identification object, and specifically comprises the following steps:

In an optional example, matching the identified object determined in step 2) with the target base library includes:

Track occupation operation monitoring system based on target detection and position constraint

According to the disclosed embodiment of the present invention, there is also provided a lane occupation operation monitoring system based on target detection and location constraint, comprising:

one or more processors;

a memory storing instructions that are operable, when executed by the one or more processors, to cause the one or more processors to perform operations comprising the processes of the lane management monitoring method based on target detection and location constraints of any of the preceding embodiments.

In an alternative embodiment, the track-occupied operation monitoring system based on target detection and location constraint according to the embodiment of the present invention may be implemented based on a computer system, where the computer system has a system architecture design including a processor, a memory, and a data bus, so as to implement logic and functions such as data transceiving and data processing on this basis.

Server

According to a disclosed embodiment of the invention, a server comprises:

one or more processors;

In an alternative embodiment, the server provided in the present invention may be implemented based on a local server or a cloud server, and whether the server is deployed locally or in a cloud, the server has a system architecture formed by components such as a processor, a memory, a network transmission unit, and the like, so as to implement logic and functions such as data transceiving and data processing on this basis.

Although the present invention has been described with reference to the preferred embodiments, it is not intended to be limited thereto. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the invention. Therefore, the protection scope of the present invention should be determined by the appended claims.

Claims

1. A method for monitoring the occupied road operation is characterized by comprising the following steps:

2. The method of claim 1, wherein the target detector based on the center point position constraint is a pre-trained target detector, and the training process comprises:

3. The method of claim 2, wherein the modifying the IOU based on the intersection of the center point distance d and the diagonal distance c comprises:

representing a hyper-parameter;

the IOU is represented as follows:

；

where anchor represents an anchor and gt represents a callout box.

4. The method of claim 1, wherein the redefining positive and negative samples based on modified union ratio D-IOU comprises:

5. The method for monitoring busy management according to claim 1, wherein said determining of detected booth Q_iWhether the bbox coordinate information is in the range of a pre-defined illegal lane occupation operating area Z or not comprises the following steps:

6. The method for monitoring the busy management according to claim 1, wherein the image P in the i +1 th frame is a picture P_i+1Detected booth Q_i+1In the middle, based on bbox coordinate information, calculating the stall Q_i+1Each booth in (1) and booth Q_iThe cross-over ratio for each booth in (a), comprising:

7. The method for monitoring the busy management according to claim 1, wherein when determining that the busy management sends the warning information, the method further comprises a step of performing target base identification on the identified object, and specifically comprises the following steps:

8. The method for monitoring busy management according to claim 7, wherein said matching the identified objects determined in said step 2) with the target base includes:

9. A system for monitoring the operation of a road occupancy comprising:

one or more processors;

a memory storing instructions that are operable, when executed by the one or more processors, to cause the one or more processors to perform operations comprising the process of the duty management monitoring method of any of claims 1-8.

10. A server, comprising:

one or more processors;