CN113642553B - Method for accurately positioning unconstrained license plate by combining whole and part target detection - Google Patents
Method for accurately positioning unconstrained license plate by combining whole and part target detection Download PDFInfo
- Publication number
- CN113642553B CN113642553B CN202110827446.1A CN202110827446A CN113642553B CN 113642553 B CN113642553 B CN 113642553B CN 202110827446 A CN202110827446 A CN 202110827446A CN 113642553 B CN113642553 B CN 113642553B
- Authority
- CN
- China
- Prior art keywords
- license plate
- vertex
- region
- areas
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000001514 detection method Methods 0.000 title claims abstract description 112
- 238000000034 method Methods 0.000 title claims abstract description 65
- 238000004422 calculation algorithm Methods 0.000 claims abstract description 24
- 230000004927 fusion Effects 0.000 claims abstract description 16
- 238000012545 processing Methods 0.000 claims description 10
- 238000012549 training Methods 0.000 claims description 10
- 230000008569 process Effects 0.000 claims description 5
- 230000001629 suppression Effects 0.000 claims description 5
- 238000013461 design Methods 0.000 abstract description 10
- 238000002474 experimental method Methods 0.000 abstract description 2
- 230000000694 effects Effects 0.000 description 30
- 238000010586 diagram Methods 0.000 description 13
- 238000013527 convolutional neural network Methods 0.000 description 8
- 238000012805 post-processing Methods 0.000 description 6
- 238000004364 calculation method Methods 0.000 description 5
- 238000000605 extraction Methods 0.000 description 4
- 238000013135 deep learning Methods 0.000 description 3
- 238000012360 testing method Methods 0.000 description 3
- 238000004458 analytical method Methods 0.000 description 2
- 230000008901 benefit Effects 0.000 description 2
- 238000012937 correction Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- IGRCWJPBLWGNPX-UHFFFAOYSA-N 3-(2-chlorophenyl)-n-(4-chlorophenyl)-n,5-dimethyl-1,2-oxazole-4-carboxamide Chemical compound C=1C=C(Cl)C=CC=1N(C)C(=O)C1=C(C)ON=C1C1=CC=CC=C1Cl IGRCWJPBLWGNPX-UHFFFAOYSA-N 0.000 description 1
- 235000008708 Morus alba Nutrition 0.000 description 1
- 240000000249 Morus alba Species 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000007796 conventional method Methods 0.000 description 1
- 238000003745 diagnosis Methods 0.000 description 1
- 230000004807 localization Effects 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000011176 pooling Methods 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000010845 search algorithm Methods 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
- 238000012706 support-vector machine Methods 0.000 description 1
- 238000011426 transformation method Methods 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
Landscapes
- Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Image Analysis (AREA)
Abstract
A method for accurately positioning the unconstrained license plate includes such steps as synchronously detecting the 4-class left-upper, right-lower and left-lower vertex target areas and license plate areas centered on the vertex of license plate by YOLOv algorithm, indirectly predicting the position of the vertex of license plate by positioning it, and obtaining accurate license plate area by combining the non-maximal value suppressing algorithm CF_NMS of neglecting class, classifying vertex areas and single-missing vertex prediction. Finally, in order to further improve the performance of the license plate positioning algorithm combined with the target detection of the components, strategies such as multi-model fusion, coarse positioning and fine positioning are integrated into the positioning algorithm, and the effectiveness of the strategies is verified through experiments. The license plate positioning algorithm combining the whole design and the component target detection realizes the detection of the license plate vertex target on the basis of not changing YOLOv model structure and not increasing extra calculated amount, and realizes the accurate positioning of the license plate.
Description
Technical Field
The invention relates to the technical field of image processing, in particular to an unconstrained license plate accurate positioning method combining whole and part target detection.
Background
In recent years, license plate recognition systems in fixed scenes such as high-speed toll booths, parking lots, community entrances and exits have been implemented for mature commercial applications, but the effect on complex license plate images photographed by various mobile devices such as mobile phones has yet to be improved. The accurate license plate detection method is a precondition of correctly identifying the license plate, and the existing license plate detection method adopts a rectangular frame to mark the license plate, as shown in fig. 1 (a), a large amount of background redundant areas are often introduced, and difficulty is caused to the identification of the license plate. As shown in fig. 1 (b), the accurate selection of license plates in non-ideal environments can effectively improve the accuracy of subsequent license plate number recognition, and is the most critical step in the problem of license plate recognition in non-ideal environments. The existing license plate detection methods are mainly divided into two types, one type is a traditional method based on image processing, and the other type is a method based on deep learning.
Conventional methods based on image processing typically utilize manually set image edge, color, texture, etc. information to detect license plates. The method for positioning and correcting distorted license plates is characterized by comprising the following steps of (1) but bin, mei Wenhao, wu Shiqian and the like, researching the distortion license plate positioning and correcting method, and (J) manufacturing automation, 2019,41 (3): a Hough transformation method recorded by 7-11, finding four sides of a license plate by detecting straight lines in an image, and obtaining license plate vertexes by utilizing intersection points of four sides to realize accurate positioning of the license plate. However, the detection effect is poor under the conditions of smaller pictures and blurred edges.
Document [2] Chen Hongzhao, xie Zhengguang, lu Hailun. License plate positioning method combining color and edge texture [ J ]. Computer hardware: the color characteristic method recorded in the modern electronic technology 2018,41 (21): 67-70 utilizes the characteristics of most license plates, namely blue bottom and yellow bottom, and according to the fixed color collocation of license plate background and characters, a license plate area is segmented, and then an accurate license plate area is obtained by utilizing a mathematical morphology method. However, under the condition of reflecting the license plate or under the condition of a traffic sign plate similar to the license plate in the picture, the false detection phenomenon is easy to occur.
Document [3] Zheng Guilin, wu Huangzi Mulberry, license plate positioning algorithm [ J ] based on MSER and edge projection, computer engineering and design, 2019,40 (1): 241-244 MSER-based method converts pictures into gray level images, and obtains license plate positions by connected domain analysis, but if extreme value regions in the images are too many, a large number of redundant regions appear in the positioned license plate regions.
From the current research situation, the following limitations exist based on the conventional image processing method: ① . The characteristics set by people are single, and good effects are difficult to achieve in various situations when dealing with unconstrained scenes. ② . For scenes with smaller license plates, blurring and large inclination degree, the traditional method is difficult to process. ③ . When areas such as traffic signs and billboards which are quite similar to license plates appear in scenes, the traditional method is easy to be interfered and misprimed.
In recent years, a Deep Convolutional Neural Network (DCNN) is developed at a high speed, and the strong multi-level feature extraction capability of the DCNN is widely applied to the field of target detection, and the existing license plate detection method based on deep learning can be divided into a single-stage method and a multi-stage method according to detection steps. The single-stage method is to directly predict the license plate region through a network model.
Literature [4]Tian Y,Lu X,Li W X.License plate detection and localization in complex scenes based on deep learning[A].2018Chinese Control and Decision Conference(CCDC)[A].Los Alamitos:IEEE Computer Society Press,2018:6569-6574. utilizes a sliding window strategy and a selective search algorithm to obtain license plate candidate regions, and finally uses a support vector machine to classify the candidate regions so as to determine license plate regions. However, when the rotation angle is large or the distance between the license plate and the camera is long, the positioning effect is still to be improved.
Document [5]Xu Z B,Yang W,Meng A J,et al.Towards end-to-end license plate detection and recognition:A large dataset and baseline[A].European conference on computer vision[C].Heidelberg:Springer,2018:261-277. proposes an end-to-end network RPNet for license plate detection and recognition, which accomplishes both the task of license plate positioning and recognition. In order to verify network performance, authors construct a China urban parking data set (CCPD) photographed in multiple scenes, and the effect is superior to that of some common target detection networks.
Document [6]Xie L L,Ahmad T,Jin L W,et al.A new CNN-based method for multi-directional car license plate detection[J].IEEE Transactions on Intelligent Transportation Systems,2018,19(2):507-517. proposes a multidirectional license plate detection frame based on CNN, and utilizes a rotation angle prediction mechanism to determine an accurate rotation rectangular region of a license plate.
Document [7] chinese patent application number: 202010225652.0' designs a deep neural network license plate positioning method based on image enhancement in a complex environment, establishes a full convolutional neural network as a license plate detection network, then carries out image enhancement on license plates under fuzzy images such as early morning, dusk, foggy and the like in various complex environments, and improves the license plate detection accuracy of the whole model. However, the method only has certain help to the rough positioning of the fuzzy license plate, and the problem of accurate positioning of the inclined license plate is not solved.
The multi-stage method is to locate the license plate through a plurality of network models, namely, firstly determining a license plate candidate area and then locating the license plate in the area. Document [8]He K M,Gkioxari G,Dollár P,et al.Mask R-CNN[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2020,42(2):386-397. proposes a multistage license plate detection method based on a Mask-RCNN structure, wherein image features are extracted by using a convolution module similar to GoogLeNet, and the module improves the fine granularity feature detection capability while ensuring the calculation speed; then inputting the extracted feature map into a Mask-RCNN network which does not contain a segmentation step, obtaining a license plate candidate region in the image, setting 12 groups of anchors according to the size and length-width ratio of the license plate to adapt to license plate detection, and outputting two branches of information, namely license plate and non-license plate classification and bounding box regression by using a full-connection layer; finally, the license plate and the non-license plate area are filtered, the RoI-Align layer in Mask-RCNN is used, the size of the pooling layer is set to be 8 multiplied by 7, and the false detection condition of the license plate is reduced as much as possible. The method achieves higher detection precision on a plurality of public license plate data sets, but has the condition of missing detection on license plate images with too bright or too dark light and images containing a plurality of license plates.
Document [9]Han J,Yao J,Zhao J,et al.Multi-oriented and scale-invariant license plate detection based on convolutional neural networks[J].Sensors,2019,19(5):1175-1193. proposes a multidirectional and scale-invariant license plate detection method based on CNN. The network consists of two subnetworks: the RPN sub-network is used for acquiring license plate candidate areas; (2) The detection sub-network is used for determining a positive sample candidate region and obtaining a license plate region in a regression mode, and the two sub-networks share a special diagnosis extraction network layer constructed based on Faster R-CNN. The method utilizes the parallelogram to position the license plate, and realizes the detection of the license plate with large scale span by combining the multi-output layer candidate region extraction strategy and the feature fusion strategy of the anchor frame, which is superior to the existing method in different directions and multi-scale license plate detection, but has false detection and missing detection for smaller license plate targets.
Document [10] chinese patent application number: 202010223731.8' designs a license plate positioning and identifying method oriented to unconstrained conditions, which comprises the steps of firstly carrying out data enhancement on data to be detected, then inputting the data into a trained license plate detection model YOLOv-tini to obtain the approximate position of a license plate, and finally inputting the data into a regression network to obtain vertex coordinates and simultaneously carrying out perspective correction to obtain a license plate region without redundancy. But the method has poor effect on license plates with larger inclination degree and blurred license plates.
Disclosure of Invention
The invention provides a non-constraint license plate accurate positioning method combining whole and part target detection, which realizes detection of a license plate vertex target by fully utilizing excellent performance of a YOLOv network on the basis of not changing YOLOv model structure and not increasing extra calculation amount, and accurately positions a license plate region.
The technical scheme adopted by the invention is as follows:
the method for accurately positioning the unconstrained license plate integrally combined with the target detection of the license plate synchronously detects a target area of the license plate taking the vertex of the license plate as the center, namely a license plate vertex area and an integral license plate area through a YOLOv target detection algorithm; the prediction of the license plate vertex position is indirectly realized by positioning the license plate vertex region; then combining post-processing operations such as non-maximum suppression algorithm CF_NMS of neglect category, license plate vertex region classification, single missing vertex prediction and the like to realize accurate positioning of license plate region; for license plates which cannot be positioned accurately, the external rectangular areas of the license plates are reserved directly.
An unconstrained license plate accurate positioning method combining whole and part target detection comprises the following steps:
Step 1: synchronously detecting a target area taking a license plate vertex as a center and a license plate area by utilizing YOLOv target detection algorithm models;
step 2: and processing the model detection result by using the CF_NMS, license plate vertex region classification and single missing vertex prediction, thereby obtaining a precise license plate region.
The method also comprises a fusion strategy for fusing the multi-model positioning result, and the specific steps are as follows:
Step 1): and (3) retaining target boundary frame information of license plate vertex regions detected by two YOLOv target detection models with different input sizes into a set D, wherein all target boundary frames only comprise center point, width and height and category information of rectangular frames, and treating all license plate vertex regions as the same type of targets when model output results are fused.
Step 2): and establishing an empty set B, wherein no confidence information exists at the moment, randomly taking out one license plate vertex region bounding box in the set D, putting the license plate vertex region bounding box in the set B, solving IoU of the rest vertex region bounding boxes in the set D, and deleting vertex region bounding box information of IoU >0.45 from the set D.
Step 3): and (2) repeating the step (2) until the number of the vertex areas contained in the set (D) is zero, and finally obtaining all the vertex area bounding boxes in the set (B), namely obtaining the vertex area fusion result.
And 4) fusing license plate region detection frames detected by two YOLOv target detection models with different input sizes according to the same mode, and reserving a final license plate vertex region and a license plate region fusion result.
A. B, C, D are respectively a lower right vertex, a lower left vertex, an upper left vertex and an upper right vertex of the license plate, and 4 square frames are respectively vertex areas corresponding to the 4 vertices, namely target areas of model training;
The side length of the target area is related to the size of the license plate, the side length of a square area corresponding to the upper left vertex C and the lower left vertex B is the same, the length is 2h left, the side length of an area corresponding to the upper right vertex D and the lower right vertex A is the same, and the length is 2h right, wherein: h left、hright is the difference between the heights of BC and AD respectively,
By using the mode, 4 vertex areas of the license plate are cut out, each type of vertex areas of the license plate have similarity, the areas containing the license plate are all positioned in the same right-angle direction, and the rest directions are all background information; and finally, taking the circumscribed rectangular area of the license plate as a class and taking the circumscribed rectangular area of the license plate and the vertex area of the license plate as a target area for model training.
In the step 2, the non-maximum suppression algorithm cf_nms of the neglect class comprises the following steps:
step 2.1: setting all license plate vertex region categories detected in the image to be uniform values;
step 2.2: and (3) putting all license plate vertex region bounding box information into the set B, and establishing an empty set D for storing bounding boxes to be reserved.
Step 2.3: extracting the bounding box with the highest confidence from the set B, adding the bounding box into the empty set D, deleting the bounding box information from the set B, performing IoU operation as shown in a formula (1) on the extracted bounding box and all other bounding boxes in the set B, and deleting all target bounding boxes with IoU larger than a specified threshold value 0.7 from the set B;
Step 2.4: and (2.3) repeating the step (2.3) until the set (B) is empty, and finally obtaining a set (D), namely a boundary box containing all license plate vertex regions needing to be reserved.
In the step 2, the license plate vertex region classification method comprises the following steps:
Step 2.5: and respectively placing the vertex region target and the license plate region target which are obtained after the CF_NMS processing into a set B1 and a set B2, and establishing an empty set D1.
Step 2.6: an empty set D0 is established, any license plate region is taken out from B2, information of the license plate region is stored in D0 and deleted from B2, all vertex region bounding boxes with intersections are taken out from B1, and information of the vertex region bounding boxes is stored in D0 and deleted from B1.
Step 2.7: and judging whether the number of the bounding boxes in the D0 is more than 3, namely, keeping the detected number of the license plate vertex areas to be 3, 4 or more, wherein when the number of the vertex areas is less than 3, the accurate license plate areas cannot be obtained. If the number of the vertex areas is more than or equal to 3, D0 is added into D1, otherwise, no treatment is carried out.
Step 2.8: repeating the steps 2.6 and 2.7 until B1 or B2 is empty.
Step 2.9: if B2 is empty and the number of the residual license plate vertex areas in B1 is greater than 3, the vertex area information is stored in an empty set D2, D2 is added into D1, and otherwise, no processing is performed.
In the step 2, after the license plate vertex region classifying step, the existing corresponding situations of the license plate region and the license plate vertex region are divided into 3 types: 1 license plate region and 3 vertex regions corresponding to the license plate region, 1 license plate region and 4 vertex regions and more corresponding to the license plate region, 0 license plate region and 4 vertex regions and more;
for the case of determining 4 vertex areas, 4 bounding box centers can be directly acquired;
For the situation that the number of the vertex areas exceeds 4, because the wrong vertex areas are included, the centers of the boundary frames of 4 license plate areas can be randomly selected from the wrong vertex areas, and then the accurate license plate areas can be obtained through connection according to a certain sequence;
For the case of only 3 vertex areas, the missing vertex positions can be predicted by combining the corresponding license plate areas.
In the step 2,4 vertex coordinates of the corresponding license plate in the image are obtained, and the license plate region can be obtained by connecting the 4 vertices according to a certain sequence, and the specific method is as follows:
Firstly, finding out a vertex A at the uppermost end, then calculating included angles alpha, beta and gamma between the connecting line of the vertex and other three vertexes and the horizontal rightward line segment of the vertex, wherein the vertex with the smallest and largest included angle degrees is adjacent to the vertex A and is marked as a point B and a point D respectively, the rest vertexes are marked as a point C, and finally, the vertexes are connected according to the sequence of A-B-C-D-A, so that the license plate region can be obtained.
The invention relates to a non-constraint license plate accurate positioning method combining whole and part target detection, which has the following technical effects:
1) The current most common commercial license plate recognition system is sensitive to scene change, and the problems of missing detection, false detection, inaccurate marking and the like of the license plate exist when the license plate recognition problem in a real non-ideal environment is solved. The main reason is that the license plate is smaller and has the conditions of inclination, blurring, breakage and the like in the unconstrained environment, and the current model is difficult to realize good positioning effect on the license plate under various conditions. The invention designs a single-stage license plate positioning algorithm combining the whole and the component target detection, and can acquire a minimum circumscribed quadrilateral area to accurately position a license plate.
2) In the existing license plate positioning method, most license plate targets are often regarded as an integral target, and rectangular frames are used for positioning the targets. In order to obtain accurate positioning, a cascade regression network is generally adopted after a target detection network to further detect the vertex of the license plate. The method is applicable to the detection of a single license plate target serial mode only because more networks are cascaded, on one hand, the calculation load is increased, and on the other hand, the method is applicable to the detection of a single license plate target serial mode. According to the invention, the whole license plate target positioning and the license plate vertex detection are unified under the target detection frame, the detection of the whole license plate region and the license plate vertex region is realized, and the license plate accurate positioning under various complex conditions is realized through a subsequent fusion strategy. The license plate positioning method based on the combination of the component target detection skillfully utilizes the rectangular detection frame to detect the license plate vertex region (the component target), and realizes the detection of the license plate vertex target on the basis of not changing YOLOv model structure and not increasing extra calculation amount.
3) In the existing general target detection algorithm, YOLOv is taken as a typical end-to-end algorithm, higher detection accuracy is realized while higher detection speed is maintained, but a small amount of omission phenomenon still exists in extremely complex scenes. Aiming at the problem of few vertex missing detection in vertex detection, the invention designs a single missing vertex prediction method which can effectively solve the problem of single vertex missing.
4) In the invention, for vehicle pictures shot in non-ideal environments such as vehicles in an outdoor parking lot and moving vehicles under a high-level camera shot by handheld equipment, because of unfixed shooting directions, license plate targets in images have different inclination angles, and a target area obtained by using a general target detection algorithm contains more redundant background information. In order to realize the subsequent accurate identification of the license plate number, the invention designs a license plate positioning method which combines the whole with the target detection of the component. According to the method, a YOLOv target detection algorithm is used for synchronously detecting an area taking a license plate vertex as a center and a license plate area, so that the license plate vertex is indirectly obtained, and then the accurate positioning of the license plate area is realized by combining CF_NMS, license plate vertex area classification, single missing vertex prediction and other post-processing operations. The license plate positioning method combining the whole and the component target detection is designed, does not add extra calculated amount on the original YOLOv model, and can acquire four vertex positions of the license plate on the basis of inheriting the excellent performance of YOLOv, so that the accurate positioning of the license plate is realized.
5) The invention provides a single missing vertex prediction method for a license plate, which can effectively solve the problem of single point missing detection of a target detection network in a complex scene. The position of the fourth vertex is deduced through the detected vertex information of the 3 license plates, the calculation process is simple, and the reliability is high.
Drawings
FIG. 1 (a) is a schematic diagram of license plate coarse positioning;
FIG. 1 (b) is a schematic diagram of license plate fine positioning.
Fig. 2 is a diagram of a YOLOv network configuration.
Fig. 3 is a flowchart of license plate accurate positioning.
Fig. 4 is a graph showing the relationship between 4 vertex regions of the license plate and the actual license plate region.
FIG. 5 (a) is a top left vertex area diagram of a license plate;
FIG. 5 (b) is a lower left vertex area diagram of the license plate;
FIG. 5 (c) is a top right vertex area view of the license plate;
FIG. 5 (d) is a lower right vertex area diagram of the license plate;
Fig. 6 is a view of cf_nms processing effect.
Fig. 7 is a view showing a case where a plurality of license plates exist.
FIG. 8 is a diagram showing the effect of license plate vertex classification.
FIG. 9 (a) is a schematic diagram of license plate single missing vertex prediction (parallelogram with AC as diagonal);
FIG. 9 (b) is a schematic diagram of license plate single missing vertex prediction (parallelogram with BC as diagonal);
Fig. 9 (c) is a schematic diagram of single missing vertex prediction of license plate (parallelogram with AB as diagonal).
FIG. 10 is a schematic illustration of determining license plate area based on 4 vertices.
FIG. 11 (a) is a graph showing the detection effect at 608×608 input sizes;
fig. 11 (b) is a diagram of detection effects at 1024×1024 input sizes;
FIG. 11 (c) shows the effect after fusion.
Fig. 12 is a diagram of a license plate fine positioning effect in an unconstrained scene.
Detailed Description
A method for accurately positioning the unconstrained license plate by combining the whole with the target detection of components includes such steps as synchronously detecting the 4-class target region (the left-upper, right-lower and left-lower vertex target region) and the whole license plate region by YOLOv algorithm. And then, the license plate vertex position is indirectly predicted by positioning the license plate vertex region, and the accurate license plate region is obtained by combining a non-maximum suppression algorithm (CF_NMS) of neglected type, vertex region classification and post-processing operation of single missing vertex prediction. Finally, in order to further improve the performance of the license plate positioning algorithm combined with the target detection of the components, strategies such as multi-model fusion, coarse positioning and fine positioning are integrated into the positioning algorithm, and the effectiveness of the strategies is verified through experiments. The license plate positioning algorithm combining the whole design with the component target detection realizes accurate positioning of the license plate on the basis of not additionally increasing YOLOv calculated amount.
When an image is acquired from a real scene, the camera is far away from a license plate, the angle is large, the license plate in the image is small, and the detection effect of the existing target detection network on a small target is generally poor, so that how to accurately detect the small target is important. YOLOv3 is one of the most popular single-stage target detection networks at present, and is characterized in that the one-time prediction of the target position and the category can be realized directly by a regression mode, so that the method has great advantage in the operation speed. YOLOv3 by fusing various model optimization strategies, the advantage of high detection speed of the previous version is maintained, and the accuracy of target detection is improved. Most importantly, the method enhances the detection capability of small targets, and has wide application, and the network structure is shown in fig. 2. YOLOv3 can be divided into a main network and a prediction network, a main network module is the front 52 layers of the Darknet-53 network, and the module uses a large number of residual structures to increase the network depth, so that the extraction effect of the network on deep features is improved. The prediction network fuses the deep features with the shallow features after upsampling, thereby improving the target detection effect, and distributes 3 groups of anchor frames with different sizes for each feature map so as to adapt to the detection of targets with different sizes. The invention selects the CCPD of the large-scale Chinese urban parking data set which is photographed in the most widely used unconstrained environment at present as training data. The data set is selected by the following steps: the Base subset was randomly split into two equal parts, 10 ten thousand as training sets, and the rest of the Base subset and the rest of the CCPD subset except NP were 25 ten thousand as test sets. During training, 20% of images in a training set are randomly selected as a verification set, and the production of the data set labels is based on 4 pieces of vertex information of a license plate. Because a small amount of errors exist in the CCPD data set labels, the model training is negatively affected, and in order to improve the model effect, the labels in the CCPD are remarked, so that the detection effect of the whole license plate is improved.
The license plate accurate positioning under the unconstrained environment has the difficulty that the acquisition of license plate images can be influenced by uncertain factors such as bad weather (rain, snow, fog and the like), foreign matter shielding, random shooting angles and distances, blurring caused by camera shake and the like, so that the license plate in the images presents forms such as small target size, large inclination angle, blurring and the like, the difficulty of license plate detection is greatly increased, and the positioning effect of the existing technology on the license plate images under the non-ideal environment is still to be improved. Therefore, the invention designs a license plate positioning method combining the whole and the target detection of the component based on the idea of local area detection with the vertex as the center. The invention comprises the following steps: firstly, a YOLOv target detection algorithm is utilized to synchronously detect a license plate vertex region and an overall license plate region, and then, a CF_NMS (no category non-maximum suppression), license plate vertex classification and single missing vertex prediction post-processing process is combined to obtain an accurate license plate region. For license plates which cannot be positioned accurately, the external rectangular areas of the license plates can be reserved directly. In addition, in order to further improve the effect of accurate positioning of license plates, the multi-model positioning result fusion strategy is fused into target detection, and a specific flow is shown in fig. 3.
Because the license plate in the unconstrained environment is relatively smaller in the area occupied by the image, if the size of the selected license plate vertex area is too small, the feature is not obvious, and the positioning of the target detection network is not facilitated; the vertex region is excessively large in size, and the target region features are not uniform, so that network learning is not facilitated, and the model cannot be converged to influence the detection effect. In order to make the characteristics of the selected license plate vertex region more consistent. The invention provides a self-adaptive mode for setting the sizes of 4 vertex target areas of a license plate. Fig. 4 is a graph showing the relationship between 4 vertex areas of the license plate and the actual license plate area, A, B, C, D in fig. 4 are respectively the lower right, lower left, upper left and upper right vertices of the license plate, and 4 square frames are respectively the vertex areas corresponding to the 4 vertices, namely the target area of model training. The side length of the target area is related to the size of the license plate, the side length of the square area corresponding to the upper left vertex C and the lower left vertex B is the same, the length is 2h left, the side length of the area corresponding to the upper right vertex D and the lower right vertex A is the same, the length is 2h right, and h left、hright is the difference between the heights of BC and AD respectively. By using the method, 4 vertex areas of 4 license plates are cut out, as shown in fig. 5 (a), 5 (b), 5 (c) and 5 (d), each type of vertex areas of the license plates have similarity, the areas containing the license plates are all positioned in the same right-angle direction, and the rest directions are all background information. Finally, taking the circumscribed rectangular area of the license plate as one class and taking 5 classes as target areas for model training.
After the synchronous detection of the license plate vertex region and the whole license plate region is finished, the model detection result is processed by using CF_NMS, license plate vertex classification and single missing vertex prediction post-processing operation, so that an accurate license plate region is obtained. Cf_nms differs most from usual NMS in that it treats bounding boxes of multiple classes as the same class, and is suitable for cases where the allowable overlap area between classes is not large. The method comprises the following steps:
Step 1: and setting all license plate vertex region categories detected in the image to be uniform values.
Step 2: and (3) putting all the vertex region bounding box information into a set B, and establishing an empty set D for storing bounding boxes which need to be reserved.
Step 3: and (3) extracting the bounding box with the highest confidence from B, adding D, deleting the bounding box information from B, performing IoU operation as shown in a formula (1) on the extracted bounding box and all other bounding boxes in B, and deleting all target bounding boxes with IoU larger than a specified threshold value of 0.7 from the set B.
Step 4: and (3) repeating the step (3) until the set (B) is empty, and finally obtaining the set (D) which comprises all license plate vertex region bounding boxes which need to be reserved. The resulting effect is shown in fig. 6, where cf_nms can suppress redundant test frames, leaving only the 5 classes needed for testing.
As shown in fig. 7, when there are a plurality of license plates in fig. 7, it is necessary to classify the vertex regions to learn the correct license plate regions.
The invention designs a license plate vertex classifying method, when a certain license plate vertex region and a certain license plate region have intersection, the center point of the vertex region is a certain license plate vertex in the corresponding license plate region. The license plate vertex classifying effect is shown in fig. 8, and the yellow rectangular frame in fig. 8 marks the classifying result.
The license plate vertex region classification steps are as follows:
Step 1): and respectively placing the vertex region target and the license plate region target which are obtained after the CF_NMS processing into a set B1 and a set B2, and establishing an empty set D1.
Step 2): an empty set D0 is established, any license plate region is taken out from B2, information of the license plate region is stored in D0 and deleted from B2, all vertex region bounding boxes with intersections are taken out from B1, and information of the vertex region bounding boxes is stored in D0 and deleted from B1.
Step 3): and judging whether the number of the bounding boxes in the D0 is more than 3, namely, keeping the detected number of the license plate vertex areas to be 3, 4 or more, wherein when the number of the vertex areas is less than 3, the accurate license plate areas cannot be obtained. If the number of the vertex areas is more than or equal to 3, D0 is added into D1, otherwise, no treatment is carried out.
Step 4): repeating the step 2) and the step 3) until B1 or B2 is empty.
And 5) if B2 is empty and the number of the residual license plate vertex areas in B1 is greater than 3, storing the vertex area information in an empty set D2, adding D2 into D1, and otherwise, not performing any processing.
After all false detection vertexes are restrained, the area 1 and the area 2 in fig. 8 are two types of classification results, and it can be seen that the license plate vertex classification method in the invention can cope with the situations that multiple license plates exist and false detection vertexes exist in the graph.
After license plate vertex classification, the existing license plate region and license plate vertex region are divided into 3 types: 1 license plate region and corresponding 3 vertex regions, 1 license plate region and corresponding 4 and more vertex regions, 0 license plate region and 4 and more vertex regions. For the case of 4 vertex region determination, 4 bounding box centers may be directly acquired. For the situation that the number of license plate vertex areas exceeds 4, the number of the license plate vertex areas is less because of the fact that the number of the license plate vertex areas exceeds 4, and therefore the number of the license plate vertex areas is small, the centers of the 4 license plate area bounding boxes can be randomly selected from the number of the license plate vertex areas, and then the accurate license plate areas can be obtained through connection according to a certain sequence. For the case of only 3 vertex areas, the missing vertex positions can be predicted by combining the corresponding license plate areas. As shown in fig. 9 (a), 9 (b) and 9 (c), A, B, C are vertex positions corresponding to the three vertex regions, a black rectangular frame is a detected rectangular license plate region, and according to A, B, C, three parallelograms, that is, a parallelogram region obtained by using AC, BC and AB as diagonal lines respectively, but only using BC as diagonal lines is the parallelogram region closest to the real license plate region. The judgment basis is that the parallelogram predicted by using BC as a diagonal is the largest with the rectangular license plate region IoU, so that the correctly predicted license plate region can be reserved by setting a corresponding IoU threshold value.
After the post-processing process, 4 vertex coordinates of the corresponding license plate in the image can be obtained, and the license plate region can be obtained by connecting the 4 vertices according to a certain sequence. However, there are a variety of known ways in which 4 vertices are connected in sequence, and only in a specific order can a correct license plate detection frame be obtained. As shown in fig. 10, a schematic diagram of determining a license plate region according to 4 vertices is shown as follows: firstly, finding out a vertex A at the uppermost end, then calculating included angles alpha, beta and gamma between the connecting line of the vertex and other three vertexes and the horizontal rightward line segment of the vertex, wherein the vertex with the minimum and maximum included angle degrees is adjacent to the vertex A and is marked as a point B and a point D respectively, the rest vertexes are marked as a point C, and finally, the vertexes are connected according to the sequence of A-B-C-D-A, so that a license plate region can be obtained.
In order to further improve the detection effect of license plate vertex region targets, particularly license plate vertex region targets with large scale span, the invention integrates a multi-model fusion strategy into a license plate region and vertex region synchronous detection algorithm based on YOLOv. As shown in fig. 11 (a), 11 (b) and 11 (c), the detection result fusion specifically includes the following steps:
Step ①: and (3) preserving target boundary frame information of license plate vertex regions output by YOLOv models with two input sizes of 608 multiplied by 608 and 1024 multiplied by 1024 into a set D, wherein all target boundary frames only comprise center points, width and height and category information of rectangular frames, and treating all license plate vertex regions as the same type of targets when the models are fused.
Step ②: and establishing an empty set B, wherein no confidence information exists at the moment, randomly taking out one license plate vertex region boundary box in the D, putting the license plate vertex region boundary box in the D into the B, solving IoU of the rest vertex region boundary boxes in the D, and deleting vertex region boundary box information of IoU >0.45 from the D.
Step ③: and repeating the step ② until the number of the vertex areas contained in the set D is zero, and finally obtaining all the vertex area bounding boxes in the set B as the vertex area fusion result.
Step ④: and according to the same mode, fusing license plate region detection frames detected by two YOLOv target detection models with different input sizes, and reserving a finally obtained license plate vertex region and a license plate region fusion result. Compared with the effect of a single detection model, the fused effect is obviously improved.
Fig. 12 shows the accurate positioning effect of the license plate positioning method combined with the target detection of the component and the license plate correction effect after determining the 4 vertices, and it can be seen that: the method provided by the invention can obtain good accurate positioning effect on the license plate image obtained under non-ideal conditions.
Claims (4)
1. The method for accurately positioning the unconstrained license plate by combining the whole and the target detection of the component is characterized by comprising the following steps of:
Step 1: synchronously detecting the license plate vertex region and the whole license plate region by utilizing YOLOv target detection algorithm model;
step 2: using CF_NMS, license plate vertex region classification and single missing vertex prediction to process model detection results, thereby obtaining a precise license plate region;
in the step 2, the non-maximum suppression algorithm cf_nms of the neglect class comprises the following steps:
step 2.1: setting all license plate vertex region categories detected in the image to be uniform values;
step 2.2: putting all license plate vertex region bounding box information into the set B, and establishing an empty set D for storing bounding boxes to be reserved;
Step 2.3: extracting the bounding box with the highest confidence from the set B, adding the bounding box into the empty set D, deleting the bounding box information from the set B, performing IoU operation as shown in a formula (1) on the extracted bounding box and all other bounding boxes in the set B, and deleting all target bounding boxes with IoU larger than a specified threshold value 0.7 from the set B;
step 2.4: repeating the step 2.3 until the set B is empty, and finally obtaining a set D which comprises all license plate vertex region bounding boxes needing to be reserved;
the license plate vertex region classification method comprises the following steps:
Step 2.5: the license plate vertex region target and the license plate region target which are obtained after the CF_NMS processing are respectively put into a set B1 and a set B2, and an empty set D1 is established;
Step 2.6: establishing an empty set D0, taking out any license plate region from B2, storing the information in D0 and deleting the information from B2, taking out all vertex region bounding boxes which are intersected with the empty set D0 from B1, storing the information in D0 and deleting the information from B1;
Step 2.7: judging whether the number of the boundary frames in the D0 is more than 3, namely, keeping the detected number of the license plate vertex areas to be 3, 4 or more, wherein when the number of the vertex areas is less than 3, the accurate license plate areas cannot be obtained; if the number of the vertex areas is more than or equal to 3, D0 is added into D1, otherwise, no treatment is carried out;
Step 2.8: repeating the steps 2.6 and 2.7 until B1 or B2 is empty;
step 2.9: if B2 is empty and the number of the residual license plate vertex areas in B1 is greater than 3, storing vertex area information in an empty set D2, adding D2 into D1, otherwise, not performing any treatment;
after the license plate vertex region classifying step, the existing corresponding situations of the license plate region and the license plate vertex region are divided into 3 types: 1 license plate region and 3 vertex regions corresponding to the license plate region, 1 license plate region and 4 vertex regions and more corresponding to the license plate region, 0 license plate region and 4 vertex regions and more;
for the case of determining 4 vertex areas, directly acquiring 4 boundary frame centers;
for the situation that the number of the vertex areas exceeds 4, because the wrong vertex areas are included, the centers of 4 license plate area boundary frames are randomly selected from the wrong vertex areas, and then the accurate license plate areas can be obtained through connection according to a certain sequence;
and for the case of only 3 vertex areas, predicting the missing vertex positions by combining the corresponding license plate areas.
2. The method for accurately positioning the unconstrained license plate by combining whole and part target detection according to claim 1, which is characterized by comprising the following steps of: the method also comprises a fusion strategy for fusing the multi-model positioning result, and the specific steps are as follows:
Step 1): the method comprises the steps that target boundary frame information of license plate vertex areas detected by two YOLOv target detection models with different input sizes is reserved in a set D, all target boundary frames only comprise center points, width and height and category information of rectangular frames, and when model output results are fused, all license plate vertex areas are treated as the same type of targets;
Step 2): an empty set B is established, and because confidence information does not exist at the moment, a license plate vertex region boundary frame in the set D is randomly taken out and put into the set B, the boundary frames of the rest vertex regions in the set D are calculated by IoU, as shown in a formula (1), and vertex region boundary frame information of IoU >0.45 is deleted from the set D;
Step 3): repeating the step 2) until the number of the vertex areas contained in the set D is zero, and finally obtaining all the vertex area bounding boxes in the set B, namely, obtaining the vertex area fusion result;
and 4) fusing license plate region detection frames detected by two YOLOv target detection models with different input sizes according to the same mode, and reserving a final license plate vertex region and a license plate region fusion result.
3. The method for accurately positioning the unconstrained license plate by combining whole and part target detection according to claim 1, which is characterized by comprising the following steps of: in the step 1,4 vertex target area sizes of the license plate are set in a self-adaptive mode:
A. B, C, D are respectively a lower right vertex, a lower left vertex, an upper left vertex and an upper right vertex of the license plate, and 4 square frames are respectively license plate vertex target areas corresponding to the 4 vertices;
The side length of the license plate vertex target area is related to the size of the license plate, the side length of a square area corresponding to the upper left vertex C and the lower left vertex B is the same, the length is 2h left, the side length of an area corresponding to the upper right vertex D and the lower right vertex A is the same, and the length is 2h right, wherein: h left、hright is the difference between the heights of BC and AD respectively,
By using the mode, 4 vertex areas of 4 license plates are cut out, each type of vertex areas of the license plates have similarity, the areas containing the license plates are all positioned in the same right-angle direction, and the rest directions are all background information; and finally, taking the circumscribed rectangular area of the license plate as a class and taking the circumscribed rectangular area of the license plate and the vertex area of the license plate as a target area for model training.
4. The method for accurately positioning the unconstrained license plate by combining whole and part target detection according to claim 1, which is characterized by comprising the following steps of: obtaining 4 vertex coordinates of a corresponding license plate in an image, and connecting the 4 vertices according to a certain sequence to obtain a license plate region, wherein the specific method comprises the following steps of:
Firstly, finding out a vertex A at the uppermost end, then calculating included angles alpha, beta and gamma between the connecting line of the vertex and other three vertexes and the horizontal rightward line segment of the vertex, wherein the vertex with the smallest and largest included angle degrees is adjacent to the vertex A and is marked as a point B and a point D respectively, the rest vertexes are marked as a point C, and finally, the vertexes are connected according to the sequence of A-B-C-D-A, so that the license plate region can be obtained.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110827446.1A CN113642553B (en) | 2021-07-21 | 2021-07-21 | Method for accurately positioning unconstrained license plate by combining whole and part target detection |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110827446.1A CN113642553B (en) | 2021-07-21 | 2021-07-21 | Method for accurately positioning unconstrained license plate by combining whole and part target detection |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113642553A CN113642553A (en) | 2021-11-12 |
| CN113642553B true CN113642553B (en) | 2024-05-28 |
Family
ID=78417996
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110827446.1A Active CN113642553B (en) | 2021-07-21 | 2021-07-21 | Method for accurately positioning unconstrained license plate by combining whole and part target detection |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113642553B (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111709416A (en) * | 2020-05-15 | 2020-09-25 | 珠海亿智电子科技有限公司 | License plate positioning method, device and system and storage medium |
| CN111860509A (en) * | 2020-07-28 | 2020-10-30 | 湖北九感科技有限公司 | Coarse-to-fine two-stage non-constrained license plate region accurate extraction method |
| CN111914839A (en) * | 2020-07-28 | 2020-11-10 | 三峡大学 | Synchronous end-to-end license plate positioning and identifying method based on YOLOv3 |
-
2021
- 2021-07-21 CN CN202110827446.1A patent/CN113642553B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111709416A (en) * | 2020-05-15 | 2020-09-25 | 珠海亿智电子科技有限公司 | License plate positioning method, device and system and storage medium |
| CN111860509A (en) * | 2020-07-28 | 2020-10-30 | 湖北九感科技有限公司 | Coarse-to-fine two-stage non-constrained license plate region accurate extraction method |
| CN111914839A (en) * | 2020-07-28 | 2020-11-10 | 三峡大学 | Synchronous end-to-end license plate positioning and identifying method based on YOLOv3 |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113642553A (en) | 2021-11-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108304798B (en) | Street level order event video detection method based on deep learning and motion consistency | |
| CN107506763B (en) | Multi-scale license plate accurate positioning method based on convolutional neural network | |
| CN101334836B (en) | License plate positioning method incorporating color, size and texture characteristic | |
| CN109271991B (en) | License plate detection method based on deep learning | |
| CN106599792B (en) | Method for detecting hand driving violation behavior | |
| CN102509098B (en) | Fisheye image vehicle identification method | |
| CN106683119B (en) | Moving vehicle detection method based on aerial video image | |
| CN109859226B (en) | Detection method of checkerboard corner sub-pixels for graph segmentation | |
| CN104978567B (en) | Vehicle checking method based on scene classification | |
| CN109255350B (en) | New energy license plate detection method based on video monitoring | |
| CN110889398B (en) | Multi-modal image visibility detection method based on similarity network | |
| CN109726717A (en) | A kind of vehicle comprehensive information detection system | |
| CN108509950B (en) | Railway contact net support number plate detection and identification method based on probability feature weighted fusion | |
| Zhang et al. | License plate localization in unconstrained scenes using a two-stage CNN-RNN | |
| CN113052170B (en) | Small target license plate recognition method under unconstrained scene | |
| CN112215190A (en) | Illegal building detection method based on YOLOV4 model | |
| CN111695373A (en) | Zebra crossing positioning method, system, medium and device | |
| US20230394829A1 (en) | Methods, systems, and computer-readable storage mediums for detecting a state of a signal light | |
| CN113159043A (en) | Feature point matching method and system based on semantic information | |
| CN111860509A (en) | Coarse-to-fine two-stage non-constrained license plate region accurate extraction method | |
| CN113435452A (en) | Electrical equipment nameplate text detection method based on improved CTPN algorithm | |
| CN111723805B (en) | Method and related device for identifying foreground region of signal lamp | |
| CN110516527B (en) | Visual SLAM loop detection improvement method based on instance segmentation | |
| CN113158954B (en) | Automatic detection method for zebra crossing region based on AI technology in traffic offsite | |
| CN117593548A (en) | Visual SLAM method for removing dynamic feature points based on weighted attention mechanism |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20240124 Address after: 1003, Building A, Zhiyun Industrial Park, No. 13 Huaxing Road, Tongsheng Community, Dalang Street, Longhua District, Shenzhen City, Guangdong Province, 518000 Applicant after: Shenzhen Wanzhida Enterprise Management Co.,Ltd. Country or region after: China Address before: 443002 No. 8, University Road, Xiling District, Yichang, Hubei Applicant before: CHINA THREE GORGES University Country or region before: China |
|
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20240401 Address after: 710000 No.2 apartment, block B, Xiaozhai East Road, Yanta District, Xi'an City, Shaanxi Province Applicant after: Yang Zan Country or region after: China Address before: 1003, Building A, Zhiyun Industrial Park, No. 13 Huaxing Road, Tongsheng Community, Dalang Street, Longhua District, Shenzhen City, Guangdong Province, 518000 Applicant before: Shenzhen Wanzhida Enterprise Management Co.,Ltd. Country or region before: China |
|
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20240506 Address after: No. 1729, 17th Floor, Building C, Dongfang Yixiang City, No. 256 Changle West Road, Xincheng District, Xi'an City, Shaanxi Province, 710000 Applicant after: Shaanxi Songyang Communication Technology Co.,Ltd. Country or region after: China Address before: 710000 No.2 apartment, block B, Xiaozhai East Road, Yanta District, Xi'an City, Shaanxi Province Applicant before: Yang Zan Country or region before: China |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |