CN112287941A - License plate recognition method based on automatic character region perception - Google Patents

License plate recognition method based on automatic character region perception Download PDF

Info

Publication number
CN112287941A
CN112287941A CN202011348240.2A CN202011348240A CN112287941A CN 112287941 A CN112287941 A CN 112287941A CN 202011348240 A CN202011348240 A CN 202011348240A CN 112287941 A CN112287941 A CN 112287941A
Authority
CN
China
Prior art keywords
license plate
character
thermodynamic diagram
predicted
gaussian thermodynamic
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202011348240.2A
Other languages
Chinese (zh)
Other versions
CN112287941B (en
Inventor
李岩
李斌阳
舒言
张敏艺
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
International Relations, University of
Original Assignee
International Relations, University of
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by International Relations, University of filed Critical International Relations, University of
Priority to CN202011348240.2A priority Critical patent/CN112287941B/en
Publication of CN112287941A publication Critical patent/CN112287941A/en
Application granted granted Critical
Publication of CN112287941B publication Critical patent/CN112287941B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
    • G06V30/10—Character recognition
    • G06V30/14—Image acquisition
    • G06V30/148—Segmentation of character regions
    • G06V30/153—Segmentation of character regions using recognition of characters or words
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/11—Region-based segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20112—Image segmentation details
    • G06T2207/20132—Image cropping

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • General Health & Medical Sciences (AREA)
  • General Engineering & Computer Science (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • Biomedical Technology (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Multimedia (AREA)
  • Character Discrimination (AREA)

Abstract

The invention relates to a license plate recognition method based on automatic character region perception, and solves the problems that a detection model and a recognition model need to be trained independently during the existing license plate detection and recognition, and manual parameter adjustment is needed in the character segmentation process. The method comprises the steps of pre-training an initial license plate detection and recognition model based on a first data set to obtain a pre-trained license plate detection and recognition model; the first data set comprises a first license plate image and character marking position information corresponding to the first license plate image; formally training the pre-trained license plate detection and recognition model based on a second data set to obtain a formally trained license plate detection and recognition model; the second data set comprises a second license plate image and license plate bounding box position information; and recognizing the license plate image to be recognized by utilizing the formally trained license plate detection recognition model to obtain the license plate information to be recognized. The integrated detection-recognition is realized, manual parameter adjustment is not needed, and the recognition result of the license plate image to be recognized is more accurate and stable.

Description

License plate recognition method based on automatic character region perception
Technical Field
The invention relates to the technical field of optical character recognition, in particular to a license plate recognition method based on automatic character region perception.
Background
Optical Character Recognition (OCR) consists of a text detection Recognition of two subtasks. The character detection mainly judges whether a word or a text line layer in a natural scene contains a character example through an algorithm and marks an area of a text line boundary box, and the character identification is a process of converting a character area into a symbol which can be read and marked by a computer on the basis of character detection.
The license plate detection and identification is a specific application of the character detection and identification in the natural scene. A License Plate Recognition system (VLPR) is a technology capable of detecting vehicles on a monitored road surface and automatically extracting and processing License Plate information (including chinese characters, english letters, arabic numbers, and License Plate colors) of the vehicles. License plate identification is one of important components in modern intelligent traffic systems, and is very widely applied. The method is based on technologies such as digital image processing, mode recognition and computer vision, and analyzes vehicle images or video sequences shot by a camera to obtain a unique license plate number of each vehicle, so that the recognition process is completed.
The existing license plate recognition system is generally realized based on the combination of a detection model and a recognition model, different data sets are needed to respectively train a license plate detection model and a license plate recognition model when the models are trained, the resource consumption and the calculation power are huge, meanwhile, the time of an inference recognition stage is increased, and the efficiency is low. In the existing license plate detection and recognition process, characters are generally manually segmented, the complexity is high, and the accuracy and the stability of a detection and recognition result are poor.
Disclosure of Invention
In view of the foregoing analysis, embodiments of the present invention provide a license plate recognition method based on automatic character region sensing, so as to solve the problems that in the existing license plate detection and recognition process, a detection model and a recognition model need to be trained separately, and in the character segmentation process, manual parameter adjustment is needed, which consumes resources and is low in efficiency.
On one hand, the embodiment of the invention provides a license plate recognition method based on automatic character region perception, which comprises the following steps:
pre-training the initial license plate detection and recognition model based on the first data set to obtain a pre-trained license plate detection and recognition model; the first data set comprises a first license plate image and character marking position information corresponding to the first license plate image;
formally training the pre-trained license plate detection and recognition model based on a second data set to obtain a formally trained license plate detection and recognition model; wherein the second data set includes a second license plate image and license plate bounding box location information;
and recognizing the license plate image to be recognized by using the formally trained license plate detection recognition model to obtain the license plate information to be recognized.
Further, the pre-training the initial license plate detection recognition model based on the first data set to obtain a pre-trained license plate detection recognition model includes:
inputting the first data set into an initial license plate detection and recognition model to obtain a first predicted character region Gaussian thermodynamic diagram, a first predicted character Unicom Gaussian thermodynamic diagram and predicted license plate characters of the first license plate image;
calculating an error value in the pre-training process according to the first predicted character region Gaussian thermodynamic diagram, the first standard character region Gaussian thermodynamic diagram, the first predicted character Unicom Gaussian thermodynamic diagram, the first standard character Unicom Gaussian thermodynamic diagram, the predicted license plate character of the first license plate image and the real license plate character of the first license plate image by using a loss function in the pre-training process;
and obtaining a pre-trained license plate detection and recognition model according to the error value in the pre-training process by using an error back propagation mechanism.
Further, the initial license plate detection and identification model comprises an initial shared convolutional layer module and an initial classifier module;
the inputting the first data set into an initial license plate detection recognition model to obtain a first predicted character region Gaussian thermodynamic diagram, a first predicted character Unicom Gaussian thermodynamic diagram and predicted license plate characters of the first license plate image comprises the following steps:
based on a first license plate image in a first data set, the initial shared convolutional layer module outputs a first predicted character region Gaussian thermodynamic diagram, a first predicted character Unicom Gaussian thermodynamic diagram and a first feature map;
based on the character labeling position information in the first data set and the first feature map, the initial classifier module outputs the predicted license plate characters of the first license plate image.
Further, the error value in the pre-training process comprises an error value in a pre-training detection process and an error value in a pre-training recognition process;
the calculating the error value in the pre-training process comprises:
calculating the mean square error of the pixel value of each point in the first predicted character region Gaussian thermodynamic diagram and the pixel value of the corresponding point in the first standard character region Gaussian thermodynamic diagram and the mean square error of the pixel value of each point in the first character communicated Gaussian thermodynamic diagram and the pixel value of the corresponding point in the first standard character communicated Gaussian thermodynamic diagram by using a loss function in the pre-training detection process to obtain an error value in the pre-training detection process;
and calculating the cross entropy of the probability distribution corresponding to each character in the predicted license plate characters of the first license plate image and the probability distribution of the corresponding character in the real license plate characters of the first license plate image by using a loss function in the pre-training recognition process to obtain an error value in the pre-training recognition process.
Further, the formally training the pre-trained license plate detection and recognition model based on the second data set to obtain a formally trained license plate detection and recognition model includes:
inputting the second data set into a pre-trained license plate detection recognition model to obtain a second pseudo standard character region Gaussian thermodynamic diagram, a second pseudo standard character Unicom Gaussian thermodynamic diagram, a second predicted character region Gaussian thermodynamic diagram, a second predicted character Unicom Gaussian thermodynamic diagram and predicted license plate characters of the second license plate image;
calculating an error value in the formal training process according to the second predicted character region Gaussian thermodynamic diagram, the second pseudo standard character region Gaussian thermodynamic diagram, the second predicted character Unicom Gaussian thermodynamic diagram, the second pseudo standard character Unicom Gaussian thermodynamic diagram, the predicted license plate character of the second license plate image and the real license plate character of the second license plate image by using a loss function in the formal training process;
and obtaining a license plate detection and recognition model after formal training according to the error value in the formal training process by using an error back propagation mechanism.
Further, the pre-trained license plate detection and recognition model comprises a pre-trained shared convolutional layer module and a pre-trained classifier module, the second data set is input into the pre-trained license plate detection and recognition model to obtain a second pseudo-standard character region gaussian thermodynamic diagram, a second pseudo-standard character connected gaussian thermodynamic diagram, a second predicted character region gaussian thermodynamic diagram, a second predicted character connected gaussian thermodynamic diagram and predicted license plate characters of a second license plate image, and the method comprises the following steps:
obtaining a clipped license plate image according to the license plate boundary frame position information in the second license plate image;
based on the cut license plate image, automatically segmenting characters of the second license plate image by using a pre-trained shared convolutional layer module to obtain character marking position information corresponding to the second license plate image, a second pseudo-standard character region Gaussian thermodynamic diagram and a second pseudo-standard character Unicom Gaussian thermodynamic diagram;
based on the second license plate image, the pre-trained shared convolutional layer module outputs a second predicted character region Gaussian thermodynamic diagram, a second predicted character Unicom Gaussian thermodynamic diagram and a second feature diagram;
and outputting the predicted license plate characters of the second license plate image by the pre-trained classifier module based on the character marking position information corresponding to the second license plate image and the second feature map.
Further, the error value in the formal training process includes an error value in the formal training detection process and an error value in the formal training recognition process;
the calculating the error value in the formal training process includes:
calculating the mean square error of the pixel value of each point in the Gaussian thermodynamic diagram of the second predicted character region and the pixel value of the corresponding point in the Gaussian thermodynamic diagram of the second pseudo-standard character region and the mean square error of the pixel value of each point in the Gaussian thermodynamic diagram of the second predicted character communicated with the second pseudo-standard character communicated with the Gaussian thermodynamic diagram by using a loss function in the formal training detection process to obtain an error value in the formal training detection process;
and calculating the cross entropy of the probability distribution corresponding to each character in the predicted license plate characters of the second license plate image and the probability distribution of each character of the real license plate characters of the second license plate image by using a loss function in the formal training and recognition process to obtain an error value in the formal training and recognition process.
Setting a confidence threshold, and calculating to obtain a confidence based on the number of character frames of the second license plate image after the characters are automatically segmented and the number of real license plate characters corresponding to the second license plate image by using a confidence calculation formula;
when the confidence coefficient is larger than or equal to the confidence coefficient threshold value, adopting a second pseudo-standard character region Gaussian thermodynamic diagram and a second pseudo-standard character Unicom Gaussian thermodynamic diagram generated by the pre-trained license plate detection recognition model to supervise formal training;
and when the confidence coefficient is smaller than the confidence coefficient threshold value, performing character segmentation on the second license plate image again by adopting a correction method.
Further, the recognizing the license plate image to be recognized by using the formally trained license plate detection recognition model to obtain the license plate information to be recognized includes:
inputting the license plate image to be recognized into the license plate detection recognition model after formal training;
in the detection process, the license plate detection recognition model after formal training outputs a character region Gaussian thermodynamic diagram and a character Unicom Gaussian thermodynamic diagram;
obtaining a binary segmentation image based on the character region Gaussian thermodynamic diagram and the character Unicom Gaussian thermodynamic diagram, and obtaining the bounding box information of the license plate to be recognized based on the binary segmentation image;
in the recognition process, the license plate detection recognition model after formal training outputs license plate characters to be recognized.
Further, the boundary frame information of the license plate to be recognized comprises boundary frame position information and boundary frame colors, and the recognition method further comprises the following steps:
and judging the color of the boundary frame of the license plate to be recognized based on the boundary frame color recognition function.
Compared with the prior art, the invention can realize at least one of the following beneficial effects:
1. the license plate detection and recognition are completed in the same license plate detection and recognition model, so that the integration of license plate detection and recognition is realized, and the training efficiency of the license plate detection and recognition model is improved;
2. the positions of characters in the license plate are actively segmented through the license plate detection and recognition model, and related parameters in the segmentation process do not need to be adjusted manually, so that the detection and recognition result is more accurate and stable.
In the invention, the technical schemes can be combined with each other to realize more preferable combination schemes. Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. The objectives and other advantages of the invention will be realized and attained by the structure particularly pointed out in the written description and drawings.
Drawings
The drawings are only for purposes of illustrating particular embodiments and are not to be construed as limiting the invention, wherein like reference numerals are used to designate like parts throughout.
FIG. 1 is a flowchart of a license plate recognition method based on automatic character region sensing in an embodiment of the present application
FIG. 2 is a schematic diagram of a first standard character region Gaussian thermodynamic diagram and a first standard character Unicom Gaussian thermodynamic diagram generated from a first data set;
FIG. 3 is a block diagram of an initial shared convolutional layer module and its specific parameters;
FIG. 4 is a schematic diagram illustrating pre-training and formal training processes in an embodiment of the present application;
FIG. 5 is a schematic diagram illustrating automatic character segmentation of a second license plate image in a second data set using a pre-trained license plate detection recognition model according to an embodiment of the present application;
FIG. 6 is a flow chart of quality evaluation of automatically segmented characters according to an embodiment of the present application;
fig. 7 is a flowchart illustrating that the license plate detection recognition model after formal training is used to recognize the license plate image to be recognized and acquire the license plate information to be recognized in the embodiment of the present application.
Detailed Description
The accompanying drawings, which are incorporated in and constitute a part of this application, illustrate preferred embodiments of the invention and together with the description, serve to explain the principles of the invention and not to limit the scope of the invention.
The embodiment of the invention discloses a license plate recognition method based on automatic character region perception, which comprises the following steps: pre-training the initial license plate detection and recognition model based on the first data set to obtain a pre-trained license plate detection and recognition model; the first data set comprises a first license plate image and character marking position information corresponding to the first license plate image; formally training the pre-trained license plate detection and recognition model based on a second data set to obtain a formally trained license plate detection and recognition model; the second data set comprises a second license plate image and license plate boundary frame position information; and (3) recognizing the license plate image to be recognized by using the formally trained license plate detection recognition model to obtain the license plate information to be recognized, as shown in figure 1.
The license plate detection and recognition are completed in an integral license plate detection and recognition model, so that the integration of license plate detection and recognition is realized, and the initial license plate detection and recognition model is pre-trained through the first data set, so that the pre-trained license plate detection and recognition model has the capability of automatically segmenting characters, the parameters are not required to be manually intervened in the segmentation process, the training efficiency of the license plate detection and recognition model is improved, and the detection and recognition result is more accurate and stable.
In an embodiment of the present invention, the pre-training of the initial license plate detection recognition model based on the first data set to obtain a pre-trained license plate detection recognition model includes:
inputting the first data set into an initial license plate detection and recognition model to obtain a first predicted character region Gaussian thermodynamic diagram, a first predicted character Unicom Gaussian thermodynamic diagram and predicted license plate characters of the first license plate image;
calculating an error value in the pre-training process according to a first predicted character region Gaussian thermodynamic diagram, a first standard character region Gaussian thermodynamic diagram, a first predicted character Unicom Gaussian thermodynamic diagram, a first standard character Unicom Gaussian thermodynamic diagram, predicted license plate characters of a first license plate image and real license plate characters of the first license plate image by using a loss function in the pre-training process;
and obtaining a pre-trained license plate detection and recognition model according to the error value in the pre-training process by using an error back propagation mechanism.
Referring to fig. 2, fig. 2 is a schematic diagram of generating a first standard character region gaussian thermodynamic diagram and a first standard character unicom gaussian thermodynamic diagram according to a first data set.
Specifically, based on data in the CCPD license plate data set, a VGG Image Annotation (VIA) labeling tool is used to label a bounding box of a character of each license plate Image in the partial data to obtain character labeling position information corresponding to the license plate Image in the partial data, that is, a first data set including a first license plate Image and character labeling position information corresponding to the first license plate Image is obtained, and further, the content of a real license plate character corresponding to the first license plate Image and the license plate bounding box position information are known.
Generating a series of two-dimensional standard Gaussian thermodynamic diagrams (2D Gaussian thermodynamic diagrams) with equal proportional sizes based on a first license plate image in a first data set, wherein the deeper the color of the position closer to the center, the larger the corresponding pixel value; obtaining the size of a character area frame corresponding to each character based on the character marking position information corresponding to the first license plate image, further calculating a scaling coefficient of the size of the standard Gaussian thermodynamic diagram relative to the size of the character area frame, carrying out affine transformation on the standard Gaussian thermodynamic diagram according to the scaling coefficient to adapt to the size of each character area frame, and further obtaining a first standard character area Gaussian thermodynamic diagram representing the occurrence position probability of the character; for each character region frame, connecting diagonal lines of the character region frames, obtaining four triangular region frames in each character region frame, connecting centers of an upper triangular region frame and a lower triangular region frame of an adjacent character region frame with centers of an upper triangular region frame and a lower triangular region frame of the adjacent character region frame to obtain a communicated region frame between two adjacent characters, and performing affine transformation on the standard Gaussian thermodynamic diagram according to a scaling coefficient to adapt to the size of each communicated region frame so as to obtain a first standard character communicated Gaussian thermodynamic diagram representing the connection probability of the adjacent characters, wherein the first standard character is a character in a square; the first standard character region Gaussian thermodynamic diagram and the first standard character Unicom Gaussian thermodynamic diagram are used for supervising pre-training of the initial license plate detection recognition model based on the first data set.
Referring to fig. 3, fig. 3 is a structural diagram and a specific parameter diagram of an initial shared convolutional layer module. The initial license plate detection and recognition model is constructed on the basis of an initial shared convolutional layer module and an initial classifier module. The framework network for extracting the image features by the initial shared convolutional layer module mainly uses a VGG16 convolutional neural network model, namely, a convolutional kernel with the size of 3 x 3 is used in all convolutional layers, the image is convolved by the operation with the step size of 1, and meanwhile, a pooling layer with the step size of 2 is arranged after each convolutional layer to compress the image features. Meanwhile, the shared convolution layer module comprises a SEnet attention module, in each SEnet attention module, a feature map with the size H W C is input, the SEnet attention module can calculate the weight of each channel and sum the weights to obtain a new feature map with the size H W C, the principle of the SEnet attention module is shown in the diagram at the lower right of the diagram in FIG. 3, PoOLING represents that global maximum POOLING is carried out on each channel, FC plays a full-connection role, the two-dimensional feature map is converted into a one-dimensional vector, and Relu and sigmoid are activation functions. On the basis that feature layers in different stages are obtained through six times of convolution operation, feature layers in different stages are spliced through a U-net structure, namely down sampling is performed again after 2 times of convolution after down sampling, and up sampling is performed in a deconvolution mode and is connected with the down sampling feature layers with corresponding sizes, and then deconvolution is performed again after 2 times of convolution, so that feature graphs with 16 channels and half width and height of original images are finally obtained. On one hand, the method copies one copy of the character region for the identification process, and on the other hand, the method convolutes the character region by using two convolution cores of 1 x 1 to obtain the output of 2 channels, namely predicted character region Gaussian thermodynamic diagram and predicted character Unicom Gaussian thermodynamic diagram. The size of the output feature map is related to the size of the input image and the parameters of the convolution layer, and may be set according to actual conditions, which is not limited in the present application.
In an embodiment of the present application, inputting a first data set into an initial license plate detection recognition model to obtain a first predicted character region gaussian thermodynamic diagram, a first predicted character unicom gaussian thermodynamic diagram, and predicted license plate characters of a first license plate image, includes:
based on a first license plate image in the first data set, the initial shared convolution layer module outputs a first predicted character region Gaussian thermodynamic diagram, a first predicted character Unicom Gaussian thermodynamic diagram and a first feature diagram;
based on the character labeling position information in the first data set and the first feature map, the initial classifier module outputs the predicted license plate characters of the first license plate image.
Specifically, referring to fig. 4, when performing pre-training, the input image is a first license plate image in the first data set, optionally, the first license plate image is input to the initial shared convolution layer module for pre-training after being subjected to image enhancement operations such as normalization processing, arbitrary angle rotation, horizontal inversion, arbitrary region clipping, color difference saturation conversion, and the like, the pre-training process includes a pre-training detection process and a pre-training recognition process, and the first standard character region gaussian thermodynamic diagram and the first standard character connected gaussian thermodynamic diagram obtained based on the flow in fig. 2 are used for supervising the performance of the pre-training detection process.
The processed first license plate image is input into an initial shared convolution layer module, and a first predicted character region Gaussian thermodynamic diagram, a first predicted character Unicom Gaussian thermodynamic diagram and a first feature diagram are output from the output end of the initial shared convolution layer module; the first feature map and the character marking position information corresponding to the first license plate image are input to an interested area pooling layer in the identification process together, wherein the first feature map comprises feature maps of sixteen channels, and the interested area pooling layer is mainly used for extracting an interested area with a fixed size from the first feature map according to the corresponding character marking position information. The specific method comprises the following steps: assuming that the first license plate image has a height h and a width w, when the first license plate image is input, the width of the corresponding first feature map output by the shared convolution layer is w1Height of h1The proportionality coefficient k can be obtained from formula 1; and mapping the character bounding box area embodied according to the corresponding character marking position information in the first license plate image according to the proportionality coefficient, and projecting the corresponding character bounding box area on the first characteristic diagram.
Figure BDA0002800593960000111
And outputting each character bounding box area on the first feature map as a fixed-size feature map of interest by using a maximum pooling method, and optionally outputting n feature maps of interest with the same size by using a region-of-interest pooling layer. Inputting n interesting feature maps into an initial classifier module for single character recognition, wherein the classifier module is a double-head classifier and comprises a first classifier and a second classifier, the first classifier only classifies the first characters of license plates, namely, the first interesting feature maps are input into the first classifier for recognition, the first classifier consists of a first hidden layer with 500 neurons and a first full connecting layer, the number of the categories output by the first full connecting layer is consistent with the number of provincial administrative districts in China, and the license plates of different provincial administrative districts are predicted respectively, for example: the category of the first fully-connected layer output may be 34; the second classifier is composed of a second hidden layer with 200 neurons and a second full-connection layer, the second full-connection layer outputs 36 categories and predicts capital letters and numbers respectively, namely n-1 interesting feature graphs behind the first interesting feature graph are input into the second classifier respectively for recognition, and therefore the initial classifier module outputs the predicted license plate characters of the first license plate image according to the character marking position information in the first data set and the first feature graph.
In an embodiment of the present application, the error value in the pre-training process includes an error value in the pre-training detection process and an error value in the pre-training recognition process;
calculating an error value in a pre-training process, comprising:
calculating the mean square error of the pixel value of each point in the first predicted character region Gaussian thermodynamic diagram and the pixel value of the corresponding point in the first standard character region Gaussian thermodynamic diagram and the mean square error of the pixel value of each point in the first character communicated Gaussian thermodynamic diagram and the pixel value of the corresponding point in the first standard character communicated Gaussian thermodynamic diagram by using a loss function in the pre-training detection process to obtain an error value in the pre-training detection process;
and calculating the cross entropy of the probability distribution corresponding to each character in the predicted license plate characters of the first license plate image and the probability distribution of the corresponding character in the real license plate characters of the first license plate image by using the loss function in the pre-training recognition process to obtain an error value in the pre-training recognition process.
Specifically, on the basis of obtaining a first predicted character region Gaussian thermodynamic diagram, a first predicted character Unicom Gaussian thermodynamic diagram and predicted license plate characters of a first license plate image in the pre-training process, combining a first standard character region Gaussian thermodynamic diagram, a first standard character Unicom Gaussian thermodynamic diagram and real license plate characters of the first license plate image, and obtaining an error value in the pre-training detection process and an error value in the pre-training recognition process by using a loss function in the pre-training detection process and the recognition process, so as to obtain an error value in the pre-training process.
The loss function L in the pre-training process is shown in equations 2 to 4, LdetFor the loss function in the pre-training detection process, LrecFor the loss function in the pre-training recognition process, w1And w2For hyper-parameters, optionally, w1＝w2＝1。
L＝w1Ldet+w2Lrec (2)
Figure BDA0002800593960000121
Figure BDA0002800593960000122
Loss function L in pre-training detection processdetAs shown in formula 3, p represents a certain pixel point in the gaussian thermodynamic diagram of the first predicted character region, Sr(p) pixel values, S, representing the correspondence of p points on a Gaussian thermodynamic diagram of the first predicted character regionr *(p) expressing the pixel value of the pixel point corresponding to the p point on the Gaussian thermodynamic diagram of the first standard character region, Sa(p) expressing the pixel value of the pixel point corresponding to the p point on the first prediction character Unicom Gaussian thermodynamic diagram, Sa *And (p) representing the pixel value of a pixel point corresponding to the p point on the first standard character Unicom Gaussian thermodynamic diagram.
Calculating to obtain the mean square error of the pixel value of each point in the first predicted character region Gaussian thermodynamic diagram and the pixel value of the corresponding point in the first standard character region Gaussian thermodynamic diagram, performing the same processing on all the points in the first predicted character region Gaussian thermodynamic diagram to obtain a plurality of mean square error values, and summing to obtain a first mean square error sum; calculating to obtain the mean square error of the pixel value of each point in the first predicted character-communicated Gaussian thermodynamic diagram and the pixel value of the corresponding point in the first standard character-communicated Gaussian thermodynamic diagram, carrying out the same treatment on all the points in the first predicted character-communicated Gaussian thermodynamic diagram to obtain a plurality of mean square error values, summing the mean square error values to obtain a second mean square error sum, adding the first mean square error sum and the second mean square error sum, and dividing the sum by the number of all the pixel points in the first predicted character region Gaussian thermodynamic diagram to obtain an error value in the pre-training detection process. Further, the sizes of the first predicted character region gaussian thermodynamic diagram, the first standard character region gaussian thermodynamic diagram, the first predicted character Union gaussian thermodynamic diagram and the first standard character Union gaussian thermodynamic diagram are equal, and pixel points in the diagrams completely correspond to each other and the number of the pixel points is equal.
Loss function L in pre-training recognition processrecAs shown in formula 4, taking the number of the feature maps of interest as n as an example, the loss value in the recognition process includes an error value c caused by the first classifier during pre-trainingilog(xi) And the error value caused by the second classifier
Figure BDA0002800593960000131
Wherein, the first classifier only predicts the first character of the license plate, i represents the category i in all categories corresponding to the first character, optionally, the category corresponding to the first character is consistent with the number of provincial administrative districts in China, for example, 34, at this time, i represents the category i, x in the categories 1 to 34iRepresenting the probability that the character class i is predicted, ciRepresenting the probability of the character category i in a real license plate; for the second classifier, the second classifier is used to predict capital letters and numbers, j represents a category j of all categories corresponding to any one character from n-1 characters left after the first character is removed, optionally, the number of all categories corresponding to the character is 36, and j represents a category j from category 1 to category 36, y representsjRepresenting the probability that the character class j is predicted, djRepresenting the probability that the character class j is in the real license plate.
And obtaining the cross entropy of the probability distribution of each character in the predicted license plate characters of the first license plate image corresponding to all types and the probability distribution of all types of corresponding characters in the real license plate characters of the first license plate image based on a formula 4, and performing the same processing on all characters in the predicted license plate characters of the first license plate image to obtain the sum of the cross entropies corresponding to all characters, thereby obtaining the error value in the pre-training recognition process.
In an embodiment of the present application, obtaining a pre-trained license plate detection recognition model according to an error value in a pre-training process by using an error back propagation mechanism includes:
continuously updating parameters of the initial shared convolutional layer module by using an error back propagation mechanism according to an error value in a pre-training detection process and an error value in a pre-training recognition process to obtain a pre-trained shared convolutional layer module;
and continuously updating the parameters of the initial classifier module by using an error back propagation mechanism according to the error value in the pre-training recognition process to obtain the pre-trained classifier module.
Specifically, when the initial license plate detection and recognition model is based on all the first license plate images in the first data set and all the character marking position information corresponding to the first license plate images, a multi-round detection and recognition process is completed, and after relevant parameters of the initial shared convolution layer module and the initial classifier module are correspondingly updated in multiple rounds, the pre-trained shared convolution layer module and the pre-trained classifier module are obtained, and the pre-trained license plate detection and recognition model is obtained.
In an embodiment of the present application, formally training the pre-trained license plate detection and recognition model based on a second data set to obtain a formally trained license plate detection and recognition model includes:
inputting a second data set into the pre-trained license plate detection and recognition model to obtain a second pseudo standard character region Gaussian thermodynamic diagram, a second pseudo standard character Unicom Gaussian thermodynamic diagram, a second predicted character region Gaussian thermodynamic diagram, a second predicted character Unicom Gaussian thermodynamic diagram and predicted license plate characters of a second license plate image;
calculating an error value in the formal training process according to a second predicted character region Gaussian thermodynamic diagram, a second pseudo standard character region Gaussian thermodynamic diagram, a second predicted character Unicom Gaussian thermodynamic diagram, a second pseudo standard character Unicom Gaussian thermodynamic diagram, a predicted license plate character of the second license plate image and a real license plate character of the second license plate image by using a loss function in the formal training process;
and obtaining a vehicle license plate detection and recognition model after formal training according to the error value in the formal training process by using an error back propagation mechanism.
Specifically, the initial parameters of the license plate detection and recognition model in the formal training process are the final parameters of the license plate detection and recognition model in the pre-training process, that is, the parameters of the license plate detection and recognition model after the pre-training process are the final parameters of the license plate detection and recognition model after the pre-training process, and are also the initial parameters of the license plate detection and recognition model at the beginning of the formal training process.
Specifically, the pre-trained license plate detection and recognition model comprises a pre-trained shared convolutional layer module and a pre-trained classifier module, the pre-trained shared convolutional layer module has a function of automatically segmenting characters, and the second data set comprises a second license plate image and does not comprise character marking position information corresponding to the second license plate image; further, the content of the real license plate characters corresponding to the second license plate image and the position information of the license plate boundary frame are known; optionally, specific data included in the second data set may be additionally adjusted according to actual conditions, which is not limited in this application.
In an embodiment of the present invention, inputting a second data set into a pre-trained license plate detection recognition model to obtain a second pseudo-standard character region gaussian thermodynamic diagram, a second pseudo-standard character unicom gaussian thermodynamic diagram, a second predicted character region gaussian thermodynamic diagram, a second predicted character unicom gaussian thermodynamic diagram, and a predicted license plate character of a second license plate image, includes:
obtaining a clipped license plate image according to the license plate boundary frame position information in the second license plate image;
based on the cut license plate image, automatically segmenting characters of the second license plate image by using the pre-trained shared convolutional layer module to obtain character marking position information corresponding to the second license plate image, a second pseudo-standard character region Gaussian thermodynamic diagram and a second pseudo-standard character Unicom Gaussian thermodynamic diagram;
based on the second license plate image, the pre-trained shared convolution layer module outputs a second predicted character region Gaussian thermodynamic diagram, a second predicted character Unicom Gaussian thermodynamic diagram and a second feature diagram;
and outputting the predicted license plate characters of the second license plate image by the pre-trained classifier module based on the character marking position information corresponding to the second license plate image and the second characteristic diagram.
Specifically, referring to fig. 4-5, in the formal training process, the generation process of the second pseudo-standard character region gaussian thermodynamic diagram and the second pseudo-standard character unicom gaussian thermodynamic diagram is as follows: cutting the second license plate image according to the license plate boundary frame position information in the second license plate image to obtain a cut license plate image; inputting the clipped license plate image into a pre-trained shared convolution layer module to obtain a region Gaussian thermodynamic diagram of each character in the clipped license plate image; automatically segmenting characters of the cut license plate image by adopting a watershed algorithm, framing the position of each character in the cut license plate image by adopting a minimum circumscribed rectangle frame, obtaining the region of each character in the cut license plate image, and obtaining a boundary frame of each character in the second license plate image according to size conversion so as to obtain the second license plate image after the characters are automatically segmented and corresponding character marking position information; by adopting the method in fig. 2, a second pseudo-standard character region gaussian thermodynamic diagram and a second pseudo-standard character Unicom gaussian thermodynamic diagram are obtained based on the second license plate image and the corresponding character marking position information, and the shared convolutional layer parameters are not affected in the process. In the specific process, reference is made to the generation processes of a first standard character region gaussian thermodynamic diagram and a first standard character Unicom gaussian thermodynamic diagram in the pre-training process, the principles are the same, and the detailed description is omitted here; in order to overcome the interference of a background region in a second license plate image, a second pseudo-standard character region Gaussian thermodynamic diagram and a second pseudo-standard character Unicom Gaussian thermodynamic diagram generated by cutting the license plate region are closer to a real label, so that the second pseudo-standard character region Gaussian thermodynamic diagram and the second pseudo-standard character Unicom Gaussian thermodynamic diagram are used for supervising formal training of a pre-trained license plate detection recognition model based on a second data set.
The second license plate image is input into the pre-trained shared convolution layer module, and the output end of the pre-trained shared convolution layer module outputs a second predicted character region Gaussian thermodynamic diagram, a second predicted character Unicom Gaussian thermodynamic diagram and a second feature diagram; and obtaining the predicted license plate characters of the second license plate image based on the second feature map and the character marking position information corresponding to the second license plate image after the characters are automatically segmented. The principle of obtaining a first predicted character region Gaussian thermodynamic diagram, a first predicted character connected Gaussian thermodynamic diagram and a predicted license plate character of the first license plate image based on the first license plate image and the character labeling position information corresponding to the first license plate image in the pre-training process is the same, and the first predicted character connected Gaussian thermodynamic diagram and the predicted license plate character of the first license plate image are not repeated one by one, but the difference is that the character labeling position information corresponding to the first license plate image is known in a first data set, and the character labeling position information corresponding to the second license plate image is obtained by automatically segmenting characters of a cut license plate image obtained based on the second license plate image based on the pre-trained shared convolution layer module, namely, the characters are automatically segmented by the pre-trained shared convolution layer module in the formal training process to obtain the character labeling position information corresponding to the second license plate image.
In an embodiment of the application, the license plate recognition method further includes setting a confidence threshold, and calculating to obtain a confidence by using a confidence calculation formula based on the number of character frames of the second license plate image after the characters are automatically segmented and the number of real license plate characters corresponding to the second license plate image;
when the confidence coefficient is larger than or equal to the confidence coefficient threshold value, adopting a second pseudo-standard character region Gaussian thermodynamic diagram and a second pseudo-standard character Unicom Gaussian thermodynamic diagram generated by the pre-trained license plate detection recognition model to supervise formal training;
and when the confidence coefficient is smaller than the confidence coefficient threshold value, performing character segmentation on the second license plate image again by adopting a correction method.
Referring to fig. 6, fig. 6 is a flowchart illustrating quality evaluation of automatically segmented characters according to an embodiment of the present disclosure. The second pseudo-standard character region Gaussian thermodynamic diagram and the second pseudo-standard character Unicom Gaussian thermodynamic diagram generated in the mode are not always completely accurate, in order to guarantee the reliability of the formal training process, the quality of the automatically segmented characters of the second license plate image is judged by adopting an automatic segmented character quality evaluation process, and if the quality of the automatically segmented characters is better, the generated second pseudo-standard character region Gaussian thermodynamic diagram and the second pseudo-standard character Unicom Gaussian thermodynamic diagram are more accurate, and the formal training effect of the license plate detection recognition model after pre-training is better.
The confidence calculation formula is shown in formula 5, wherein for the second license plate image w, l (w) is the number of the real license plate characters corresponding to the second license plate image l*And (w) is the number of character frames after the second segmentation image automatically segments the character.
Figure BDA0002800593960000181
Setting a confidence threshold in the formal training process, and calculating to obtain a confidence based on the formula 5; when the confidence coefficient is larger than or equal to the confidence coefficient threshold value, adopting a second pseudo-standard character region Gaussian thermodynamic diagram and a second pseudo-standard character Unicom Gaussian thermodynamic diagram generated by the pre-trained license plate detection recognition model to supervise formal training; and when the confidence coefficient is smaller than the confidence coefficient threshold value, performing character segmentation on the second license plate image again by adopting a correction method, and supervising formal training by adopting a new second pseudo-standard character region Gaussian thermodynamic diagram and a second pseudo-standard character Unicom Gaussian thermodynamic diagram which are obtained based on the second license plate image subjected to character segmentation again.
Further, the character segmentation of the second license plate image by the correction method comprises:
obtaining a clipped license plate image according to the license plate boundary frame position information in the second license plate image; the method comprises the steps of converting a cut license plate into a gray-scale image, carrying out edge detection and expansion on the gray-scale image by using a Canny operator, after the edge detection is completed, segmenting a character region and a background region, marking and screening out the character position, then using a minimum external rectangular frame to frame out the position of each character in the cut license plate image, converting according to the size to obtain a boundary frame of each character in a second license plate image, and finally obtaining a new second pseudo-standard character region Gaussian thermodynamic diagram and a second pseudo-standard character connected Gaussian thermodynamic diagram which are obtained by the second license plate image subjected to character segmentation again, wherein the new second pseudo-standard character region Gaussian thermodynamic diagram and the second pseudo-standard character connected Gaussian thermodynamic diagram are used for supervising formal training of a pre-trained. Optionally, the confidence threshold may be set to 0.7, and the specific size of the confidence threshold is not limited in the present application and may be selected according to the actual situation.
In an embodiment of the present application, the error value in the formal training process includes an error value in the formal training detection process and an error value in the formal training recognition process;
calculating an error value in the formal training process, including:
calculating the mean square error of the pixel value of each point in the Gaussian thermodynamic diagram of the second predicted character region and the pixel value of the corresponding point in the Gaussian thermodynamic diagram of the second pseudo-standard character region and the mean square error of the pixel value of each point in the Gaussian thermodynamic diagram of the second character communicated with the pixel value of the corresponding point in the Gaussian thermodynamic diagram of the second pseudo-standard character communicated with the second character by using a loss function in the formal training detection process to obtain an error value in the formal training detection process;
and calculating the cross entropy of the probability distribution corresponding to each character in the predicted license plate characters of the second license plate image and the probability distribution of each character of the real license plate characters of the second license plate image by using a loss function in the formal training and recognition process to obtain an error value in the formal training and recognition process.
Specifically, on the basis of obtaining a second predicted character region Gaussian thermodynamic diagram, a second predicted character Union Gaussian thermodynamic diagram and a predicted license plate character of a second license plate image in the formal training process, combining a second pseudo standard character region Gaussian thermodynamic diagram, a second pseudo standard character Union Gaussian thermodynamic diagram and a real license plate character of the second license plate image, and obtaining an error value in the formal training detection process and an error value in the formal training recognition process by using a loss function in the formal training detection process and the recognition process, so as to obtain an error value in the formal training process.
Referring to formula 2 to formula 4, the loss function in the formal training process, the loss function in the formal training detection process, and the loss function in the formal training recognition process is different from that in the pre-training process in that a concept of confidence is introduced in the formal training process, and the loss function in the formal training detection process is updated to formula 6 on the basis of formula 3 of the loss function in the pre-training detection process.
Figure BDA0002800593960000201
p' represents a certain pixel point in the Gaussian thermodynamic diagram of the second predicted character region, Sr(p ') represents the corresponding pixel value of the point p' on the Gaussian thermodynamic diagram of the second predicted character region, Sr *(p ') representing the pixel value of a pixel point corresponding to the p' point on the Gaussian thermodynamic diagram of the second pseudo-standard character region, Sa(p ') representing the pixel value of a pixel point corresponding to the p' point in the Gaussian thermodynamic diagram of the second predicted character Unicom, Sa *(p ') representing the pixel value of the pixel point corresponding to the p' point on the second pseudo standard character Unicom Gaussian thermodynamic diagram.
Obtaining an error value in a formal training detection process and a recognition process based on a second predicted character region Gaussian thermodynamic diagram, a second predicted character Union Gaussian thermodynamic diagram, a second pseudo standard character region Gaussian thermodynamic diagram, a second pseudo standard character Union Gaussian thermodynamic diagram, a predicted license plate character of a second license plate image and a real license plate character of the second license plate image, and further referring to a process of obtaining the error value in a pre-training process for a specific process of obtaining the error value in the pre-training process, wherein a formula 6 is referred for differences, and the details are omitted here.
Further, in the pre-training and the formal training, for the license plate detection and recognition model, an ADAM optimizer is used for gradient back propagation, the learning rate is set to be 0.00003, the weight attenuation rate is set to be 0.0004, the momentum is set to be 0.9, and training is performed by taking 64 pictures as one batch until convergence, the setting of the hyper-parameters can be determined according to the actual situation, and the application does not limit the setting.
In an embodiment of the present application, obtaining a vehicle license plate detection recognition model after formal training according to an error value in a formal training process by using an error back propagation mechanism includes:
continuously updating the parameters of the pre-trained shared convolutional layer module by utilizing an error back propagation mechanism according to the error value in the formal training detection process and the error value in the identification process to obtain the formally trained shared convolutional layer module;
and continuously updating the parameters of the classifier module after the pre-training by utilizing an error back propagation mechanism according to the error value in the formal training identification process to obtain the classifier module after the formal training.
Specifically, in the process of performing formal training on the pre-trained license plate detection and recognition model based on the second license plate image in the second data set and the character marking position information corresponding to the second license plate image, when an error function in the formal training process is converged, that is, an error value in the formal training process oscillates near a certain value, the formal training process is ended, and the formally trained license plate detection and recognition model is obtained.
In an embodiment of the present application, referring to fig. 7, recognizing a license plate image to be recognized by using a license plate detection recognition model after formal training to obtain license plate information to be recognized includes:
inputting a license plate image to be recognized into the license plate detection recognition model after formal training;
in the detection process, the license plate detection recognition model after formal training outputs a character region Gaussian thermodynamic diagram and a character Unicom Gaussian thermodynamic diagram;
obtaining a binary segmentation image based on a character region Gaussian thermodynamic diagram and a character connected Gaussian thermodynamic diagram, and obtaining the bounding box information of the license plate to be recognized based on the binary segmentation image;
in the recognition process, the license plate detection recognition model after formal training outputs license plate characters to be recognized.
Optionally, the license plate image to be recognized is a vehicle image in a natural scene.
Specifically, the license plate image to be recognized is input into a license plate detection recognition model (i.e., a trained license plate detection recognition model) after formal training, and the model outputs the bounding box information (e.g., the position of the bounding box, the color of the bounding box) and the character content of the license plate in the license plate image to be recognized. The specific way of outputting the position information of the boundary frame of the license plate to be recognized in the detection process is as follows: inputting a license plate image to be recognized into a license plate detection recognition model after formal training, and outputting a character region Gaussian thermodynamic diagram and a character connected Gaussian thermodynamic diagram in the detection process; constructing a binary image M with the same size as the two Gaussian thermodynamic diagrams, and respectively setting a minimum threshold value tau of Gaussian thermodynamic image pixels in a character region by combining a formula 7rAnd a character Union Gaussian thermodynamic diagram pixel minimum threshold τaFor a certain pixel point p of the binary image M*In other words, when the character region is in Gaussian thermodynamic diagram with p*Pixel value S of corresponding pixel pointr(p*) Is greater than or equal to taurOr in character-connected Gaussian thermodynamic diagram with p*Pixel value S of corresponding pixel pointa(p*) Is greater than or equal to tauaThen pixel point p*The pixel value of (1) is set to be 1, otherwise, the pixel value of (0) is set to be 0, all pixel points in the binary image M are judged, and then a binary segmentation image is obtained; separating a point with a pixel value of 1 from a point with a pixel value of 0 in the binary segmentation image by using a connected region analysis method, and marking a region with a pixel value of 1; and framing the marked area by using the minimum circumscribed rectangle and outputting position information to obtain the position information of the boundary frame of the license plate to be recognized.
Figure BDA0002800593960000221
Further, the border frame information of the license plate to be recognized also comprises border frame colors, and the border frame colors of the license plate to be recognized are judged by utilizing a border frame color recognition function.
Specifically, the color recognition function provided by OpenCV is used to perform color recognition on the output bounding box to determine the color of the license plate.
Specifically, in the recognition process, the license plate detection recognition model after formal training outputs license plate characters to be recognized in the following specific mode: the classifier module after formal training outputs the probability distribution of all classes corresponding to each character (namely each interesting feature map in the content) to be identified, and after the probability is normalized by the softmax layer, the class with the maximum probability is selected as the content of the corresponding character to be output; all the characters needing to be recognized (namely all the extracted interesting feature maps) are output as the content of the corresponding characters by selecting the category with the maximum probability; and then all the output characters are combined to obtain the license plate characters to be recognized.
And further, the color of the license plate judged in the detection process and the number of the characters of the license plate identified in the identification process are combined, and the license plate to be identified belongs to a blue common license plate or a green new energy license plate.
Compared with the traditional license plate recognition method, the license plate recognition method realizes the integration of license plate detection and recognition, improves the training efficiency of a license plate detection recognition model, actively segments the positions of characters in the license plate through the license plate detection recognition model, does not need to participate in the character segmentation process artificially, and ensures that the detection recognition result is more accurate and stable; in addition, the license plate detection and identification method can detect and identify a wide range of license plate types, has high detection and identification rate, and has good robustness and use value.
Those skilled in the art will appreciate that all or part of the flow of the method implementing the above embodiments may be implemented by a computer program, which is stored in a computer readable storage medium, to instruct related hardware. The computer readable storage medium is a magnetic disk, an optical disk, a read-only memory or a random access memory.
The above description is only for the preferred embodiment of the present invention, but the scope of the present invention is not limited thereto, and any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope of the present invention are included in the scope of the present invention.

Claims (10)

1. A license plate recognition method based on automatic character region perception is characterized by comprising the following steps:
pre-training the initial license plate detection and recognition model based on the first data set to obtain a pre-trained license plate detection and recognition model; the first data set comprises a first license plate image and character marking position information corresponding to the first license plate image;
formally training the pre-trained license plate detection and recognition model based on a second data set to obtain a formally trained license plate detection and recognition model; wherein the second data set comprises a second license plate image and license plate bounding box position information;
and recognizing the license plate image to be recognized by using the formally trained license plate detection recognition model to obtain the license plate information to be recognized.
2. The license plate recognition method of claim 1, wherein the pre-training the initial license plate detection recognition model based on the first data set to obtain a pre-trained license plate detection recognition model comprises:
inputting the first data set into an initial license plate detection and recognition model to obtain a first predicted character region Gaussian thermodynamic diagram, a first predicted character Unicom Gaussian thermodynamic diagram and predicted license plate characters of the first license plate image;
calculating an error value in the pre-training process according to the first predicted character region Gaussian thermodynamic diagram, the first standard character region Gaussian thermodynamic diagram, the first predicted character Unicom Gaussian thermodynamic diagram, the first standard character Unicom Gaussian thermodynamic diagram, the predicted license plate character of the first license plate image and the real license plate character of the first license plate image by using a loss function in the pre-training process;
and obtaining a pre-trained license plate detection and recognition model according to the error value in the pre-training process by using an error back propagation mechanism.
3. The license plate recognition method of claim 2, wherein the initial license plate detection recognition model comprises an initial shared convolutional layer module and an initial classifier module;
the inputting the first data set into an initial license plate detection recognition model to obtain a first predicted character region Gaussian thermodynamic diagram, a first predicted character Unicom Gaussian thermodynamic diagram and predicted license plate characters of the first license plate image comprises the following steps:
based on a first license plate image in a first data set, the initial shared convolutional layer module outputs a first predicted character region Gaussian thermodynamic diagram, a first predicted character Unicom Gaussian thermodynamic diagram and a first feature map;
based on the character labeling position information in the first data set and the first feature map, the initial classifier module outputs the predicted license plate characters of the first license plate image.
4. The license plate recognition method of claim 3, wherein the error value in the pre-training process comprises an error value in a pre-training detection process and an error value in a pre-training recognition process;
the calculating the error value in the pre-training process comprises:
calculating the mean square error of the pixel value of each point in the first predicted character region Gaussian thermodynamic diagram and the pixel value of the corresponding point in the first standard character region Gaussian thermodynamic diagram and the mean square error of the pixel value of each point in the first character communicated Gaussian thermodynamic diagram and the pixel value of the corresponding point in the first standard character communicated Gaussian thermodynamic diagram by using a loss function in the pre-training detection process to obtain an error value in the pre-training detection process;
and calculating the cross entropy of the probability distribution corresponding to each character in the predicted license plate characters of the first license plate image and the probability distribution of the corresponding character in the real license plate characters of the first license plate image by using a loss function in the pre-training recognition process to obtain an error value in the pre-training recognition process.
5. The license plate recognition method of any one of claims 1 to 4, wherein the formally training the pre-trained license plate detection recognition model based on the second data set to obtain a formally trained license plate detection recognition model comprises:
inputting the second data set into a pre-trained license plate detection recognition model to obtain a second pseudo standard character region Gaussian thermodynamic diagram, a second pseudo standard character Unicom Gaussian thermodynamic diagram, a second predicted character region Gaussian thermodynamic diagram, a second predicted character Unicom Gaussian thermodynamic diagram and predicted license plate characters of the second license plate image;
calculating an error value in the formal training process according to the second predicted character region Gaussian thermodynamic diagram, the second pseudo standard character region Gaussian thermodynamic diagram, the second predicted character Unicom Gaussian thermodynamic diagram, the second pseudo standard character Unicom Gaussian thermodynamic diagram, the predicted license plate character of the second license plate image and the real license plate character of the second license plate image by using a loss function in the formal training process;
and obtaining a license plate detection and recognition model after formal training according to the error value in the formal training process by using an error back propagation mechanism.
6. The license plate recognition method of claim 5, wherein the pre-trained license plate detection recognition model comprises a pre-trained shared convolutional layer module and a pre-trained classifier module, and the inputting the second data set into the pre-trained license plate detection recognition model to obtain a second pseudo-standard character region Gaussian thermodynamic diagram, a second pseudo-standard character UnionGaussian thermodynamic diagram, a second predicted character region Gaussian thermodynamic diagram, a second predicted character UnionGaussian thermodynamic diagram, and predicted license plate characters of the second license plate image comprises:
obtaining a clipped license plate image according to the license plate boundary frame position information in the second license plate image;
based on the cut license plate image, automatically segmenting characters of the second license plate image by using a pre-trained shared convolutional layer module to obtain character marking position information corresponding to the second license plate image, a second pseudo-standard character region Gaussian thermodynamic diagram and a second pseudo-standard character Unicom Gaussian thermodynamic diagram;
based on the second license plate image, the pre-trained shared convolutional layer module outputs a second predicted character region Gaussian thermodynamic diagram, a second predicted character Unicom Gaussian thermodynamic diagram and a second feature diagram;
and outputting the predicted license plate characters of the second license plate image by the pre-trained classifier module based on the character marking position information corresponding to the second license plate image and the second feature map.
7. The license plate recognition method of claim 6, wherein the error value in the formal training process comprises an error value in a formal training detection process and an error value in a formal training recognition process;
the calculating the error value in the formal training process includes:
calculating the mean square error of the pixel value of each point in the Gaussian thermodynamic diagram of the second predicted character region and the pixel value of the corresponding point in the Gaussian thermodynamic diagram of the second pseudo-standard character region and the mean square error of the pixel value of each point in the Gaussian thermodynamic diagram of the second predicted character communicated with the second pseudo-standard character communicated with the Gaussian thermodynamic diagram by using a loss function in the formal training detection process to obtain an error value in the formal training detection process;
and calculating the cross entropy of the probability distribution corresponding to each character in the predicted license plate characters of the second license plate image and the probability distribution of each character of the real license plate characters of the second license plate image by using a loss function in the formal training and recognition process to obtain an error value in the formal training and recognition process.
8. The license plate recognition method of claim 6, further comprising:
setting a confidence threshold, and calculating to obtain a confidence based on the number of character frames of the second license plate image after the characters are automatically segmented and the number of real license plate characters corresponding to the second license plate image by using a confidence calculation formula;
when the confidence coefficient is larger than or equal to the confidence coefficient threshold value, adopting a second pseudo-standard character region Gaussian thermodynamic diagram and a second pseudo-standard character Unicom Gaussian thermodynamic diagram generated by the pre-trained license plate detection recognition model to supervise formal training;
and when the confidence coefficient is smaller than the confidence coefficient threshold value, performing character segmentation on the second license plate image again by adopting a correction method.
9. The license plate recognition method of claim 1, wherein the recognizing the license plate image to be recognized by using the formally trained license plate detection recognition model to obtain the license plate information to be recognized comprises:
inputting the license plate image to be recognized into the license plate detection recognition model after formal training;
in the detection process, the license plate detection recognition model after formal training outputs a character region Gaussian thermodynamic diagram and a character Unicom Gaussian thermodynamic diagram;
obtaining a binary segmentation image based on the character region Gaussian thermodynamic diagram and the character Unicom Gaussian thermodynamic diagram, and obtaining the bounding box information of the license plate to be recognized based on the binary segmentation image;
in the recognition process, the license plate detection recognition model after formal training outputs license plate characters to be recognized.
10. The license plate recognition method of claim 9, wherein the bounding box information of the license plate to be recognized includes bounding box position information and bounding box color, the recognition method further comprising:
and judging the color of the boundary frame of the license plate to be recognized based on the boundary frame color recognition function.
CN202011348240.2A 2020-11-26 2020-11-26 License plate recognition method based on automatic character region perception Active CN112287941B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202011348240.2A CN112287941B (en) 2020-11-26 2020-11-26 License plate recognition method based on automatic character region perception

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202011348240.2A CN112287941B (en) 2020-11-26 2020-11-26 License plate recognition method based on automatic character region perception

Publications (2)

Publication Number Publication Date
CN112287941A true CN112287941A (en) 2021-01-29
CN112287941B CN112287941B (en) 2021-11-09

Family

ID=74426449

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202011348240.2A Active CN112287941B (en) 2020-11-26 2020-11-26 License plate recognition method based on automatic character region perception

Country Status (1)

Country Link
CN (1) CN112287941B (en)

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113221867A (en) * 2021-05-11 2021-08-06 北京邮电大学 Deep learning-based PCB image character detection method
CN113392837A (en) * 2021-07-09 2021-09-14 超级视线科技有限公司 License plate recognition method and device based on deep learning
CN113554030A (en) * 2021-07-27 2021-10-26 上海大学 Multi-type license plate recognition method and system based on single character attention
CN115116047A (en) * 2022-08-29 2022-09-27 松立控股集团股份有限公司 License plate character region thermodynamic diagram-based license plate detection method
CN115170807A (en) * 2022-09-05 2022-10-11 浙江大华技术股份有限公司 Image segmentation and model training method, device, equipment and medium
CN115205861A (en) * 2022-08-17 2022-10-18 北京睿企信息科技有限公司 Method for acquiring abnormal character recognition area, electronic equipment and storage medium

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150049914A1 (en) * 2013-08-13 2015-02-19 James Alves License Plate Recognition
CN109753914A (en) * 2018-12-28 2019-05-14 安徽清新互联信息科技有限公司 A kind of license plate character recognition method based on deep learning
CN110555460A (en) * 2019-07-31 2019-12-10 国网江苏省电力有限公司 Image slice-based bird detection method for power transmission line at mobile terminal
CN110619327A (en) * 2018-06-20 2019-12-27 湖南省瞬渺通信技术有限公司 Real-time license plate recognition method based on deep learning in complex scene
CN111401361A (en) * 2020-03-06 2020-07-10 南京理工大学 End-to-end lightweight deep license plate recognition method
CN111814860A (en) * 2020-07-01 2020-10-23 浙江工业大学 Multi-target detection method for garbage classification
CN111860378A (en) * 2020-07-27 2020-10-30 顾俊成 Market fire-fighting equipment inspection method based on gun-ball linkage and video event perception

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150049914A1 (en) * 2013-08-13 2015-02-19 James Alves License Plate Recognition
CN110619327A (en) * 2018-06-20 2019-12-27 湖南省瞬渺通信技术有限公司 Real-time license plate recognition method based on deep learning in complex scene
CN109753914A (en) * 2018-12-28 2019-05-14 安徽清新互联信息科技有限公司 A kind of license plate character recognition method based on deep learning
CN110555460A (en) * 2019-07-31 2019-12-10 国网江苏省电力有限公司 Image slice-based bird detection method for power transmission line at mobile terminal
CN111401361A (en) * 2020-03-06 2020-07-10 南京理工大学 End-to-end lightweight deep license plate recognition method
CN111814860A (en) * 2020-07-01 2020-10-23 浙江工业大学 Multi-target detection method for garbage classification
CN111860378A (en) * 2020-07-27 2020-10-30 顾俊成 Market fire-fighting equipment inspection method based on gun-ball linkage and video event perception

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
YOUNGMIN BAEK.ET AL: ""character region awareness for text detection"", 《ARXIV:1904.01941V1》 *
程欣: ""基于深度学习的图像目标定位识别研究"", 《中国优秀硕士学位论文全文数据库（电子期刊）》 *

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113221867A (en) * 2021-05-11 2021-08-06 北京邮电大学 Deep learning-based PCB image character detection method
CN113392837A (en) * 2021-07-09 2021-09-14 超级视线科技有限公司 License plate recognition method and device based on deep learning
CN113554030A (en) * 2021-07-27 2021-10-26 上海大学 Multi-type license plate recognition method and system based on single character attention
CN115205861A (en) * 2022-08-17 2022-10-18 北京睿企信息科技有限公司 Method for acquiring abnormal character recognition area, electronic equipment and storage medium
CN115205861B (en) * 2022-08-17 2023-03-31 北京睿企信息科技有限公司 Method for acquiring abnormal character recognition area, electronic equipment and storage medium
CN115116047A (en) * 2022-08-29 2022-09-27 松立控股集团股份有限公司 License plate character region thermodynamic diagram-based license plate detection method
CN115170807A (en) * 2022-09-05 2022-10-11 浙江大华技术股份有限公司 Image segmentation and model training method, device, equipment and medium

Also Published As

Publication number Publication date
CN112287941B (en) 2021-11-09

Similar Documents

Publication Publication Date Title
CN112287941B (en) License plate recognition method based on automatic character region perception
CN109949317B (en) Semi-supervised image example segmentation method based on gradual confrontation learning
CN109886121B (en) Human face key point positioning method for shielding robustness
CN110276264B (en) Crowd density estimation method based on foreground segmentation graph
CN113673338B (en) Automatic labeling method, system and medium for weak supervision of natural scene text image character pixels
CN111950453A (en) Optional-shape text recognition method based on selective attention mechanism
CN113255659B (en) License plate correction detection and identification method based on MSAFF-yolk 3
CN111639564B (en) Video pedestrian re-identification method based on multi-attention heterogeneous network
CN111612008A (en) Image segmentation method based on convolution network
CN113609896A (en) Object-level remote sensing change detection method and system based on dual-correlation attention
CN114663346A (en) Strip steel surface defect detection method based on improved YOLOv5 network
CN111986125A (en) Method for multi-target task instance segmentation
CN113052170B (en) Small target license plate recognition method under unconstrained scene
CN110969171A (en) Image classification model, method and application based on improved convolutional neural network
CN113297956B (en) Gesture recognition method and system based on vision
CN112215190A (en) Illegal building detection method based on YOLOV4 model
CN112084859A (en) Building segmentation method based on dense boundary block and attention mechanism
CN110852327A (en) Image processing method, image processing device, electronic equipment and storage medium
CN115424017B (en) Building inner and outer contour segmentation method, device and storage medium
CN113743505A (en) Improved SSD target detection method based on self-attention and feature fusion
CN111274964B (en) Detection method for analyzing water surface pollutants based on visual saliency of unmanned aerial vehicle
CN113052215A (en) Sonar image automatic target identification method based on neural network visualization
CN113361467A (en) License plate recognition method based on field adaptation
CN115240259A (en) Face detection method and face detection system based on YOLO deep network in classroom environment
CN114550134A (en) Deep learning-based traffic sign detection and identification method

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant