CN110298791B

CN110298791B - Super-resolution reconstruction method and device for license plate image

Info

Publication number: CN110298791B
Application number: CN201910610927.XA
Authority: CN
Inventors: 王殿伟; 郝元杰; 宋鸽; 李娜; 李大湘; 刘颖
Original assignee: Xian University of Posts and Telecommunications
Current assignee: Xian University of Posts and Telecommunications
Priority date: 2019-07-08
Filing date: 2019-07-08
Publication date: 2022-10-28
Anticipated expiration: 2039-07-08
Also published as: CN110298791A

Abstract

The disclosure relates to the field of image processing, in particular to a super-resolution reconstruction method and device for a license plate image, and is applied to a terminal. The method comprises the following steps: firstly, shallow feature extraction is carried out on a low-resolution license plate image to be processed through a first convolution layer; secondly, depth feature extraction is carried out through N identical residual error sampling blocks and a second convolution layer which are sequentially connected in series, and different branches built in the residual error sampling blocks fully utilize the scale and depth information of features; and finally, amplifying the plurality of deep characteristic maps obtained by the characteristic extraction module by multiple times step by step through a reconstruction module, and obtaining a reconstructed target high-resolution license plate image according to the plurality of high-resolution characteristic maps. According to the method, the problem that the extracted characteristic information is lost is solved by performing super-resolution reconstruction processing on the low-resolution license plate image, so that the reconstructed license plate avoids the conditions of character blurring, character adhesion, insufficient information and the like, and the recognition degree of the license plate image is improved.

Description

Super-resolution reconstruction method and device for license plate image

Technical Field

The disclosure relates to the field of image processing, in particular to a super-resolution reconstruction method and device for a license plate image.

Background

The super-resolution reconstruction of the license plate image means that the corresponding most possible high-resolution license plate image is obtained from the degraded low-resolution license plate image by a software method. The super-resolution reconstruction technology is widely applied and mainly applied to the fields of public safety, medical images, remote sensing images, high-definition televisions and the like, so the super-resolution reconstruction of the images is very important.

At present, a super-resolution reconstruction method of a license plate image is an interpolation-based method, and the method obtains a value of an estimation point by using a neighborhood sampling point of a low-resolution image through a certain functional relationship, and inserts the value between the neighborhood sampling points. However, the number plate image reconstructed by the interpolation-based method has less detail information, and characters in the number plate image are adhered to each other, so that the characters in the number plate image are difficult to recognize, that is, the recognition degree of the number plate image is low.

Disclosure of Invention

In order to solve the problems in the prior art, the embodiment of the disclosure provides a super-resolution reconstruction method and device for a license plate image. The technical scheme is as follows:

on one hand, the super-resolution reconstruction method of the license plate image is provided and applied to a terminal, an image processing network is deployed on the terminal and comprises a feature extraction module and a reconstruction module, the feature extraction module comprises an initialization feature extraction module and a depth feature extraction module which are connected, the initialization feature extraction module comprises a first convolution layer, the depth feature extraction module comprises N identical residual sampling blocks and a second convolution layer which are sequentially connected in series, N is an integer greater than or equal to 1, each residual sampling block comprises a first branch, a second branch and a third branch, the first branch comprises an upper sampling layer, a third convolution layer and a lower sampling layer which are sequentially connected, the second branch comprises a fourth convolution layer, the third branch is an identity mapping branch, an output feature map of the identity mapping branch is identical to an input feature map, and the method comprises the following steps:

performing shallow feature extraction on the low-resolution license plate image to be processed through the first convolution layer to obtain a plurality of shallow feature maps;

carrying out first depth feature extraction on the shallow feature maps through the N identical residual error sampling blocks which are sequentially connected in series to obtain a plurality of local fourth deep feature maps output by the Nth residual error sampling block;

performing second depth feature extraction on a plurality of local fourth deep feature maps output by the Nth residual sampling block through the second convolution layer to obtain a plurality of fifth deep feature maps, and performing feature fusion on the plurality of fifth deep feature maps and the plurality of shallow feature maps in a one-to-one correspondence manner to obtain a plurality of sixth deep feature maps;

the reconstruction module is used for amplifying the sixth deep feature maps obtained by the feature extraction module by multiple times step by step to obtain a plurality of high-resolution feature maps, and a target high-resolution license plate image is obtained according to the high-resolution feature maps and is obtained by performing super-resolution reconstruction on the low-resolution license plate image through the image processing network;

the first depth feature extraction is performed on the multiple shallow feature maps through the N identical residual sampling blocks connected in series in sequence to obtain multiple local fourth deep feature maps output by the nth residual sampling block, and the method includes:

carrying out scale transformation on a plurality of target feature maps through a first branch in an ith residual sampling block in the N same residual sampling blocks which are sequentially connected in series to obtain a plurality of local first deep feature maps, carrying out feature extraction on the plurality of target feature maps through a second branch in the ith residual sampling block to obtain a plurality of local second deep feature maps, and carrying out identity mapping on the plurality of target feature maps through a third branch in the ith residual sampling block to obtain a plurality of local third deep feature maps; performing feature fusion on the plurality of local first deep feature maps, the plurality of local second deep feature maps and the plurality of local third deep feature maps in a one-to-one correspondence manner through the ith residual sampling block to obtain a plurality of local fourth deep feature maps; acquiring a plurality of local fourth deep feature maps output by the Nth residual error sampling block; when i is not equal to 1, the target feature maps are local fourth deep feature maps output by the i-1 th residual sample block.

Optionally, the performing scale transformation on the multiple target feature maps through a first branch in an ith residual sample block of the N identical residual sample blocks connected in series in sequence to obtain multiple local first deep feature maps includes:

the target feature maps are up-sampled through an up-sampling layer in the first branch to obtain a plurality of up-sampling feature maps;

performing feature extraction on the plurality of upsampled feature maps through a third convolution layer in the first branch to obtain a plurality of third convolution feature maps;

and downsampling the plurality of third convolution feature maps through a downsampling layer in the first branch to obtain a plurality of downsampling feature maps, and determining the plurality of downsampling feature maps as the plurality of local first deep feature maps, wherein the upsampling adopts transposition convolution and the downsampling adopts convolution.

Optionally, the upsampling the multiple target feature maps by the upsampling layer in the first branch to obtain multiple upsampled feature maps includes:

through an upsampling layer in the first branch, upsampling the multiple target feature maps according to a first formula as follows to obtain multiple upsampled feature maps:

the first formula:

wherein the content of the first and second substances,

is the ith residueA plurality of upsampled feature maps of the difference sample blocks phi being the non-linear mapping function or activation function of the ith residual sample block R _i-1 For the plurality of target feature maps, W _i And b _i Weights and offsets, respectively, # e @ f, for the ith residual sample block _s Represents the upsampling by a factor of s;

performing feature extraction on the plurality of upsampled feature maps through a third convolutional layer in the first branch to obtain a plurality of third convolutional feature maps, including:

performing feature extraction on the plurality of upsampled feature maps through a third convolution layer in the first branch according to a second formula as follows to obtain a plurality of third convolution feature maps:

the second formula:

wherein the content of the first and second substances,

a plurality of third convolution maps for an ith residual sample block;

the downsampling the plurality of third convolution feature maps by the downsampling layer in the first branch to obtain a plurality of downsampled feature maps includes:

downsampling the plurality of third convolution feature maps according to a third formula to obtain a plurality of downsampled feature maps by using a downsampling layer in the first branch:

the third formula:

wherein the content of the first and second substances,

for a plurality of downsampled feature maps of the ith residual sample block, ↓ s is downsampled by s times.

The depth feature extraction is performed on the multiple target feature maps through a second branch in the ith residual sampling block to obtain multiple local second deep feature maps, and the method comprises the following steps:

performing feature extraction on the plurality of target feature maps through a second branch in the ith residual sampling block according to a fourth formula to obtain a plurality of local second deep feature maps:

the fourth formula:

wherein, the first and the second end of the pipe are connected with each other,

a plurality of local second depth profiles are represented.

Optionally, the performing feature fusion on the plurality of local first deep feature maps, the plurality of local second deep feature maps and the plurality of local third deep feature maps through the i-th residual sampling block to obtain a plurality of local fourth deep feature maps includes:

performing feature fusion on the plurality of local first deep feature maps, the plurality of local second deep feature maps and the plurality of local third deep feature maps according to a fifth formula to obtain a plurality of local fourth deep feature maps by using the ith residual sampling block:

the fifth formula:

wherein R is _i A plurality of local fourth deep feature maps output for the ith residual sample block,

a plurality of local first deep feature maps from a first branch of an ith residual sample block,

multiple local second deep feature maps, R, obtained for a second branch in an ith residual sample block _i-1 Multiple bureaus derived for the third branch of the ith residual sample blockAnd (c) a third deep level feature map.

On the other hand, a super-resolution reconstruction device for license plate images is provided, which is applied to a terminal, an image processing network is deployed on the terminal, the image processing network includes a feature extraction module and a reconstruction module, the feature extraction module includes an initialization feature extraction module and a depth feature extraction module which are connected, the initialization feature extraction module includes a first convolution layer, the depth feature extraction module includes N same residual sampling blocks and a second convolution layer which are connected in series in sequence, N is an integer greater than or equal to 1, each residual sampling block includes a first branch, a second branch and a third branch, the first branch includes an upper sampling layer, a third convolution layer and a lower sampling layer which are connected in sequence, the second branch includes a fourth convolution layer, the third branch is an identity mapping branch, an output feature map of the identity mapping branch is the same as an input feature map, and the device includes:

the initialization feature extraction module is used for performing shallow feature extraction on the low-resolution license plate image to be processed through the first convolution layer to obtain a plurality of shallow feature maps;

the depth feature extraction module is used for performing first depth feature extraction on the plurality of shallow feature maps through the N same residual error sampling blocks which are sequentially connected in series to obtain a plurality of local fourth deep feature maps output by the Nth residual error sampling block;

the feature fusion module is configured to perform second depth feature extraction on a plurality of local fourth deep feature maps output by the nth residual sampling block through the second convolution layer to obtain a plurality of fifth deep feature maps, and perform feature fusion on the plurality of fifth deep feature maps and the plurality of shallow feature maps in a one-to-one correspondence manner to obtain a plurality of sixth deep feature maps;

the super-resolution reconstruction module is used for amplifying the sixth deep feature maps obtained by the feature extraction module by multiple times step by step through the reconstruction module to obtain a plurality of high-resolution feature maps, and obtaining a target high-resolution license plate image according to the high-resolution feature maps, wherein the target high-resolution license plate image is a license plate image obtained by performing super-resolution reconstruction on the low-resolution license plate image through the image processing network;

the depth feature extraction module is further configured to perform scale transformation on multiple target feature maps through a first branch in an ith residual sampling block of the N identical residual sampling blocks connected in series in sequence to obtain multiple local first deep feature maps, perform feature extraction on the multiple target feature maps through a second branch in the ith residual sampling block to obtain multiple local second deep feature maps, and perform identity mapping on the multiple target feature maps through a third branch in the ith residual sampling block to obtain multiple local third deep feature maps; performing feature fusion on the plurality of local first deep feature maps, the plurality of local second deep feature maps and the plurality of local third deep feature maps in a one-to-one correspondence manner through the ith residual sampling block to obtain a plurality of local fourth deep feature maps; acquiring a plurality of local fourth deep feature maps output by the Nth residual error sampling block; when i is not equal to 1, the target feature maps are local fourth deep feature maps output by the i-1 th residual sample block.

Optionally, the depth feature extraction module includes:

the first sampling submodule is used for performing upsampling on the plurality of target characteristic graphs through an upsampling layer in the first branch to obtain a plurality of upsampled characteristic graphs;

the feature extraction sub-module is used for performing feature extraction on the plurality of upsampled feature maps through a third convolution layer in the first branch to obtain a plurality of third convolution feature maps;

and the second sampling sub-module is configured to perform downsampling on the plurality of third convolution feature maps through a downsampling layer in the first branch to obtain a plurality of downsampling feature maps, and determine the plurality of downsampling feature maps as the plurality of local first deep feature maps, where the upsampling is performed by using a transposed convolution and the downsampling is performed by using a convolution.

Optionally, the first sampling sub-module is configured to, through an upsampling layer in the first branch, upsample the multiple target feature maps according to a first formula as follows to obtain multiple upsampled feature maps:

the first formula:

wherein the content of the first and second substances,

a plurality of up-sampling feature maps for the ith residual sample block, phi is a non-linear mapping function or activation function, R, of the ith residual sample block _i-1 For the target feature map, W _i And b _i Weights and offsets, respectively, # e @ f, for the ith residual sample block _s Represents the upsampling by a factor of s;

the feature extraction submodule is configured to perform feature extraction on the plurality of upsampled feature maps through a plurality of third convolution layers in the first branch according to a second formula as follows to obtain a plurality of third convolution feature maps:

the second formula:

wherein the content of the first and second substances,

a plurality of third convolution maps for an ith residual sample block;

the second sampling sub-module is configured to perform downsampling on the plurality of third convolution feature maps according to a following third formula by using a downsampling layer in the first branch to obtain a plurality of downsampled feature maps:

the third formula:

wherein the content of the first and second substances,

a plurality of down-sampled feature maps, ↓, for the i-th residual sample block _s Is downsampled by a factor of s.

Optionally, the depth feature extraction module is further configured to perform feature extraction on the multiple target feature maps through a second branch in the ith residual sample block according to a fourth formula as follows to obtain multiple local second deep feature maps:

the fourth formula:

wherein the content of the first and second substances,

a plurality of local second depth profiles are represented.

Optionally, the depth feature extraction module is configured to perform feature fusion on the multiple local first deep feature maps, the multiple local second deep feature maps, and the multiple local third deep feature maps according to a fifth formula below by using the ith residual sampling block to obtain multiple local fourth deep feature maps:

the fifth formula:

a plurality of local first deep feature maps obtained from a first branch of an ith residual sample block,

multiple local second deep feature maps, R, for a second branch of the ith residual sample block _i-1 A plurality of local third deep feature maps for a third branch of the ith residual sample block.

The technical scheme provided by the embodiment of the disclosure at least has the following beneficial effects:

in the embodiment of the disclosure, shallow feature extraction is performed on a low-resolution license plate image to be processed through a first convolution layer to obtain a plurality of shallow feature maps, first depth feature extraction is performed on the plurality of shallow feature maps through N residual sampling blocks which are sequentially connected in series to obtain a plurality of local fourth deep feature maps output by the Nth residual sampling block, and then, based on the plurality of local fourth deep feature maps, a target high-resolution license plate image is finally obtained through a second convolution layer and a reconstruction module. Each residual sampling block comprises three branches, so that when depth feature extraction is carried out for the first time, a plurality of target feature maps can be processed through the three branches of each residual sampling block, feature fusion is carried out on a plurality of local first deep feature maps, a plurality of local second deep feature maps and a plurality of local third deep feature maps which are obtained through processing in a one-to-one correspondence mode, a plurality of local fourth deep feature maps are obtained, and then a plurality of local fourth deep feature maps output by the Nth residual sampling blocks are obtained. Because the first branch includes the upsampling layer, the third convolution layer and the downsampling layer that connect gradually, can increase the detailed information of characteristic map, the second branch fully draws the characteristic of low resolution characteristic map through the fourth convolution layer, the third branch can avoid appearing the gradient disappearance and lead to rebuilding the poor problem of effect among the reconstruction process, consequently, carry out the feature fusion with a plurality of local deep characteristic maps that these three branches obtained and can provide more information for the reconstruction module, the problem of the characteristic information loss of having solved the extraction makes the license plate of rebuilding avoid the characters fuzzy, the character adhesion, the condition such as information is not abundant, the degree of discerning of license plate image has been improved.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present disclosure, and it is obvious for those skilled in the art to obtain other drawings based on the drawings without creative efforts.

Fig. 1 is a schematic diagram of an image processing network according to an embodiment of the present disclosure.

FIG. 2 is a flowchart of a super-resolution reconstruction method for license plate images provided by an embodiment of the present disclosure

Fig. 3 is a flowchart of a super-resolution reconstruction method for a license plate image according to an embodiment of the present disclosure.

Fig. 4 is a schematic diagram of a super-resolution reconstruction apparatus for a license plate image according to an embodiment of the present disclosure.

Detailed Description

To make the objects, technical solutions and advantages of the present disclosure more apparent, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

Fig. 1 is a schematic diagram of an image processing network provided by an embodiment of the present disclosure, and referring to fig. 1, the image processing network may be deployed on a terminal. The image processing network comprises a feature extraction module 1 and a reconstruction module 2, the feature extraction module 1 comprises a shallow feature extraction module 11 and a depth feature extraction module 12 which are connected, the initialization feature extraction module 11 comprises a first convolution layer 111, the depth feature extraction module 12 comprises N identical residual error sampling blocks 121 and a second convolution layer 122 which are sequentially connected in series, each residual error sampling block 121 comprises a first branch A, a second branch B and a third branch C, the first branch A comprises an upper sampling layer A1, a third convolution layer A2 and a lower sampling layer A3 which are sequentially connected, the second branch B comprises a fourth convolution layer B1, and the third branch C is an identity mapping branch. In the image processing network, the solid line "a" may indicate that feature fusion is performed between a plurality of fifth deep-layer feature maps output from the second convolution layer 122 and a plurality of shallow-layer feature maps in one-to-one correspondence.

The embodiment of the disclosure provides a super-resolution reconstruction method of a license plate image, and referring to fig. 2, the method comprises the following steps:

step 201: and performing shallow feature extraction on the low-resolution license plate image to be processed through the first convolution layer to obtain a plurality of shallow feature maps.

Step 202: and performing first depth feature extraction on the plurality of shallow feature maps through the N same residual error sampling blocks which are sequentially connected in series to obtain a plurality of local fourth deep feature maps output by the Nth residual error sampling block.

Carrying out scale transformation on a plurality of target characteristic maps through first branches in an ith residual sampling block in N identical residual sampling blocks which are sequentially connected in series to obtain a plurality of local first deep characteristic maps, carrying out characteristic extraction on the plurality of target characteristic maps through second branches in the ith residual sampling block to obtain a plurality of local second deep characteristic maps, and carrying out identity mapping on the plurality of target characteristic maps through third branches in the ith residual sampling block to obtain a plurality of local third deep characteristic maps; performing feature fusion on the plurality of local first deep feature maps, the plurality of local second deep feature maps and the plurality of local third deep feature maps in a one-to-one correspondence manner through the ith residual sampling block to obtain a plurality of local fourth deep feature maps; obtaining a plurality of local fourth deep characteristic maps output by the Nth residual error sampling block; when i is not equal to 1, the target feature maps are local fourth deep feature maps output by the i-1 th residual sampling block.

Step 203: and performing second depth feature extraction on a plurality of local fourth deep feature maps output by the Nth residual sampling block through the second convolution layer to obtain a plurality of fifth deep feature maps, and performing feature fusion on the plurality of fifth deep feature maps and the plurality of shallow feature maps in a one-to-one correspondence manner to obtain a plurality of sixth deep feature maps.

Step 204: and the reconstruction module is used for amplifying the plurality of sixth deep feature maps obtained by the feature extraction module by multiple times step by step to obtain a plurality of high-resolution feature maps, and obtaining a target high-resolution license plate image according to the plurality of high-resolution feature maps, wherein the target high-resolution license plate image is a license plate image obtained by performing super-resolution reconstruction on a low-resolution license plate image through an image processing network.

Optionally, performing scale transformation on the multiple target feature maps through a first branch in an ith residual sample block of the N identical residual sample blocks connected in series in sequence to obtain multiple local first deep feature maps, where the scale transformation includes:

the plurality of target characteristic graphs are subjected to upsampling through an upsampling layer in the first branch to obtain a plurality of upsampling characteristic graphs;

performing feature extraction on the plurality of upsampling feature maps through a third convolution layer in the first branch to obtain a plurality of third convolution feature maps;

and downsampling the plurality of third convolution feature maps through a downsampling layer in the first branch to obtain a plurality of downsampling feature maps, and determining the plurality of downsampling feature maps into a plurality of local first deep feature maps, wherein the transposed convolution is adopted for upsampling, and the convolution is adopted for downsampling.

the first formula:

a plurality of up-sampling feature maps for the ith residual sample block, phi is a non-linear mapping function or activation function, R, for the ith residual sample block _i-1 For a plurality of target feature maps, W _i And b _i Weights and offsets, respectively, # e @ f, for the ith residual sample block _s Represents the upsampling by a factor of s;

and performing feature extraction on the plurality of upsampled feature maps through a third convolution layer in the first branch according to a second formula to obtain a plurality of third convolution feature maps:

the second formula:

wherein the content of the first and second substances,

a plurality of third convolution maps for an ith residual sample block;

the method for downsampling the third convolution feature map through the downsampling layer in the first branch to obtain a plurality of downsampled feature maps comprises the following steps:

and downsampling the plurality of third convolution feature maps according to a following third formula by a downsampling layer in the first branch to obtain a plurality of downsampled feature maps:

the third formula:

wherein the content of the first and second substances,

for a plurality of down-sampled feature maps of the i-th residual sample block, ↓ s is the down-sample s times.

Optionally, performing depth feature extraction on the multiple target feature maps through a second branch in the ith residual sample block to obtain multiple local second deep feature maps, including:

performing feature extraction on the plurality of target feature maps through a second branch in the ith residual error sampling block according to a fourth formula to obtain a plurality of local second deep feature maps:

the fourth formula:

representing a plurality of local second depth profiles。

Optionally, feature fusing the multiple local first deep feature maps, the multiple local second deep feature maps, and the multiple local third deep feature maps through an ith residual sampling block to obtain multiple local fourth deep feature maps, including:

performing feature fusion on the multiple local first deep feature maps, the multiple local second deep feature maps and the multiple local third deep feature maps through an ith residual sampling block according to a fifth formula to obtain multiple local fourth deep feature maps:

the fifth formula:

In summary, in the embodiment of the present disclosure, shallow feature extraction is performed on a low-resolution license plate image to be processed through a first convolution layer to obtain a plurality of shallow feature maps, first depth feature extraction is performed on the plurality of shallow feature maps through N residual sampling blocks connected in series in sequence to obtain a plurality of local fourth deep feature maps output by the nth residual sampling block, and then, based on the plurality of local fourth deep feature maps, a target high-resolution license plate image is finally obtained through a second convolution layer and a reconstruction module. Each residual sampling block comprises three branches, so that when depth feature extraction is carried out for the first time, a plurality of target feature maps can be processed through the three branches of each residual sampling block, feature fusion is carried out on a plurality of local first deep feature maps, a plurality of local second deep feature maps and a plurality of local third deep feature maps which are obtained through processing in a one-to-one correspondence mode, a plurality of local fourth deep feature maps are obtained, and then a plurality of local fourth deep feature maps output by the Nth residual sampling blocks are obtained. Because the first branch comprises an upper sampling layer, a third convolution layer and a lower sampling layer which are sequentially connected, the detailed information of the feature map can be increased, the second branch fully extracts the features of the low-resolution feature map through the fourth convolution layer, and the third branch can avoid the problem of poor reconstruction effect caused by gradient disappearance in the reconstruction process.

The embodiment of the present disclosure provides a super-resolution reconstruction method for a license plate image, referring to fig. 3, the implementation environment includes deep learning frames such as tensrflow, tensrlayer, and the like, that is, the operation environment of the embodiment of the present disclosure is applied to a terminal, and the method includes:

step 301: and performing shallow feature extraction on the low-resolution license plate image to be processed through the first convolution layer to obtain a plurality of shallow feature maps.

Optionally, before the low-resolution image is input into the image processing network, certain processing is required to be performed and a license plate image training library is constructed. The license plate image is acquired through a monitoring video or a picture shot by a mobile phone camera, a license plate area is cut out, the cut image containing the license plate area forms an original high-resolution license plate image, and the low-resolution license plate image to be processed can be obtained by down-sampling the original high-resolution license plate image through an interpolation method.

It should be noted that the parameter setting of the first convolution layer includes: the step size is 1, the convolution kernel size is 3 x 3, and the convolution operation adopts a zero filling mode to keep the consistency of the image resolution. The number of input channels is RGB three-channel images, the number of output shallow feature maps is 64, the feature extraction aims to map the low-resolution license plate image to a feature space from an image space, the feature space is a high-dimensional vector formed by a plurality of vectors, the number of the vectors is equal to the number of the feature maps, and features such as shape edges of the license plate image are mainly extracted.

Step 302: and performing first depth feature extraction on the shallow feature maps through the N identical residual error sampling blocks which are sequentially connected in series to obtain a plurality of local fourth deep feature maps output by the Nth residual error sampling block.

Carrying out scale transformation on a plurality of target characteristic maps through first branches in an ith residual sampling block in N identical residual sampling blocks which are sequentially connected in series to obtain a plurality of local first deep characteristic maps, carrying out characteristic extraction on the plurality of target characteristic maps through second branches in the ith residual sampling block to obtain a plurality of local second deep characteristic maps, and carrying out identity mapping on the plurality of target characteristic maps through third branches in the ith residual sampling block to obtain a plurality of local third deep characteristic maps; performing feature fusion on the plurality of local first deep feature maps, the plurality of local second deep feature maps and the plurality of local third deep feature maps in a one-to-one correspondence manner through the ith residual sampling block to obtain a plurality of local fourth deep feature maps; acquiring a plurality of local fourth deep characteristic maps output by the Nth residual error sampling block; when i is not equal to 1, the target feature maps are local fourth deep feature maps output by the i-1 th residual sampling block.

Optionally, performing scale transformation on the multiple target feature maps through a first branch in an ith residual sample block of N identical residual sample blocks connected in series in sequence to obtain multiple local first deep feature maps, including:

the method comprises the steps that a plurality of target characteristic graphs are up-sampled through an up-sampling layer in a first branch to obtain a plurality of up-sampling characteristic graphs;

Upsampling the plurality of target feature maps by an upsampling layer in the first branch to obtain a plurality of upsampled feature maps, comprising:

the first formula:

a plurality of up-sampling feature maps for the ith residual sample block, phi is a non-linear mapping function or activation function, R, for the ith residual sample block _i-1 For a plurality of target feature maps, W _i And b _i Weights and offsets, respectively, # e @ f, for the ith residual sample block _s Represents an upsampling by a factor of s;

and performing feature extraction on the plurality of upsampled feature maps through a third convolutional layer in the first branch according to a second formula to obtain a plurality of third convolutional feature maps:

the second formula:

wherein the content of the first and second substances,

a plurality of third convolution maps for an ith residual sample block;

the downsampling of the plurality of third convolution feature maps by the downsampling layer in the first branch to obtain a plurality of downsampled feature maps includes:

and downsampling the plurality of third convolution feature maps according to a following third formula through a downsampling layer in the first branch to obtain a plurality of downsampled feature maps:

the third formula:

wherein the content of the first and second substances,

Performing feature extraction on the plurality of target feature maps through a second branch in the ith residual sampling block to obtain a plurality of local second deep feature maps, wherein the method comprises the following steps:

and performing depth feature extraction on the plurality of target feature maps through a second branch in the ith residual error sampling block according to a fourth formula to obtain a plurality of local second deep feature maps:

the fourth formula:

a plurality of local second depth profiles are represented.

Performing feature fusion on the plurality of local first deep feature maps, the plurality of local second deep feature maps and the plurality of local third deep feature maps through the ith residual sampling block to obtain a plurality of local fourth deep feature maps, including:

the fifth formula:

It should be noted that 1, the first branch does not include a batch normalization layer, which is applied in the super-resolution field to cause loss of image pixel value scale information and affect the change of network parameters in a free range, the present disclosure removes the batch normalization layer, and applies computing resources to feature extraction, such as a third convolution layer in the first branch and a fourth convolution layer in the second branch; 2. the first branch mainly comprises an upper sampling layer, a third convolution layer and a lower sampling layer which are connected in sequence, and the scale transformation is performed to be beneficial to enhancing the expression capability of the image processing network and improve the information extracted by the network; the upper sampling layer adopts transposition convolution, and parameters of the transposition convolution are set as follows: the step size is 4, the input and output characteristic graphs are the same and are 64, the convolution kernel size is 6 multiplied by 6, the third convolution layer parameter setting is that the step size is 1, the input and output characteristic graphs are the same and are 64, the convolution kernel size is 3 multiplied by 3, the down-sampling layer adopts convolution, and the parameter setting of the convolution is as follows: the step length is 4, the input characteristic diagram and the output characteristic diagram are the same and are both 64, the size of a convolution kernel is 6 multiplied by 6, and zero padding operation is adopted in the operations; 3. the method has the advantages that the multiple local first deep characteristic maps, the multiple local second deep characteristic maps and the multiple local third deep characteristic maps are subjected to characteristic fusion in a one-to-one correspondence mode through the ith residual sampling block, the multiple local fourth deep characteristic maps are obtained, fusion is carried out, more information can be provided for the reconstruction module, the problem that extracted characteristic information is lost is solved, meanwhile, information flow is increased in the local part, and the situation that training errors caused by gradient disappearance are larger and larger is avoided.

Step 303: and performing second depth feature extraction on a plurality of local fourth deep feature maps output by the Nth residual sampling block through the second convolution layer to obtain a plurality of fifth deep feature maps, and performing feature fusion on the plurality of fifth deep feature maps and the plurality of shallow feature maps in a one-to-one correspondence manner to obtain a plurality of sixth deep feature maps.

Optionally, the image processing network is gradually deepened, which may weaken the backward propagation capability of the network, because the gradient of each layer in the backward propagation is calculated on the basis of the previous layer, the larger the number of layers is, which may cause the gradient to become smaller and smaller during propagation until the gradient disappears, and the larger the training error is, so that feature fusion is performed by one-to-one correspondence between the fifth deep layer feature maps and the shallow layer feature maps, which may increase information flow, further improve the expression capability of the network, and avoid the training error caused by the disappearance of the gradient from becoming larger.

Step 304: and the reconstruction module is used for amplifying the sixth deep feature maps obtained by the feature extraction module by multiple times step by step to obtain a plurality of high-resolution feature maps, and a target high-resolution license plate image is obtained according to the high-resolution feature maps and is obtained by performing super-resolution reconstruction on a low-resolution license plate image through an image processing network.

Optionally, the reconstruction module is configured to amplify the resolutions of the sixth deep feature maps obtained by the feature extraction module by a certain factor, where the step-by-step magnification is performed step by step when the magnification factor is higher, and if the magnification factor is 4, the step may be divided into two steps, where each step amplifies the resolution of the feature map by 2 times. The total number of pixels is unchanged, the pixels in the low-resolution feature map space are recombined and arranged in the high-resolution feature map space according to a certain multiple, the number of the corresponding high-resolution feature maps is reduced according to a certain multiple compared with the number of the low-resolution feature maps before arrangement due to the length and width amplification of the high-resolution feature maps, for example, for an amplification factor of 2, the specific operation of the sub-pixel layer is to convert the spatial positions of 256 feature maps with the resolution of H multiplied by W to obtain 64 feature maps with the resolution of 2H multiplied by 2W; and finally, inputting the multiple high-resolution feature maps amplified by multiple times into a3 x 3 convolutional layer to obtain a three-channel target high-resolution license plate image. Generally, since the number of feature maps is reduced after each pass through the sub-pixel layer, and when step-by-step amplification is performed, the number of feature maps is increased by a convolution layer before the sub-pixel layer to ensure that there is enough feature maps, each step of step-by-step multiple amplification includes two modules: 1. the number of the characteristic graphs is increased according to a certain multiple; 2. a sub-pixel layer.

Step 305: a difference between the original high-resolution license plate image and the target high-resolution license plate image is minimized through a loss function to generate a first high-resolution license plate image.

The original high-resolution license plate image is a license plate image which is not processed through a network, the first high-resolution license plate image is a license plate image which is obtained through the steps that network parameters are continuously updated through a loss function, and when the network parameters approach convergence, the finally obtained super-resolution reconstruction quality is the optimal license plate image.

Optionally, the loss function measures the difference between the predicted value and the true value, and the disclosure employs a minimum mean square error criterion:

the sixth formula:

wherein l ^SR Represents the loss value, r represents the magnification factor, W and H represent the width and height of the low-resolution license plate image respectively,

representing the corresponding pixel points of the original high-resolution license plate image,

and representing corresponding pixel points of the first high-resolution license plate image.

The embodiment of the present disclosure provides a super-resolution reconstruction device for license plate images, referring to fig. 4, the device includes:

the initialization feature extraction module 401 is configured to perform shallow feature extraction on the to-be-processed low-resolution license plate image through the first convolution layer to obtain a plurality of shallow feature maps.

A depth feature extraction module 402, configured to perform first depth feature extraction on the multiple shallow feature maps through N identical residual sampling blocks connected in series in sequence, so as to obtain multiple local fourth deep feature maps output by an nth residual sampling block.

The depth feature extraction module is further configured to perform scale transformation on the multiple target feature maps through a first branch in an ith residual sampling block of the N identical residual sampling blocks connected in series in sequence to obtain multiple local first deep feature maps, perform feature extraction on the multiple target feature maps through a second branch in the ith residual sampling block to obtain multiple local second deep feature maps, and perform identity mapping on the multiple target feature maps through a third branch in the ith residual sampling block to obtain multiple local third deep feature maps; performing feature fusion on the plurality of local first deep feature maps, the plurality of local second deep feature maps and the plurality of local third deep feature maps in a one-to-one correspondence manner through the ith residual sampling block to obtain a plurality of local fourth deep feature maps; acquiring a plurality of local fourth deep characteristic maps output by the Nth residual error sampling block; when i is not equal to 1, the target feature maps are local fourth deep feature maps output by the i-1 th residual sampling block.

The feature fusion module 403 is configured to perform second depth feature extraction on the multiple local fourth deep feature maps output by the nth residual sampling block through the second convolution layer to obtain multiple fifth deep feature maps, and perform feature fusion on the multiple fifth deep feature maps and the multiple shallow feature maps in a one-to-one correspondence manner to obtain multiple sixth deep feature maps.

The super-resolution reconstruction module 404 is configured to perform progressive multiple amplification on the sixth deep feature maps obtained by the feature extraction module through the reconstruction module to obtain a plurality of high-resolution feature maps, and obtain a target high-resolution license plate image according to the plurality of high-resolution feature maps, where the target high-resolution license plate image is a license plate image obtained by performing super-resolution reconstruction on a low-resolution license plate image through an image processing network.

Optionally, the depth feature extraction module 402 includes:

the first sampling submodule is used for performing up-sampling on the plurality of target characteristic graphs through an up-sampling layer in the first branch to obtain a plurality of up-sampling characteristic graphs;

the feature extraction submodule is used for performing feature extraction on the plurality of up-sampling feature maps through a third convolution layer in the first branch to obtain a plurality of third convolution feature maps;

and the second sampling sub-module is used for carrying out downsampling on the plurality of third convolution feature maps through a downsampling layer in the first branch to obtain a plurality of downsampling feature maps, and determining the plurality of downsampling feature maps into a plurality of local first deep feature maps, wherein the up-sampling adopts transposed convolution and the downsampling adopts convolution.

Optionally, the first sampling sub-module performs upsampling on the multiple target feature maps through an upsampling layer in the first branch according to a first formula as follows to obtain multiple upsampled feature maps:

the first formula:

wherein the content of the first and second substances,

a plurality of up-sampling feature maps for the ith residual sample block, phi is a non-linear mapping function or activation function, R, of the ith residual sample block _i-1 For a plurality of target feature maps, W _i And b _i Weights and offsets, respectively, # e @ f, for the ith residual sample block _s Represents the upsampling by a factor of s;

the feature extraction sub-module is configured to perform feature extraction on the multiple upsampled feature maps through the third convolution layer in the first branch according to a second formula as follows to obtain multiple third convolution feature maps:

the second formula:

a plurality of third convolutions for the ith residual sample blockA feature map;

the second sampling submodule is used for downsampling the plurality of third convolution feature maps according to a following third formula through a downsampling layer in the first branch to obtain a plurality of downsampling feature maps:

the third formula:

a plurality of down-sampled feature maps, ↓, for the i-th residual sample block _s Is s times the down sampling.

Optionally, the depth feature extraction module is further configured to perform feature extraction on the multiple target feature maps through a second branch in the ith residual sampling block according to a fourth formula as follows to obtain multiple local second deep feature maps:

the fourth formula:

wherein the content of the first and second substances,

a plurality of local second depth profiles are represented.

Optionally, the depth feature extraction module is configured to perform feature fusion on the multiple local first deep feature maps, the multiple local second deep feature maps, and the multiple local third deep feature maps through the ith residual sampling block according to a fifth formula as follows to obtain multiple local fourth deep feature maps:

the fifth formula:

multiple local second deep feature maps, R, obtained for a second branch in an ith residual sample block _i-1 A plurality of local third deep feature maps for a third branch of the ith residual sample block.

In summary, in the embodiment of the present disclosure, shallow feature extraction is performed on a to-be-processed low-resolution license plate image through a first convolution layer to obtain a plurality of shallow feature maps, first depth feature extraction is performed on the plurality of shallow feature maps through N residual sampling blocks connected in series in sequence to obtain a plurality of local fourth deep feature maps output by an nth residual sampling block, and then, based on the plurality of local fourth deep feature maps, a target high-resolution license plate image is finally obtained through a second convolution layer and a reconstruction module. Each residual sampling block comprises three branches, so that when depth feature extraction is carried out for the first time, a plurality of target feature maps can be processed through the three branches of each residual sampling block, feature fusion is carried out on a plurality of local first deep feature maps, a plurality of local second deep feature maps and a plurality of local third deep feature maps which are obtained through processing in a one-to-one correspondence mode, a plurality of local fourth deep feature maps are obtained, and then a plurality of local fourth deep feature maps output by the Nth residual sampling blocks are obtained. Because the first branch includes the upsampling layer, the third convolution layer and the downsampling layer that connect gradually, can increase the detailed information of characteristic map, the second branch fully draws the characteristic of low resolution characteristic map through the fourth convolution layer, the third branch can avoid appearing the gradient disappearance and lead to rebuilding the poor problem of effect among the reconstruction process, consequently, carry out the feature fusion with a plurality of local deep characteristic maps that these three branches obtained and can provide more information for the reconstruction module, the problem of the characteristic information loss of having solved the extraction makes the license plate of rebuilding avoid the characters fuzzy, the character adhesion, the condition such as information is not abundant, the degree of discerning of license plate image has been improved.

The disclosed embodiment provides a super-resolution reconstruction method and device for a license plate image, and the method and device are applied to a terminal, wherein at least one instruction, at least one section of program, a code set or an instruction set is stored in a computer-readable storage medium, and the instruction, the program, the code set or the instruction set is loaded and executed by a processor to implement the operations executed by the terminal in the above-mentioned license plate image super-resolution reconstruction method.

Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosed invention. This application is intended to cover any variations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains.

It should be understood that the present disclosure is not limited to the precise construction and arrangements shown and described above, and that various modifications, equivalent substitutions, improvements, etc. can be made without departing from the spirit and scope of the present disclosure.

Claims

1. The super-resolution reconstruction method of the license plate image is characterized by being applied to a terminal, wherein an image processing network is deployed on the terminal and comprises a feature extraction module and a reconstruction module, the feature extraction module comprises an initialization feature extraction module and a depth feature extraction module which are connected, the initialization feature extraction module comprises a first convolution layer, the depth feature extraction module comprises N identical residual sampling blocks and a second convolution layer which are sequentially connected in series, N is an integer greater than or equal to 1, each residual sampling block comprises a first branch, a second branch and a third branch, the first branch comprises an upper sampling layer, a third convolution layer and a lower sampling layer which are sequentially connected, the second branch comprises a fourth convolution layer, the third branch is an identity mapping branch, an output feature map of the identity mapping branch is identical to an input feature map, and the method comprises the following steps:

amplifying the sixth deep feature maps obtained by the feature extraction module by multiple times step by step through the reconstruction module to obtain a plurality of high-resolution feature maps, and obtaining a target high-resolution license plate image according to the high-resolution feature maps, wherein the target high-resolution license plate image is a license plate image obtained by performing super-resolution reconstruction on the low-resolution license plate image through the image processing network;

carrying out scale transformation on a plurality of target feature maps through a first branch in an ith residual sampling block in the N identical residual sampling blocks which are sequentially connected in series to obtain a plurality of local first deep feature maps, carrying out feature extraction on the plurality of target feature maps through a second branch in the ith residual sampling block to obtain a plurality of local second deep feature maps, and carrying out identity mapping on the plurality of target feature maps through a third branch in the ith residual sampling block to obtain a plurality of local third deep feature maps; performing feature fusion on the plurality of local first deep feature maps, the plurality of local second deep feature maps and the plurality of local third deep feature maps in a one-to-one correspondence manner through the ith residual sampling block to obtain a plurality of local fourth deep feature maps; acquiring a plurality of local fourth deep feature maps output by the Nth residual error sampling block; when i is not equal to 1, the target feature maps are local fourth deep feature maps output by the i-1 th residual sample block.

2. The method of claim 1, wherein scaling the plurality of target feature maps through the first branch in an ith residual sample block of the N identical residual sample blocks connected in series in sequence to obtain a plurality of local first deep feature maps comprises:

and downsampling the plurality of third convolution feature maps through a downsampling layer in the first branch to obtain a plurality of downsampling feature maps, and determining the plurality of downsampling feature maps as the plurality of local first deep feature maps, wherein the upsampling adopts transposition convolution, and the downsampling adopts convolution.

3. The method of claim 2, wherein the upsampling the plurality of target feature maps by the upsampling layer in the first branch to obtain a plurality of upsampled feature maps, comprises:

the first formula:

wherein the content of the first and second substances,

a plurality of up-sampling feature maps for the ith residual sample block, phi is a non-linear mapping function or activation function, R, for the ith residual sample block _i-1 For said plurality of target feature maps, W _i And b _i Weight and bias of the ith residual sample block, # r ≠ f _s Represents an upsampling by a factor of s;

performing feature extraction on the plurality of upsampled feature maps through a third convolutional layer in the first branch according to a second formula as follows to obtain a plurality of third convolutional feature maps:

the second formula:

wherein the content of the first and second substances,

a plurality of third convolution maps for an ith residual sample block;

the third formula:

a plurality of down-sampling feature maps for the ith residual sample block, ↓ s is down-sampling sAnd (4) multiplying.

4. The method of claim 1, wherein said feature extracting the plurality of target feature maps through a second branch in the i-th residual sample block to obtain a plurality of local second deep feature maps comprises:

performing depth feature extraction on the plurality of target feature maps through a second branch in the ith residual sampling block according to a fourth formula to obtain a plurality of local second deep feature maps:

the fourth formula:

a plurality of local second depth profiles are represented.

5. The method of claim 1, wherein feature fusing the plurality of local first, second, and third deep feature maps by the ith residual sample block to obtain a plurality of local fourth deep feature maps comprises:

the fifth formula:

6. The utility model provides a super-resolution of license plate image rebuilds device which characterized in that is applied to the terminal, the terminal is last to be deployed and has been had the image processing network, the image processing network includes feature extraction module and rebuild the module, the feature extraction module is including the initialized feature extraction module and the depth feature extraction module of connecting, the initialized feature extraction module includes first volume layer, the depth feature extraction module is including N the same residual error sampling block and a second volume layer that connect gradually in series, N is the integer that is more than or equal to 1, and each residual error sampling block includes first branch, second branch and third branch, first branch is including the upsampling layer, third volume layer and the downsampling layer that connect gradually, the second branch includes the fourth volume layer, the third branch is the mapping branch of identity, the output characteristic map of mapping branch is the same with the input characteristic map, the device includes:

the depth feature extraction module is used for performing first depth feature extraction on the shallow feature maps through the N identical residual error sampling blocks which are sequentially connected in series to obtain a plurality of local fourth deep feature maps output by the Nth residual error sampling block;

the depth feature extraction module is further configured to perform scale transformation on multiple target feature maps through a first branch in an ith residual sampling block of the N identical residual sampling blocks connected in series in sequence to obtain multiple local first deep feature maps, perform feature extraction on the multiple target feature maps through a second branch in the ith residual sampling block to obtain multiple local second deep feature maps, and perform identity mapping on the multiple target feature maps through a third branch in the ith residual sampling block to obtain multiple local third deep feature maps; performing feature fusion on the plurality of local first deep feature maps, the plurality of local second deep feature maps and the plurality of local third deep feature maps in a one-to-one correspondence manner through the ith residual sampling block to obtain a plurality of local fourth deep feature maps; acquiring a plurality of local fourth deep feature maps output by the Nth residual error sampling block; when i is equal to 1, the plurality of target feature maps are the plurality of shallow feature maps, and when i is not equal to 1, the plurality of target feature maps are a plurality of local fourth deep feature maps output by the i-1 th residual sampling block.

7. The apparatus of claim 6, wherein the depth feature extraction module comprises:

the first sampling submodule is used for performing upsampling on the plurality of target characteristic graphs through an upsampling layer in the first branch to obtain a plurality of upsampling characteristic graphs;

8. The apparatus of claim 7,

the first sampling sub-module is configured to, through an upsampling layer in the first branch, upsample the multiple target feature maps according to a first formula as follows to obtain multiple upsampled feature maps:

the first formula:

a plurality of up-sampling feature maps for the ith residual sample block, phi is a non-linear mapping function or activation function, R, of the ith residual sample block _i-1 For said plurality of target feature maps, W _i And b _i Weight and bias of the ith residual sample block, # r ≠ f _s Represents the upsampling by a factor of s;

the feature extraction sub-module is configured to perform feature extraction on the plurality of upsampled feature maps through a third convolution layer in the first branch according to a second formula as follows to obtain a plurality of third convolution feature maps:

the second formula:

wherein the content of the first and second substances,

a plurality of third convolution maps for an ith residual sample block;

the third formula:

9. The apparatus of claim 6,

the depth feature extraction module is further configured to perform feature extraction on the multiple target feature maps through a second branch in the ith residual sampling block according to a fourth formula as follows to obtain multiple local second deep feature maps:

the fourth formula:

wherein the content of the first and second substances,

a plurality of local second depth profiles are represented.

10. The apparatus of claim 6,

the depth feature extraction module is configured to perform feature fusion on the multiple local first deep feature maps, the multiple local second deep feature maps, and the multiple local third deep feature maps according to a fifth formula below by using the ith residual sampling block to obtain multiple local fourth deep feature maps:

the fifth formula: