CN116029905A - Face super-resolution reconstruction method and system based on progressive difference complementation - Google Patents
Face super-resolution reconstruction method and system based on progressive difference complementation Download PDFInfo
- Publication number
- CN116029905A CN116029905A CN202310089839.6A CN202310089839A CN116029905A CN 116029905 A CN116029905 A CN 116029905A CN 202310089839 A CN202310089839 A CN 202310089839A CN 116029905 A CN116029905 A CN 116029905A
- Authority
- CN
- China
- Prior art keywords
- resolution
- image
- face
- low
- super
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Image Processing (AREA)
Abstract
The invention discloses a face super-resolution reconstruction method and a face super-resolution reconstruction system based on progressive difference complementation, wherein the method comprises the following steps: collecting a monitoring video to be detected, and extracting a low-resolution face image of a pedestrian from the monitoring video; preprocessing the low-resolution face image; inputting the preprocessed low-resolution image into a progressive difference complementary face super-resolution reconstruction network, outputting a high-low-resolution characteristic image group through coarse processing in a first stage after the low-resolution face image is input into the network, and outputting a final high-resolution characteristic image through difference complementation in a second stage; and carrying out super-resolution reconstruction on the output final high-resolution characteristic image, carrying out pixel-by-pixel addition on the final high-resolution characteristic image and an image output by up-sampling the final high-resolution characteristic image through bicubic interpolation with a low-resolution face image, outputting a super-resolution face reconstruction image, and reducing the complexity, parameter quantity and calculated quantity of a reconstruction network while realizing a better reconstruction effect.
Description
Technical Field
The invention relates to the technical field of computer vision, in particular to a face super-resolution reconstruction method and system based on progressive difference complementation.
Background
The statements in this section merely provide background information related to the present disclosure and may not necessarily constitute prior art.
The image super-resolution reconstruction refers to reconstructing an input low-resolution image, that is, an image with a relatively small number of pixels and poor visual effect, into a clear image with a relatively large number of pixels and good visual effect. The reconstruction of the super-resolution of the human face is a subtask of the reconstruction of the super-resolution of the image, which is similar to but different from the common reconstruction of the super-resolution of the image, and the super-resolution of the human face is focused on the reconstruction of the facial features and the contours of the human face as the name implies, so that the reconstruction of the global detail and the local detail is performed to different degrees. The face super-resolution reconstruction has wide application, and because the performance of the public monitoring platform equipment is limited, the collected face is often a low-resolution face image with serious degradation, the face super-resolution reconstruction is needed to be carried out, namely the face super-resolution reconstruction is carried out on the face, the resolution of the reconstructed face image is greatly improved, and the five sense organs are clear and discernable, so that the face super-resolution reconstruction has great effect on the subsequent tasks of face analysis, face alignment, face recognition and the like. The traditional methods of early face super-resolution reconstruction mainly comprise two methods: image interpolation-based methods, such as bicubic interpolation, nearest neighbor interpolation, etc., but the reconstructed image by this method has the problem of detail loss; methods based on image reconstruction, such as iterative back projection and maximum a posteriori methods, have small algorithm models and high calculation speeds, but have limited reconstruction performance. With the advent of the deep learning era, the convolutional neural network has made breakthrough progress in super-resolution reconstruction, and the image super-resolution reconstruction field has raised the hot tide of the neural network.
The inventors have found that the image super-resolution reconstruction differs from the face super-resolution reconstruction in the size of the reconstructed object in addition to the reconstruction emphasis set forth above. Image super-resolution tends to be a reconstruction of a picture of larger size, for example, a magnification of 2, 3 or 4 times for a 128 x 128 image; while the size of the reconstruction object for super-resolution reconstruction of faces is much smaller, as in recent years, current research has focused on 8-fold reconstruction of 16×16 ultra-small faces. That is, considering that the low-resolution face image contains little information and lacks high-frequency information, the existing image super-resolution reconstruction method cannot be suitable for face super-resolution reconstruction, which requires a deeper network to learn the feature information of the input face image, so as to construct a high-resolution image with better effect.
Disclosure of Invention
In order to solve the defects in the prior art, the invention provides a face super-resolution reconstruction method and a face super-resolution reconstruction system based on progressive difference complementation, which are used for carrying out two-stage processing on a low-resolution face image, constructing a high-low resolution characteristic image group input to a second stage through coarse processing of the first stage, extracting more characteristic information from the high-low resolution characteristic image group through difference complementation of the second stage, fusing shallow characteristics of the image and high-layer characteristic information together, avoiding waste of image information, and realizing a better reconstruction effect while reducing the complexity, the parameter quantity and the calculated quantity of a reconstruction network.
In a first aspect, the present disclosure provides a face super-resolution reconstruction method based on progressive difference complementation.
A face super-resolution reconstruction method based on progressive difference complementation comprises the following steps:
collecting a monitoring video to be detected, and extracting a low-resolution face image of a pedestrian from the monitoring video;
preprocessing the low-resolution face image;
inputting the preprocessed low-resolution image into a progressive difference complementary face super-resolution reconstruction network, outputting a high-low-resolution characteristic image group through coarse processing in a first stage after the low-resolution face image is input into the network, and outputting a final high-resolution characteristic image through difference complementation in a second stage;
and carrying out super-resolution reconstruction on the output final high-resolution characteristic image, carrying out pixel-by-pixel addition on the final high-resolution characteristic image and an image output by double cubic interpolation up-sampling of the final high-resolution characteristic image and the low-resolution face image, and outputting a super-resolution face reconstruction image.
According to a further technical scheme, the preprocessing process comprises the steps of firstly cutting out a low-resolution face image, and then randomly rotating by 90 degrees, 180 degrees, 270 degrees and horizontally turning over to enhance data.
In a further technical scheme, in a coarse processing stage, a low-resolution face image is input into a coarse processing module of a reconstruction network, wherein the coarse processing module comprises two branches, namely a multiple convolution branch and a sub-pixel convolution branch;
the sub-pixel convolution branch consists of 1 multiplied by 1 convolution, an activation function and a sub-pixel convolution layer and is used for outputting a low-resolution characteristic image based on an input low-resolution face image; the multiple convolution branches are formed by serially connecting 1×1 convolution, a sub-pixel convolution layer, n 3×3 convolutions and 1×1 convolution, and are used for outputting a high-resolution characteristic image based on an input low-resolution face image.
In a further technical scheme, in a difference complementation stage, a high-low resolution characteristic image group is input into a difference complementation module of a reconstruction network;
in the difference complementation module, firstly, a multi-core residual error feature extraction module is used for extracting feature information of a high-low resolution feature image group, carrying out difference on the extracted feature information of the high-low resolution feature image group and the extracted feature information of the high-low resolution feature image group, respectively adding a difference result to the extracted feature images, and outputting an updated high-low resolution feature image group; and then taking the output high-low resolution characteristic image group as input, repeating the next difference value complementation operation, and outputting the final high-resolution characteristic image after repeating the difference value complementation operation for m times.
According to a further technical scheme, the multi-core residual feature extraction module comprises 3×3 convolutions, 5×5 convolutions and 7×7 convolutions which are connected in parallel and are used for respectively extracting feature information of an input image, and each convolution layer adopts a residual structure; and cascading the extracted characteristic information of different receptive fields with the input original characteristic image, and inputting the cascaded characteristic information into 1 3 multiplied by 3 convolution to obtain the characteristic information of the original characteristic image.
According to a further technical scheme, the training process of the progressive difference complementary face super-resolution reconstruction network comprises the following steps of:
acquiring a monitoring video, extracting a low-resolution face image of a pedestrian from the monitoring video, and simultaneously acquiring a high-resolution face image of the pedestrian, wherein the corresponding high-resolution face image and the corresponding low-resolution face image are used as a training sample set;
respectively preprocessing high-low resolution face images in a training sample set;
training a progressive difference value complementary face super-resolution reconstruction network by utilizing the preprocessed training sample set; the progressive difference complementary face super-resolution reconstruction network comprises two stages, after a low-resolution face image is input into the network, a high-low resolution feature image group is output through coarse processing in a first stage, a final high-resolution feature image is output through difference complementation in a second stage, the image and the low-resolution face image are added pixel by pixel through images output through bicubic interpolation up-sampling, a super-resolution face reconstruction image is output, and the network is trained based on the super-resolution face reconstruction image and the high-resolution face image in a training sample set.
According to the further technical scheme, pixel loss between the super-resolution face reconstruction image and the high-resolution face image is calculated through an L1 loss function, so that a reconstruction network is optimized; the loss function calculation formula is as follows:
wherein L is Pixel Indicating total loss of network, I HR And I SR Respectively representing a high-resolution face image and a super-resolution face reconstruction image.
In a second aspect, the present disclosure provides a face super-resolution reconstruction system based on progressive difference complementation.
A face super-resolution reconstruction system based on progressive difference complementation, comprising:
the face image acquisition module is used for acquiring a monitoring video to be detected and extracting a low-resolution face image of a pedestrian from the monitoring video;
the image processing module is used for preprocessing the low-resolution face image;
the image feature extraction module is used for inputting the preprocessed low-resolution image into a progressive difference complementary face super-resolution reconstruction network, outputting a high-low-resolution feature image group through coarse processing in a first stage after the low-resolution face image is input into the network, and outputting a final high-resolution feature image through difference complementation in a second stage after the high-low-resolution feature image group is input into the network;
and the image reconstruction module is used for carrying out super-resolution reconstruction on the output final high-resolution characteristic image, carrying out pixel-by-pixel addition on the final high-resolution characteristic image and the image output by double-cubic interpolation up-sampling of the final high-resolution characteristic image and the low-resolution face image, and outputting a super-resolution face reconstruction image.
In a third aspect, the present disclosure also provides an electronic device comprising a memory and a processor, and computer instructions stored on the memory and running on the processor, which when executed by the processor, perform the steps of the method of the first aspect.
In a fourth aspect, the present disclosure also provides a computer readable storage medium storing computer instructions which, when executed by a processor, perform the steps of the method of the first aspect.
The one or more of the above technical solutions have the following beneficial effects:
1. the invention provides a face super-resolution reconstruction method and a face super-resolution reconstruction system based on progressive difference complementation, which solve the problems of poor face super-resolution reconstruction effect and low efficiency in a monitoring video by utilizing a progressive difference complementation face super-resolution reconstruction network, perform two-stage processing on a low-resolution face image, construct a high-low-resolution characteristic image group input into a second stage through multiple convolution kernel sub-pixel convolution of a coarse processing module in the first stage, initially extract image characteristics, provide priori information for the second stage, extract more characteristic information from the high-low-resolution characteristic image group through difference complementation of the second stage, fuse shallow-layer characteristics and high-layer characteristic information of the image together, avoid the waste of the image information, and realize a better reconstruction effect while reducing the complexity, the parameter quantity and the calculated quantity of the reconstruction network.
2. In the invention, in the multi-core residual error feature extraction module of the second stage, the feature extraction is performed in a parallel mode by using convolution kernels with different sizes, so that richer receptive field information can be obtained, the waste of image information is avoided, and a high-resolution image with better construction effect is realized.
3. The face super-resolution reconstruction method based on progressive difference complementation can effectively provide help for face recognition and the like, and can solve the problems of high network complexity, low reconstruction rate, large calculation amount and the like caused by using a deep network for recovering better reconstruction quality.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description serve to explain the invention.
FIG. 1 is a flowchart of a face super-resolution reconstruction method based on progressive difference complementation in an embodiment of the invention;
fig. 2 is a schematic diagram of a face super-resolution reconstruction method based on progressive difference complementation in an embodiment of the present invention;
FIG. 3 is a schematic diagram of a coarse processing module according to an embodiment of the present invention;
fig. 4 is a schematic diagram of a multi-core residual feature extraction module in an embodiment of the present invention.
Detailed Description
It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the invention. Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.
It is noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of exemplary embodiments according to the present invention. As used herein, the singular is also intended to include the plural unless the context clearly indicates otherwise, and furthermore, it is to be understood that the terms "comprises" and/or "comprising" when used in this specification are taken to specify the presence of stated features, steps, operations, devices, components, and/or combinations thereof.
Term interpretation:
feature map: a feature map with three-dimensional information of width W, height H and channel number C;
sub-pixel convolution: namely, the pixel reorganization, the main function is to obtain a high-resolution characteristic map through convolution and reorganization among multiple channels for the low-resolution characteristic map.
Example 1
The embodiment provides a face super-resolution reconstruction method based on progressive difference complementation, which is applied to the fields of face recognition, face analysis and the like and comprises the following steps:
collecting a monitoring video to be detected, and extracting a low-resolution face image of a pedestrian from the monitoring video;
preprocessing the low-resolution face image;
inputting the preprocessed low-resolution image into a progressive difference complementary face super-resolution reconstruction network, outputting a high-low-resolution characteristic image group through coarse processing in a first stage after the low-resolution face image is input into the network, and outputting a final high-resolution characteristic image through difference complementation in a second stage;
and carrying out super-resolution reconstruction on the output final high-resolution characteristic image, carrying out pixel-by-pixel addition on the final high-resolution characteristic image and an image output by double cubic interpolation up-sampling of the final high-resolution characteristic image and the low-resolution face image, and outputting a super-resolution face reconstruction image.
The training process of the progressive difference complementary face super-resolution reconstruction network is shown in fig. 1, and comprises the following steps:
step S1, acquiring a monitoring video, extracting a low-resolution face image of a pedestrian from the monitoring video, and simultaneously acquiring a high-resolution face image of the pedestrian, wherein the high-resolution face image and the low-resolution face image which correspond to each other are used as a training sample set;
s2, respectively preprocessing high-low resolution face images in the training sample set;
s3, training the progressive difference complementary face super-resolution reconstruction network by utilizing the preprocessed training sample set; the progressive difference complementary face super-resolution reconstruction network comprises two stages, after a low-resolution face image is input into the network, a high-low resolution feature image group is output through coarse processing in a first stage, a final high-resolution feature image is output through difference complementation in a second stage, the image and the low-resolution face image are added pixel by pixel through images output through bicubic interpolation up-sampling, a super-resolution face reconstruction image is output, and the network is trained based on the super-resolution face reconstruction image and the high-resolution face image.
In this embodiment, in the step S1, the monitoring video is acquired and obtained, and the low-resolution face image of the pedestrian is extracted from the monitoring video, and in consideration of the fact that the monitoring video is formed by a series of rapidly changing frame images, the same pedestrian may appear in tens of frame images, therefore, by adopting the line acquisition method, the low-resolution image of the pedestrian is acquired when the pedestrian passes through the line defined in the video, and the low-resolution face image is obtained by the clipping operation. The acquired image is named for any scale, so that a training sample set of the low-resolution face image is formed. Furthermore, the acquired images can be divided into a training sample set and a testing sample set of the low-resolution face images according to the proportion of 9:1 after being named, so that the subsequent testing of the trained network is facilitated.
Meanwhile, the high-resolution face images of the pedestrians are obtained, the corresponding low-resolution face images are named, a training sample set of the high-resolution face images is formed, namely, the corresponding high-resolution face images and the corresponding low-resolution face images are named with the same name, and then the corresponding high-resolution face images and the corresponding low-resolution face images are respectively stored into the high-resolution face image training set to serve as the training sample set. Furthermore, the acquired images can be divided into a training sample set and a testing sample set of the high-resolution face images according to the proportion of 9:1 after being named, so that the subsequent testing of the trained network is facilitated.
The training sample set of the high-low resolution face image is obtained through the steps, the low-resolution face image in the training sample set is input into the constructed reconstruction network to reconstruct, the pixel loss value of the high-resolution face image in the training sample set and the reconstructed super-resolution face image is calculated, so that the parameters of all layers of convolution layers in the reconstruction network are optimized, and the optimal reconstruction network related parameters are obtained through training.
In the step S2, the high-resolution face image and the low-resolution face image in the training sample set are preprocessed respectively, and the preprocessing includes:
step S2.1, respectively unifying the sizes of the high-resolution face images and the low-resolution face images; the low-resolution face image is unified to be 16 multiplied by 16 by using a bicubic interpolation method, and the high-resolution face image is unified to be 128 multiplied by 128 by using a bicubic interpolation method;
and S2.2, cutting the low-resolution face image, randomly rotating by 90 degrees, 180 degrees, 270 degrees and horizontally turning over to enhance the data, and expanding the training sample set data.
In the step S3, the training of the progressive difference complementary face super-resolution reconstruction network is performed by using the preprocessed training sample set.
In this embodiment, the progressive difference complementary face super-resolution reconstruction network mainly includes two stages, a first stage is a coarse processing stage, and a second stage is a difference complementary stage. Specifically, as shown in fig. 2, after a 16×16 low-resolution face image is input into the network, in a first stage, image processing is performed by multiple convolution and sub-pixel convolution of a coarse processing module, so as to obtain a high-low-resolution characteristic image group; then, in the second stage, respectively extracting the characteristic information of the high-resolution and low-resolution characteristic image groups through a multi-core residual error characteristic extraction module in a difference complementation module, differencing the extracted characteristic information, and outputting a final high-resolution characteristic image through difference complementation; and finally, up-sampling the final high-resolution characteristic image and the low-resolution face image to 128×128 images through bicubic interpolation, adding pixels by pixels, outputting a super-resolution face reconstruction image, and training the reconstruction network based on the super-resolution face reconstruction image and the high-resolution face image.
In the coarse processing stage, the 16×16 low resolution face image is input to a coarse processing module, which, as shown in fig. 3, includes two branches, a multiple convolution branch and a sub-pixel convolution branch, respectively. The sub-pixel convolution branch consists of 1 multiplied by 1 convolution, an activation function and a sub-pixel convolution layer and is used for outputting a low-resolution characteristic image based on an input low-resolution face image; the multiple convolution branches are formed by serially connecting 1×1 convolution, a sub-pixel convolution layer, n 3×3 convolutions and 1×1 convolution, and are used for outputting a high-resolution characteristic image based on an input low-resolution face image.
In the difference complementation stage, the high-low resolution characteristic image group is input into a difference complementation module, firstly, the characteristic information of the high-low resolution characteristic image group is extracted through a multi-core residual characteristic extraction module (FE), the extracted characteristic information of the high-low resolution characteristic image group and the extracted characteristic information of the high-low resolution characteristic image group are subjected to difference, the difference result is respectively added to the extracted characteristic images, the updated high-low resolution characteristic image group is output, then the output high-low resolution characteristic image group is taken as input, next difference complementation operation is repeatedly carried out, and after m difference complementation operations are circularly repeated, the final high-resolution characteristic image is output.
It is noted that the dimensions of all feature maps are fixed at 128×128×48 in this stage. Since the size of the high-resolution feature image obtained after the coarse processing stage is 128×128×48, the size of the low-resolution image is 32×32×48, and the FE operation does not change the size of the image, if the sizes are not unified, the subsequent difference complementary operation cannot be performed, so in this embodiment, after the first multi-core residual feature extraction module FE for extracting the feature information of the low-resolution feature image, a deconvolution operation is added, so that the sizes of the high-resolution feature image and the low-resolution feature image are unified. The subsequent deconvolution operation is not required because the sizes of the high and low resolution feature image groups are uniform.
The multi-core residual feature extraction module is shown in fig. 4, and comprises 3×3 convolutions, 5×5 convolutions and 7×7 convolutions which are connected in parallel and are used for respectively extracting feature information from an input image, on the basis, each convolution layer adopts a residual structure so as to further improve network performance, different receptive field information can be obtained through multi-core use, and finally, the extracted feature information of different receptive fields is input into 1 3×3 convolutions after being cascaded with the input original feature image, so that the feature information of the original feature image is obtained. In addition, a ReLU layer is added after each convolution layer for increasing the nonlinear expression capacity. It is noted that the dimensions of all feature maps are fixed at 128×128×48 in this stage.
Finally, the final high-resolution characteristic image and the low-resolution face image are output, the images which are up-sampled to 128 multiplied by 128 through bicubic interpolation are added pixel by pixel, a super-resolution face reconstruction image is output, the reconstruction network is trained based on the super-resolution face reconstruction image and the high-resolution face image, specifically, the pixel loss between the super-resolution face reconstruction image and the high-resolution face image is calculated through an L1 loss function, so that the reconstruction network is optimized, and the loss function calculation formula is as follows:
wherein L is Pixel Indicating total loss of network, I HR And I SR Respectively representing a high-resolution face image and a super-resolution face reconstruction image.
After training of the progressive difference complementary face super-resolution reconstruction network is completed, extracting a low-resolution face image of a pedestrian in a monitored video to be detected, preprocessing the low-resolution face image, including cutting the low-resolution face image, randomly rotating 90 degrees, 180 degrees, 270 degrees, horizontally turning over to enhance data, inputting the preprocessed low-resolution image into the trained progressive difference complementary face super-resolution reconstruction network, and finally outputting a super-resolution face reconstruction image.
As another implementation mode, the low-resolution face image in the test sample set can be directly selected and input into a training-completed progressive difference complementary face super-resolution reconstruction network for reconstruction, and a super-resolution face reconstruction image is obtained.
The method and the device solve the problems of poor face super-resolution reconstruction effect and low efficiency in a monitoring video by utilizing a progressive difference complementary face super-resolution reconstruction network, extract low-resolution face images from the monitoring video, uniformly scale and name the low-resolution face images, input the low-resolution face images into the reconstruction network for two-stage reconstruction, wherein in the first stage, a high-low resolution characteristic image group required by the second stage is constructed through multiple convolution and sub-pixel convolution of a coarse processing module, image characteristics are initially extracted, and prior information is provided for the second stage; in the second stage, the shallow layer characteristic and the high layer characteristic information of the image are fused together by a difference complementation method, so that the waste of the image information is avoided, and in the multi-core residual characteristic extraction module proposed in the second stage, the characteristic extraction is performed in a parallel mode by using convolution kernels with different sizes, so that richer receptive field information can be obtained.
The face super-resolution reconstruction method based on progressive difference complementation can effectively provide assistance for face recognition and the like, and can solve the problems of high network complexity, low reconstruction rate, large calculation amount and the like caused by using a deep network for recovering better reconstruction quality.
Example two
The embodiment provides a face super-resolution reconstruction system based on progressive difference complementation, which comprises:
the face image acquisition module is used for acquiring a monitoring video to be detected and extracting a low-resolution face image of a pedestrian from the monitoring video;
the image processing module is used for preprocessing the low-resolution face image;
the image feature extraction module is used for inputting the preprocessed low-resolution image into a progressive difference complementary face super-resolution reconstruction network, outputting a high-low-resolution feature image group through coarse processing in a first stage after the low-resolution face image is input into the network, and outputting a final high-resolution feature image through difference complementation in a second stage after the high-low-resolution feature image group is input into the network;
and the image reconstruction module is used for carrying out super-resolution reconstruction on the output final high-resolution characteristic image, carrying out pixel-by-pixel addition on the final high-resolution characteristic image and the image output by double-cubic interpolation up-sampling of the final high-resolution characteristic image and the low-resolution face image, and outputting a super-resolution face reconstruction image.
Example III
The embodiment provides an electronic device, which comprises a memory, a processor and computer instructions stored on the memory and running on the processor, wherein the computer instructions complete the steps in the face super-resolution reconstruction method based on progressive difference complementation.
Example IV
The present embodiment also provides a computer-readable storage medium storing computer instructions that, when executed by a processor, perform the steps in a face super-resolution reconstruction method based on progressive difference complementation as described above.
The steps involved in the second to fourth embodiments correspond to the first embodiment of the method, and the detailed description of the second embodiment refers to the relevant description of the first embodiment. The term "computer-readable storage medium" should be taken to include a single medium or multiple media including one or more sets of instructions; it should also be understood to include any medium capable of storing, encoding or carrying a set of instructions for execution by a processor and that cause the processor to perform any one of the methods of the present invention.
It will be appreciated by those skilled in the art that the modules or steps of the invention described above may be implemented by general-purpose computer means, alternatively they may be implemented by program code executable by computing means, whereby they may be stored in storage means for execution by computing means, or they may be made into individual integrated circuit modules separately, or a plurality of modules or steps in them may be made into a single integrated circuit module. The present invention is not limited to any specific combination of hardware and software.
The above description is only of the preferred embodiments of the present invention and is not intended to limit the present invention, but various modifications and variations can be made to the present invention by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
While the foregoing description of the embodiments of the present invention has been presented in conjunction with the drawings, it should be understood that it is not intended to limit the scope of the invention, but rather, it is intended to cover all modifications or variations within the scope of the invention as defined by the claims of the present invention.
Claims (10)
1. The face super-resolution reconstruction method based on progressive difference complementation is characterized by comprising the following steps of:
collecting a monitoring video to be detected, and extracting a low-resolution face image of a pedestrian from the monitoring video;
preprocessing the low-resolution face image;
inputting the preprocessed low-resolution image into a progressive difference complementary face super-resolution reconstruction network, outputting a high-low-resolution characteristic image group through coarse processing in a first stage after the low-resolution face image is input into the network, and outputting a final high-resolution characteristic image through difference complementation in a second stage;
and carrying out super-resolution reconstruction on the output final high-resolution characteristic image, carrying out pixel-by-pixel addition on the final high-resolution characteristic image and an image output by double cubic interpolation up-sampling of the final high-resolution characteristic image and the low-resolution face image, and outputting a super-resolution face reconstruction image.
2. The face super-resolution reconstruction method based on progressive difference complementation as claimed in claim 1, wherein the preprocessing process includes first performing a cropping operation on the low-resolution face image, and then randomly rotating by 90 °, 180 °, 270 ° and horizontally flipping for data enhancement.
3. The face super-resolution reconstruction method based on progressive difference complementation of claim 1, wherein in the coarse processing stage, the low-resolution face image is input into a coarse processing module of a reconstruction network, and the coarse processing module comprises two branches, namely a multiple convolution branch and a sub-pixel convolution branch;
the sub-pixel convolution branch consists of 1 multiplied by 1 convolution, an activation function and a sub-pixel convolution layer and is used for outputting a low-resolution characteristic image based on an input low-resolution face image; the multiple convolution branches are formed by serially connecting 1×1 convolution, a sub-pixel convolution layer, n 3×3 convolutions and 1×1 convolution, and are used for outputting a high-resolution characteristic image based on an input low-resolution face image.
4. The face super-resolution reconstruction method based on progressive difference complementation according to claim 1, wherein in the difference complementation stage, the high-low resolution characteristic image group is input into a difference complementation module of a reconstruction network;
in the difference complementation module, firstly, a multi-core residual error feature extraction module is used for extracting feature information of a high-low resolution feature image group, carrying out difference on the extracted feature information of the high-low resolution feature image group and the extracted feature information of the high-low resolution feature image group, respectively adding a difference result to the extracted feature images, and outputting an updated high-low resolution feature image group; and then taking the output high-low resolution characteristic image group as input, repeating the next difference value complementation operation, and outputting the final high-resolution characteristic image after repeating the difference value complementation operation for m times.
5. The face super-resolution reconstruction method based on progressive difference complementation of claim 4, wherein the multi-core residual feature extraction module comprises 3 x 3 convolutions, 5 x 5 convolutions and 7 x 7 convolutions which are connected in parallel and are used for respectively extracting feature information from an input image, and each convolution layer adopts a residual structure; and cascading the extracted characteristic information of different receptive fields with the input original characteristic image, and inputting the cascaded characteristic information into 1 3 multiplied by 3 convolution to obtain the characteristic information of the original characteristic image.
6. The face super-resolution reconstruction method based on progressive difference complementation as claimed in claim 1, wherein the training process of the progressive difference complementation face super-resolution reconstruction network comprises:
acquiring a monitoring video, extracting a low-resolution face image of a pedestrian from the monitoring video, and simultaneously acquiring a high-resolution face image of the pedestrian, wherein the corresponding high-resolution face image and the corresponding low-resolution face image are used as a training sample set;
respectively preprocessing high-low resolution face images in a training sample set;
training a progressive difference value complementary face super-resolution reconstruction network by utilizing the preprocessed training sample set; the progressive difference complementary face super-resolution reconstruction network comprises two stages, after a low-resolution face image is input into the network, a high-low resolution feature image group is output through coarse processing in a first stage, a final high-resolution feature image is output through difference complementation in a second stage, the image and the low-resolution face image are added pixel by pixel through images output through bicubic interpolation up-sampling, a super-resolution face reconstruction image is output, and the network is trained based on the super-resolution face reconstruction image and the high-resolution face image in a training sample set.
7. The face super-resolution reconstruction method based on progressive difference complementation of claim 6, wherein the reconstruction network is optimized by calculating pixel loss between the super-resolution face reconstruction image and the high-resolution face image through an L1 loss function; the loss function calculation formula is as follows:
wherein L is Pixel Indicating total loss of network, I HR And I SR Respectively representing a high-resolution face image and a super-resolution face reconstruction image.
8. The face super-resolution reconstruction system based on progressive difference complementation is characterized by comprising:
the face image acquisition module is used for acquiring a monitoring video to be detected and extracting a low-resolution face image of a pedestrian from the monitoring video;
the image processing module is used for preprocessing the low-resolution face image;
the image feature extraction module is used for inputting the preprocessed low-resolution image into a progressive difference complementary face super-resolution reconstruction network, outputting a high-low-resolution feature image group through coarse processing in a first stage after the low-resolution face image is input into the network, and outputting a final high-resolution feature image through difference complementation in a second stage after the high-low-resolution feature image group is input into the network;
and the image reconstruction module is used for carrying out super-resolution reconstruction on the output final high-resolution characteristic image, carrying out pixel-by-pixel addition on the final high-resolution characteristic image and the image output by double-cubic interpolation up-sampling of the final high-resolution characteristic image and the low-resolution face image, and outputting a super-resolution face reconstruction image.
9. An electronic device comprising a memory and a processor and computer instructions stored on the memory and running on the processor, which when executed by the processor, perform the steps of a face super-resolution reconstruction method based on progressive difference complementation as claimed in any one of claims 1 to 7.
10. A computer readable storage medium storing computer instructions which, when executed by a processor, perform the steps of a face super-resolution reconstruction method based on progressive difference complementation as claimed in any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310089839.6A CN116029905A (en) | 2023-02-02 | 2023-02-02 | Face super-resolution reconstruction method and system based on progressive difference complementation |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310089839.6A CN116029905A (en) | 2023-02-02 | 2023-02-02 | Face super-resolution reconstruction method and system based on progressive difference complementation |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116029905A true CN116029905A (en) | 2023-04-28 |
Family
ID=86075803
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310089839.6A Pending CN116029905A (en) | 2023-02-02 | 2023-02-02 | Face super-resolution reconstruction method and system based on progressive difference complementation |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116029905A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116452424A (en) * | 2023-05-19 | 2023-07-18 | 山东大学 | Face super-resolution reconstruction method and system based on double generalized distillation |
-
2023
- 2023-02-02 CN CN202310089839.6A patent/CN116029905A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116452424A (en) * | 2023-05-19 | 2023-07-18 | 山东大学 | Face super-resolution reconstruction method and system based on double generalized distillation |
| CN116452424B (en) * | 2023-05-19 | 2023-10-10 | 山东大学 | Face super-resolution reconstruction method and system based on double generalized distillation |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Liu et al. | Residual feature aggregation network for image super-resolution | |
| CN109741260B (en) | Efficient super-resolution method based on depth back projection network | |
| CN110120011B (en) | Video super-resolution method based on convolutional neural network and mixed resolution | |
| CN110136062B (en) | Super-resolution reconstruction method combining semantic segmentation | |
| CN111105352A (en) | Super-resolution image reconstruction method, system, computer device and storage medium | |
| CN115358932B (en) | Multi-scale feature fusion face super-resolution reconstruction method and system | |
| CN109949224B (en) | Deep learning-based cascade super-resolution reconstruction method and device | |
| Li et al. | A lightweight multi-scale channel attention network for image super-resolution | |
| CN115953303B (en) | Multi-scale image compressed sensing reconstruction method and system combining channel attention | |
| CN112200720A (en) | Super-resolution image reconstruction method and system based on filter fusion | |
| CN112017116A (en) | Image super-resolution reconstruction network based on asymmetric convolution and construction method thereof | |
| CN110852948A (en) | Image super-resolution method based on feature correlation, storage medium and terminal equipment | |
| CN115841420A (en) | Polarization image super-resolution reconstruction method based on deep learning | |
| CN117575915A (en) | Image super-resolution reconstruction method, terminal equipment and storage medium | |
| Liu et al. | A deep recursive multi-scale feature fusion network for image super-resolution | |
| CN116029905A (en) | Face super-resolution reconstruction method and system based on progressive difference complementation | |
| CN111553861A (en) | Image super-resolution reconstruction method, device and equipment and readable storage medium | |
| Shen et al. | RSHAN: Image super-resolution network based on residual separation hybrid attention module | |
| CN116895037A (en) | Frame insertion method and system based on edge information and multi-scale cross fusion network | |
| CN116188272A (en) | Two-stage depth network image super-resolution reconstruction method suitable for multiple fuzzy cores | |
| CN114863094A (en) | Industrial image region-of-interest segmentation algorithm based on double-branch network | |
| Zhou et al. | Mixed Attention Densely Residual Network for Single Image Super-Resolution. | |
| CN114219738A (en) | Single-image multi-scale super-resolution reconstruction network structure and method | |
| Shen et al. | Itsrn++: Stronger and better implicit transformer network for continuous screen content image super-resolution | |
| Liu et al. | Lightweight image super-resolution with a feature-refined network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |