WO2020143513A1 - Super-resolution image reconstruction method, apparatus and device - Google Patents

Super-resolution image reconstruction method, apparatus and device Download PDF

Info

Publication number
WO2020143513A1
WO2020143513A1 PCT/CN2019/130826 CN2019130826W WO2020143513A1 WO 2020143513 A1 WO2020143513 A1 WO 2020143513A1 CN 2019130826 W CN2019130826 W CN 2019130826W WO 2020143513 A1 WO2020143513 A1 WO 2020143513A1
Authority
WO
WIPO (PCT)
Prior art keywords
resolution image
image
super
generator
ranking
Prior art date
Application number
PCT/CN2019/130826
Other languages
French (fr)
Chinese (zh)
Inventor
乔宇
张文龙
刘翼豪
董超
Original Assignee
深圳先进技术研究院
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 深圳先进技术研究院 filed Critical 深圳先进技术研究院
Publication of WO2020143513A1 publication Critical patent/WO2020143513A1/en

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis

Definitions

  • the present application belongs to the field of image processing, and particularly relates to a super-resolution image reconstruction method, device, and equipment.
  • Super-resolution image reconstruction technology has important academic research and industrial application value in the field of computer vision and image processing.
  • the purpose of super-resolution image reconstruction technology is to reconstruct the corresponding high-resolution image according to the given low-resolution image, and to obtain the best possible visual effect and small reconstruction error.
  • the learning method of convolutional neural network uses square error (Mean Square Error (MSE) to find the error between the high-resolution image reconstructed by the convolutional neural network and the real high-resolution image.
  • MSE Green Square Error
  • the mean square error can be used to obtain a reconstruction result with little error between the mean square error and the true high-resolution image.
  • the high-resolution image reconstructed by means of the mean square error shows an excessively smooth effect, which is not good for human eyes.
  • the existing cGAN-based super-resolution restoration and recovery technology is similar to the idea of super-resolution reconstruction using convolutional neural networks in the overall solution.
  • the generator G is used to learn the mapping between low-resolution images and high-resolution images.
  • the discriminator D and the generator G are further introduced to conduct adversarial training.
  • the high-resolution image generated by the generator G is calculated and back-propagated with the real image in a specific feature space.
  • the perceptual quality evaluation index uses feature extraction and further transformation operations, so it is not differentiable, and gradient solution and back propagation cannot be performed. This leads to high difficulty in optimizing the perceptual quality evaluation index, and visual The effect improvement is limited.
  • the embodiments of the present application provide a high-resolution image reconstruction method, device, and equipment to solve the problems in the prior art.
  • a first aspect of the embodiments of the present application provides a high-resolution image reconstruction method.
  • the super-resolution image reconstruction method includes:
  • the low-resolution image is reconstructed according to the trained generator.
  • the step of calculating the perceptual quality evaluation score of images of different perceptual quality through the perceptual quality evaluation index includes:
  • Image blocks with the same image content and different perceptual qualities are selected to form image pairs, and the perceptual quality evaluation scores of image pairs with different perceptual qualities are obtained through the non-reference image evaluation index.
  • the step of training the ranking estimation network according to the calculated perceptual quality evaluation score includes:
  • the ranking error of the convolutional neural network is obtained
  • the parameters of the convolutional neural network are updated by a back propagation algorithm.
  • the calculation based on the trained ranking estimation network calculates and generates the ranking content loss of the image generated by the generator against the network, and guides according to the ranking content loss guidance
  • the steps to train a generator that generates an adversarial network include:
  • the generator is trained according to the sorted content loss.
  • the step of training the generator according to the sorted content loss further includes:
  • the generator is trained by combining the sorted content loss, the discriminator error and the perceptual loss error.
  • the method before the step of training the ranking estimation network according to the calculated perceptual quality evaluation score, the method further includes:
  • a second aspect of an embodiment of the present application provides a super-resolution image reconstruction device, the super-resolution image reconstruction device includes:
  • Image generation unit for generating images of different perceptual qualities through different super-resolution image generation methods
  • the evaluation score calculation unit is used to calculate the perception quality evaluation score of images with different perception qualities through the perception quality evaluation index;
  • the ranking estimation network training unit is used to train the ranking estimation network according to the calculated perceptual quality evaluation score
  • the generator training unit is configured to calculate the sequence content loss of the image generated by the generator of the confrontation network according to the trained sequence estimation network, and guide the generator of the confrontation network generation training according to the sequence content loss;
  • the reconstruction unit is used to reconstruct the super-resolution image of the low-resolution image according to the trained generator.
  • the evaluation score calculation unit includes:
  • a cropping subunit configured to crop the images of different perceptual qualities into image blocks
  • the evaluation subunit is used to select image blocks of the same image content and having different perceptual qualities to form an image pair, and to obtain the perceptual quality evaluation scores of image pairs of different perceptual qualities through a non-reference image evaluation index.
  • a third aspect of the embodiments of the present application provides a high-resolution image reconstruction device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, the processor When the computer program is executed, the steps of the high-resolution image reconstruction method according to any one of the first aspect are realized.
  • a fourth aspect of the embodiments of the present application provides a computer-readable storage medium that stores a computer program, and when the computer program is executed by a processor, the computer program is implemented as described in any one of the first aspects. Resolution image reconstruction method steps.
  • the embodiment of the present application has the following beneficial effects: the present application can use the ranking estimation network to directly optimize the perceptual quality evaluation index to obtain better image perceptual quality, which can reduce the cost of development time and can be significantly improved Reconstruct the perceptual quality of high-resolution images; and, through the order estimation network, you can learn training data with different a priori information, be able to flexibly expand according to needs, and constrain the generator to generate super-resolution reconstructed images with different characteristics .
  • FIG. 1 is a schematic flowchart of an implementation method of a super-resolution image reconstruction method provided by an embodiment of the present application
  • FIG. 2 is a schematic diagram of an implementation process of a training method for ranking estimation network provided by an embodiment of the present application
  • FIG. 3 is a schematic structural diagram of a ranking estimation network provided by an embodiment of the present application.
  • FIG. 4 is a schematic diagram of an implementation process of a generator training method provided by an embodiment of the present application.
  • FIG. 5 is a schematic diagram of a super-resolution image reconstruction system provided by an embodiment of the present application.
  • FIG. 5a is a comparison schematic diagram of experimental effects provided by examples of the present application.
  • FIG. 6 is a schematic diagram of a super-resolution image reconstruction device provided by an embodiment of the present application.
  • FIG. 7 is a schematic diagram of a high-resolution image reconstruction device provided by an embodiment of the present application.
  • FIG. 1 is a schematic diagram of an implementation process of a super-resolution image reconstruction method provided by an embodiment of the present application, and the details are as follows:
  • step S101 images of different perceptual qualities are generated by different super-resolution image generation methods
  • one or more low-resolution images may be selected, and different super-resolution image generation methods may be used to generate images of different perceptual qualities for each image.
  • the super-resolution image generation method may include a pixel-loss super-resolution method, a conventional generation-resistant super-resolution image generation method, or a super-resolution image generation method based on a convolutional neural network, and the like.
  • the training data set for generating low-resolution images of images with different perceptual qualities can be transformed and expanded according to requirements, that is, training data with different prior information can be selected.
  • step S102 a perceptual quality evaluation score of images of different perceptual quality is calculated through a perceptual quality evaluation index
  • images of different perceptual quality can be cropped into image blocks, image blocks with the same image content and different perceptual quality can be selected to form image pairs, and the image pair can be calculated through the perceptual quality evaluation index.
  • Perceived quality evaluation score may include no reference image quality evaluation (English is referred to as No Reference Image Quality Assessment, referred to as NR-IQA in English, for example, natural image evaluation (English is called Natural image quality evaluator, referred to as NIQE in English), etc.
  • NIQE Natural image quality evaluator
  • step S103 train the ranking estimation network according to the calculated perceptual quality evaluation score
  • the step of training the ranking estimation network may be specifically shown in FIG. 2 and includes:
  • step S201 the convolutional neural network selected as the ranking estimation network calculates the ranking score of the image
  • the ranking estimation network an appropriate convolutional neural network model can be selected to calculate the ranking score of the image. Since the sorting estimation task and the classification task have a certain degree of correlation, you can refer to the classic classification network for design, such as VGG-Net (Visual Geometry Group, Network).
  • the ranking estimation network can be implemented by a twin structure of a basic convolutional neural network. As shown in FIG. 3, the convolutional neural network includes a feature extraction module with parameter sharing, a pooling module, a fully connected module, and a ranking loss function module , Where the feature extraction module can use VGG-16 network.
  • step S202 according to the evaluation score calculated by the perceptual quality evaluation index and the ranking score, the ranking error of the convolutional neural network is obtained;
  • images 1 and 2 with different perceptual qualities of the same content are input to the feature extraction module, and then through the pooling layer and the fully connected layer to obtain their corresponding ranking score 1 and ranking score 2, respectively, and then use the ranking Score 1 and ranking score 2 use the ranking loss function to update the parameters of the overall network.
  • step S203 the parameters of the convolutional neural network are updated by a back propagation algorithm.
  • the convolutional neural network can output their ranking scores.
  • the perceptual quality evaluation index for example, the smaller the NIQE, the better the perceptual quality of the image, and the smaller the ranking score under the same image content, the better the perceptual quality of the image.
  • the ranking loss (margin-ranking loss) can get the sorting error.
  • Reflectation Backpropagation Algorithm
  • step S104 calculate the sequence content loss of the image generated by the generator of the confrontation network according to the trained sequence estimation network, and guide the generator of the generation of the confrontation network according to the sequence content loss for training;
  • the process of training the generator includes:
  • step S401 a high-resolution image is generated by the generator
  • the generator may generate a high-resolution image according to the low-resolution image or the image block of the low-resolution image in the sample.
  • step S402 the ranked content loss of the high-resolution image is calculated according to the trained ranking estimation network
  • step S403 the generator is trained according to the sorted content loss.
  • the step of training the generator according to the sorted content loss further includes:
  • the generator is trained by combining the sorted content loss, the discriminator error and the perceptual loss error.
  • the generator Based on the loss of sorted content, combined with the discriminant error of the discriminator and the perceptual loss error of the feature space of the VGG network, the generator can be trained more effectively, the training efficiency of the generator can be improved, and the super-resolution can be improved.
  • step S105 super-resolution image reconstruction is performed on the low-resolution image according to the trained generator.
  • this application can use the ranking estimation network to directly optimize on specific perceptual quality evaluation standards to obtain better image perceptual quality. It is manifested that the existing super-resolution image methods mostly use the error in mean square error or feature space to optimize the super-resolution model. In the specific perceptual quality evaluation standard, the improvement of such methods is very time-consuming and laborious, because neither the mean square error nor the error optimization in the feature space can directly represent the improvement in specific perceptual quality.
  • This application proposes to use a ranking estimation network to constrain the super-resolution model to directly optimize the specific perceptual quality, which can reduce the cost of development time and can significantly improve the perceptual quality of reconstructed high-resolution images.
  • the present invention can have stronger expansibility for specific image perception quality.
  • the performance is that the ranking estimation network can learn training data with different prior information, and can be flexibly expanded according to needs. Different prior information can be learned through the ordering of different perceptual quality data in training data and its own statistical characteristics. The ranking estimation network with different prior information can constrain the generator to generate super-resolution reconstruction images with different characteristics.
  • FIG. 5 is a schematic diagram of a super-rank generation anti-super-resolution image network provided by an embodiment of the present application, which is mainly composed of a generator, a discriminator, a feature extractor, and a rank estimator, where the feature extractor and rank estimator can For the trained model, only the loss calculation can be performed without updating the model parameters.
  • the low-resolution image can be generated by the generator to generate a reconstructed high-resolution image
  • the discriminator can calculate the original high-resolution image and the reconstructed high-resolution image to combat loss
  • the feature extractor can separately reconstruct The high-resolution image and the original high-resolution image calculate the error in the feature space to obtain the perceptual loss error.
  • the reconstructed high-resolution image can be obtained by calculating the ranking score under the specific perceptual quality evaluation standard through the ranking estimator. Loss of sorted content. Finally, it uses joint loss, perceptual loss error and ordered content loss to jointly train the generator.
  • 5a is a schematic diagram of a comparison of reconstruction effects of a super-resolution image provided by an embodiment of the application, where the first left is a low-resolution image, the second left is a super-resolution image generated by a super-resolution method based on pixel loss, and the right
  • the second is the traditional generation of the super-resolution image generated by the super-resolution resolution method
  • the first one is the super-resolution image generated by the reconstruction method of the super-resolution image described in this application.
  • the super-resolution image reconstruction method described in this application can generate a more realistic image from a low-resolution image.
  • FIG. 6 is a schematic structural diagram of a super-resolution image reconstruction device provided by an embodiment of the present application, and the details are as follows:
  • the super-resolution image reconstruction device includes:
  • the image generating unit 601 is used to generate images with different perceptual qualities through different super-resolution image generating methods
  • the evaluation score calculation unit 602 is used to calculate the perception quality evaluation scores of images with different perception qualities through the perception quality evaluation indicators;
  • the ranking estimation network training unit 603 is used to train the ranking estimation network according to the calculated perceptual quality evaluation score
  • the generator training unit 604 is configured to calculate the sequence content loss of the image generated by the generator of the confrontation network according to the trained sequence estimation network, and guide the generator of the generation of the confrontation network according to the sequence content loss for training;
  • the reconstruction unit 605 is configured to reconstruct the super-resolution image of the low-resolution image according to the trained generator.
  • the evaluation score calculation unit includes:
  • a cropping subunit configured to crop the images of different perceptual qualities into image blocks
  • the evaluation subunit is used to select image blocks of the same image content and having different perceptual qualities to form an image pair, and to obtain the perceptual quality evaluation scores of image pairs of different perceptual qualities through a non-reference image evaluation index.
  • the high-resolution image reconstruction device described in FIG. 6 corresponds to the high-resolution image reconstruction method described in FIG. 1.
  • the high-resolution image reconstruction device 7 of this embodiment includes: a processor 70, a memory 71, and a computer program 72 stored in the memory 71 and executable on the processor 70, For example, high-resolution image reconstruction procedures.
  • the processor 70 executes the computer program 72, the steps in the above embodiments of the method for reconstructing each high-resolution image are implemented.
  • the processor 70 executes the computer program 72, the functions of the modules/units in the foregoing device embodiments are realized.
  • the computer program 72 may be divided into one or more modules/units, and the one or more modules/units are stored in the memory 71 and executed by the processor 70 to complete This application.
  • the one or more modules/units may be a series of computer program instruction segments capable of performing specific functions, which are used to describe the execution process of the computer program 72 in the high-resolution image reconstruction device 7 .
  • the computer program 72 may be divided into:
  • Image generation unit for generating images of different perceptual qualities through different super-resolution image generation methods
  • the evaluation score calculation unit is used to calculate the perception quality evaluation score of images with different perception qualities through the perception quality evaluation index;
  • the ranking estimation network training unit is used to train the ranking estimation network according to the calculated perceptual quality evaluation score
  • the generator training unit is configured to calculate the sequence content loss of the image generated by the generator of the confrontation network according to the trained sequence estimation network, and guide the generator of the confrontation network generation training according to the sequence content loss;
  • the reconstruction unit is used to reconstruct the super-resolution image of the low-resolution image according to the trained generator.
  • the reconstruction device 7 of the high-resolution image may be a computing device such as a desktop computer, a notebook, a palmtop computer and a cloud server.
  • the high-resolution image reconstruction device may include, but is not limited to, the processor 70 and the memory 71.
  • FIG. 7 is only an example of the high-resolution image reconstruction device 7 and does not constitute a limitation on the high-resolution image reconstruction device 7, and may include more or less than the illustration.
  • Components, or a combination of certain components, or different components, for example, the high-resolution image reconstruction device may further include an input and output device, a network access device, a bus, and the like.
  • the so-called processor 70 may be a central processing unit (Central Processing Unit (CPU), can also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (Application Specific Integrated Circuit (ASIC), ready-made programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.
  • the general-purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
  • the memory 71 may be an internal storage unit of the high-resolution image reconstruction device 7, for example, a hard disk or a memory of the high-resolution image reconstruction device 7.
  • the memory 71 may also be an external storage device of the high-resolution image reconstruction device 7, for example, a plug-in hard disk equipped on the high-resolution image reconstruction device 7, a smart memory card (Smart Media Card , SMC), Secure Digital (SD) card, flash memory card (Flash Card) etc.
  • the memory 71 may also include both the internal storage unit of the high-resolution image reconstruction device 7 and the external storage device.
  • the memory 71 is used to store the computer program and other programs and data required by the high-resolution image reconstruction device.
  • the memory 71 can also be used to temporarily store data that has been or will be output.
  • the disclosed device/terminal device and method may be implemented in other ways.
  • the device/terminal device embodiments described above are only schematic.
  • the division of the module or unit is only a logical function division, and in actual implementation, there may be another division manner, such as multiple units Or components can be combined or integrated into another system, or some features can be ignored or not implemented.
  • the displayed or discussed mutual coupling or direct coupling or communication connection may be indirect coupling or communication connection through some interfaces, devices or units, and may be electrical, mechanical or other forms.
  • the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
  • each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units may be integrated into one unit.
  • the above integrated unit can be implemented in the form of hardware or software function unit.
  • the integrated module/unit is implemented in the form of a software functional unit and sold or used as an independent product, it may be stored in a computer-readable storage medium.
  • the present application can implement all or part of the processes in the methods of the above embodiments, or it can be completed by a computer program instructing related hardware.
  • the computer program can be stored in a computer-readable storage medium. When the program is executed by the processor, the steps of the foregoing method embodiments may be implemented.
  • the computer program includes computer program code, and the computer program code may be in a source code form, an object code form, an executable file, or some intermediate form, etc.
  • the computer readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a U disk, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signals, telecommunications signals, and software distribution media.
  • ROM Read-Only Memory
  • RAM Random Access Memory
  • electrical carrier signals telecommunications signals
  • software distribution media any entity or device capable of carrying the computer program code
  • a recording medium a U disk, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signals, telecommunications signals, and software distribution media.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Biomedical Technology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Image Analysis (AREA)

Abstract

A super-resolution image reconstruction method, comprising: generating images of different perceptual qualities by means of different super-resolution image generation methods; calculating, by means of a perceptual quality evaluation index, perceptual quality evaluation scores of the images of different perceptual qualities; training a ranking estimation network according to the calculated perceptual quality evaluation scores; calculating, according to the trained ranking estimation network, a ranking content loss of an image generated by a generator of a generative adversarial network, and guiding, according to the ranking content loss, the generator of the generative adversarial network to train; performing super-resolution image reconstruction on a low-resolution image according to the trained generator. The present invention uses the ranking estimation network to directly perform optimization on the perceptual quality evaluation index, so as to obtain a better image perceptual quality, can flexibly perform expansion according to requirements, and can constrain the generator to generate super-resolution reconstructed images of different characteristics.

Description

一种超分辨率图像的重构方法、装置及设备Super-resolution image reconstruction method, device and equipment 技术领域Technical field
本申请属于图像处理领域，尤其涉及一种超分辨率的图像的重构方法、装置及设备。The present application belongs to the field of image processing, and particularly relates to a super-resolution image reconstruction method, device, and equipment.
背景技术Background technique
超分辨率图像重构技术在计算机视觉和图像处理领域拥有重要的学术研究和工业应用价值。超分辨率图像重构技术的目的就是，根据给定的低分辨率图像去重构出其所对应的高分辨率图像，并且能够得到尽量好的视觉效果和小的重构误差。Super-resolution image reconstruction technology has important academic research and industrial application value in the field of computer vision and image processing. The purpose of super-resolution image reconstruction technology is to reconstruct the corresponding high-resolution image according to the given low-resolution image, and to obtain the best possible visual effect and small reconstruction error.
在对低分辨率图像进行重构，目前一般包括卷积神经网络的学习方法和对抗生成网络的方法。其中，卷积神经网络的学习方法使用方误差（Mean Square Error, MSE）来求卷积神经网络重建的高分辨率图像和真实高分辨率图像之间的误差。虽然通过均方误差的方式能够得到在均方误差上和真实高分辨率图像误差很小的重建结果。然而，通过均方误差的方式重建出的高分辨率图像表现出了过于平滑的效果，对于人眼观测的视觉效果不好。In the reconstruction of low-resolution images, currently generally include the learning method of convolutional neural network and the method of confrontation generation network. Among them, the learning method of convolutional neural network uses square error (Mean Square Error (MSE) to find the error between the high-resolution image reconstructed by the convolutional neural network and the real high-resolution image. Although the mean square error can be used to obtain a reconstruction result with little error between the mean square error and the true high-resolution image. However, the high-resolution image reconstructed by means of the mean square error shows an excessively smooth effect, which is not good for human eyes.
现有的基于cGAN的超分辨恢复恢复技术在整体解决思路上和利用卷积神经网络进行超分辨率重建的思路类似，通过生成器G来学习低分率图像和高分辨率图像之间的映射关系，然后进一步引入了判别器D来和生成器G来进行对抗训练，同时对于生成器G生成的高分辨率图像在特定的特征空间上和真实图像进行误差计算并进行反向传播。相对于均方误差，这种方式虽然能够得到较好的视觉效果，但是和真实的高分辨率图像相比纹理细节特征却不够真实。另外一方面，感知质量评价指标使用了特征的提取和进一步的变换操作所以是不可微的，不能进行梯度求解和反向传播，这就导致了在感知质量评价指标上的优化难度高，并且视觉效果提升受限。The existing cGAN-based super-resolution restoration and recovery technology is similar to the idea of super-resolution reconstruction using convolutional neural networks in the overall solution. The generator G is used to learn the mapping between low-resolution images and high-resolution images. Then, the discriminator D and the generator G are further introduced to conduct adversarial training. At the same time, the high-resolution image generated by the generator G is calculated and back-propagated with the real image in a specific feature space. Compared with the mean square error, although this method can get a better visual effect, compared with the real high-resolution image, the texture details are not real enough. On the other hand, the perceptual quality evaluation index uses feature extraction and further transformation operations, so it is not differentiable, and gradient solution and back propagation cannot be performed. This leads to high difficulty in optimizing the perceptual quality evaluation index, and visual The effect improvement is limited.
技术问题technical problem
有鉴于此，本申请实施例提供了一种高分辨率图像的重构方法、装置及设备，以解决现有技术中的问题。In view of this, the embodiments of the present application provide a high-resolution image reconstruction method, device, and equipment to solve the problems in the prior art.
技术解决方案Technical solution
本申请实施例的第一方面提供了一种高分辨率图像的重构方法，所述超分辨率图像的重构方法包括：A first aspect of the embodiments of the present application provides a high-resolution image reconstruction method. The super-resolution image reconstruction method includes:
通过不同的超分辨率图像生成方法生成不同感知质量的图像；Generate images with different perceptual qualities through different super-resolution image generation methods;
通过感知质量评价指标计算不同感知质量的图像的感知质量评价分数；Calculate the perceptual quality evaluation score of images with different perceptual quality through the perceptual quality evaluation index;
根据所计算的感知质量评价分数对排序估计网络进行训练；Train the ranking estimation network according to the calculated perceptual quality evaluation score;
根据训练后的排序估计网络计算生成对抗网络的生成器所生成的图像的排序内容损失，根据所述排序内容损失指导生成对抗网络的生成器进行训练；Calculating the sequence content loss of the image generated by the generator of the confrontation network according to the trained sequence estimation network, and instructing the generator of generating the confrontation network according to the sequence content loss for training;
根据训练后的生成器对低分辨率图像进行超分辨率图像的重构。The low-resolution image is reconstructed according to the trained generator.
结合第一方面，在第一方面的第一种可能实现方式中，所述通过感知质量评价指标计算不同感知质量的图像的感知质量评价分数的步骤包括：With reference to the first aspect, in a first possible implementation manner of the first aspect, the step of calculating the perceptual quality evaluation score of images of different perceptual quality through the perceptual quality evaluation index includes:
将所述不同感知质量的图像裁剪为图像块；Crop the images of different perceptual qualities into image blocks;
选择同一图像内容且具有不同感知质量的图像块构成图像对，通过无参考图像评价指标获取不同感知质量的图像对的感知质量评价分数。Image blocks with the same image content and different perceptual qualities are selected to form image pairs, and the perceptual quality evaluation scores of image pairs with different perceptual qualities are obtained through the non-reference image evaluation index.
结合第一方面的第一种可能实现方式，在第一方面的第二种可能实现方式中，所述根据所计算的感知质量评价分数对排序估计网络进行训练的步骤包括：With reference to the first possible implementation manner of the first aspect, in a second possible implementation manner of the first aspect, the step of training the ranking estimation network according to the calculated perceptual quality evaluation score includes:
选择作为排序估计网络的卷积神经网络计算所述图像的排序分数；Selecting a convolutional neural network as the ranking estimation network to calculate the ranking score of the image;
根据感知质量评价指标计算的评价分数和所述排序分数，得到所述卷积神经网络的排序误差；According to the evaluation score calculated by the perceptual quality evaluation index and the ranking score, the ranking error of the convolutional neural network is obtained;
通过反向传播算法更新所述卷积神经网络的参数。The parameters of the convolutional neural network are updated by a back propagation algorithm.
结合第一方面，在第一方面的第三种可能实现方式中，所述根据训练后的排序估计网络计算生成对抗网络的生成器所生成的图像的排序内容损失，根据所述排序内容损失指导生成对抗网络的生成器进行训练的步骤包括：With reference to the first aspect, in a third possible implementation manner of the first aspect, the calculation based on the trained ranking estimation network calculates and generates the ranking content loss of the image generated by the generator against the network, and guides according to the ranking content loss guidance The steps to train a generator that generates an adversarial network include:
通过生成器生成高分辨率图像；Generate high-resolution images through the generator;
根据所述训练后的排序估计网络计算所述高分辨率图像的排序内容损失；Calculating the ranking content loss of the high-resolution image according to the trained ranking estimation network;
根据所述排序内容损失对生成器进行训练。The generator is trained according to the sorted content loss.
结合第一方面的第三种可能实现方式，在第一方面的第四种可能实现方式中，所述根据所述排序内容损失对生成器进行训练的步骤还包括：With reference to the third possible implementation manner of the first aspect, in a fourth possible implementation manner of the first aspect, the step of training the generator according to the sorted content loss further includes:
通过判别器判断所生成的高分辨率图像与真实高分辨率图像的误差；Determine the error between the generated high-resolution image and the real high-resolution image through the discriminator;
通过VGG网络的特征空间计算生成的高分辨率图像与真实图像高分辨率图像的感知损失误差；The perceptual loss error of the high-resolution image generated by the feature space calculation of the VGG network and the real image high-resolution image;
联合所述排序内容损失、所述判别器判断的误差和所述感知损失误差，对生成器进行训练。The generator is trained by combining the sorted content loss, the discriminator error and the perceptual loss error.
结合第一方面，在第一方面的第五种可能实现方式中，在所述根据所计算的感知质量评价分数对排序估计网络进行训练的步骤之前，所述方法还包括：With reference to the first aspect, in a fifth possible implementation manner of the first aspect, before the step of training the ranking estimation network according to the calculated perceptual quality evaluation score, the method further includes:
计算不同感知质量的图像的差异，选择所述图像的差异大于预设的阈值的图像。Calculate the difference of images with different perceptual qualities, and select an image whose difference between the images is greater than a preset threshold.
本申请实施例的第二方面提供了一种超分辨率图像的重构装置，所述超分辨率图像的重构装置包括：A second aspect of an embodiment of the present application provides a super-resolution image reconstruction device, the super-resolution image reconstruction device includes:
图像生成单元，用于通过不同的超分辨率图像生成方法生成不同感知质量的图像；Image generation unit for generating images of different perceptual qualities through different super-resolution image generation methods;
评价分数计算单元，用于通过感知质量评价指标计算不同感知质量的图像的感知质量评价分数；The evaluation score calculation unit is used to calculate the perception quality evaluation score of images with different perception qualities through the perception quality evaluation index;
排序估计网络训练单元，用于根据所计算的感知质量评价分数对排序估计网络进行训练；The ranking estimation network training unit is used to train the ranking estimation network according to the calculated perceptual quality evaluation score;
生成器训练单元，用于根据训练后的排序估计网络计算生成对抗网络的生成器所生成的图像的排序内容损失，根据所述排序内容损失指导生成对抗网络的生成器进行训练；The generator training unit is configured to calculate the sequence content loss of the image generated by the generator of the confrontation network according to the trained sequence estimation network, and guide the generator of the confrontation network generation training according to the sequence content loss;
重构单元，用于根据训练后的生成器对低分辨率图像进行超分辨率图像的重构。The reconstruction unit is used to reconstruct the super-resolution image of the low-resolution image according to the trained generator.
结合第二方面，在第二方面的第一种可能实现方式中，所述评价分数计算单元包括：With reference to the second aspect, in a first possible implementation manner of the second aspect, the evaluation score calculation unit includes:
裁剪子单元，用于将所述不同感知质量的图像裁剪为图像块；A cropping subunit, configured to crop the images of different perceptual qualities into image blocks;
评价子单元，用于选择同一图像内容且具有不同感知质量的图像块构成图像对，通过无参考图像评价指标获取不同感知质量的图像对的感知质量评价分数。The evaluation subunit is used to select image blocks of the same image content and having different perceptual qualities to form an image pair, and to obtain the perceptual quality evaluation scores of image pairs of different perceptual qualities through a non-reference image evaluation index.
本申请实施例的第三方面提供了一种高分辨率图像的重构设备，包括存储器、处理器以及存储在所述存储器中并可在所述处理器上运行的计算机程序，所述处理器执行所述计算机程序时实现如第一方面任一项所述高分辨率图像的重构方法的步骤。A third aspect of the embodiments of the present application provides a high-resolution image reconstruction device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, the processor When the computer program is executed, the steps of the high-resolution image reconstruction method according to any one of the first aspect are realized.
本申请实施例的第四方面提供了一种计算机可读存储介质，所述计算机可读存储介质存储有计算机程序，所述计算机程序被处理器执行时实现如第一方面任一项所述高分辨率图像的重构方法的步骤。A fourth aspect of the embodiments of the present application provides a computer-readable storage medium that stores a computer program, and when the computer program is executed by a processor, the computer program is implemented as described in any one of the first aspects. Resolution image reconstruction method steps.
有益效果Beneficial effect
本申请实施例与现有技术相比存在的有益效果是：本申请能够使用排序估计网络在感知质量评价指标上直接优化而得到更好的图像感知质量，可以降低研发时间成本并且能够显著的提升重构高分辨率图像的感知质量；并且，通过排序估计网络可以学习具有不同先验信息的训练数据，能够灵活的根据需求来进行拓展，可以约束生成器生成具有不同特点的超分辨率重建图像。Compared with the prior art, the embodiment of the present application has the following beneficial effects: the present application can use the ranking estimation network to directly optimize the perceptual quality evaluation index to obtain better image perceptual quality, which can reduce the cost of development time and can be significantly improved Reconstruct the perceptual quality of high-resolution images; and, through the order estimation network, you can learn training data with different a priori information, be able to flexibly expand according to needs, and constrain the generator to generate super-resolution reconstructed images with different characteristics .
附图说明BRIEF DESCRIPTION
为了更清楚地说明本申请实施例中的技术方案，下面将对实施例或现有技术描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本申请的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动性的前提下，还可以根据这些附图获得其他的附图。In order to more clearly explain the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required in the embodiments or the description of the prior art. Obviously, the drawings in the following description are only for the application For some embodiments, those of ordinary skill in the art can obtain other drawings based on these drawings without creative labor.
图1是本申请实施例提供的一种超分辨率图像的重构方法的实现流程示意图；1 is a schematic flowchart of an implementation method of a super-resolution image reconstruction method provided by an embodiment of the present application;
图2是本申请实施例提供的一种排序估计网络训练方法的实现流程示意图；2 is a schematic diagram of an implementation process of a training method for ranking estimation network provided by an embodiment of the present application;
图3是本申请实施例提供的排序估计网络的结构示意图；3 is a schematic structural diagram of a ranking estimation network provided by an embodiment of the present application;
图4是本申请实施例提供的生成器训练方法的实现流程示意图；4 is a schematic diagram of an implementation process of a generator training method provided by an embodiment of the present application;
图5是本申请实施例提供的超分辨率图像的重构***示意图；5 is a schematic diagram of a super-resolution image reconstruction system provided by an embodiment of the present application;
图5a为本申请实施例提供的实验效果对比示意图；FIG. 5a is a comparison schematic diagram of experimental effects provided by examples of the present application;
图6是本申请实施例提供的一种超分辨率图像的重构装置的示意图；6 is a schematic diagram of a super-resolution image reconstruction device provided by an embodiment of the present application;
图7是本申请实施例提供的高分辨率图像的重构设备的示意图。7 is a schematic diagram of a high-resolution image reconstruction device provided by an embodiment of the present application.
本发明的实施方式Embodiments of the invention
以下描述中，为了说明而不是为了限定，提出了诸如特定***结构、技术之类的具体细节，以便透彻理解本申请实施例。然而，本领域的技术人员应当清楚，在没有这些具体细节的其它实施例中也可以实现本申请。在其它情况中，省略对众所周知的***、装置、电路以及方法的详细说明，以免不必要的细节妨碍本申请的描述。In the following description, for the purpose of illustration rather than limitation, specific details such as specific system structures and technologies are proposed to thoroughly understand the embodiments of the present application. However, those skilled in the art should understand that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid unnecessary details hindering the description of the present application.
为了说明本申请所述的技术方案，下面通过具体实施例来进行说明。In order to explain the technical solutions described in the present application, the following will be described with specific embodiments.
图1为本申请实施例提供的一种超分辨率图像的重构方法的实现流程示意图，详述如下：FIG. 1 is a schematic diagram of an implementation process of a super-resolution image reconstruction method provided by an embodiment of the present application, and the details are as follows:
在步骤S101中，通过不同的超分辨率图像生成方法生成不同感知质量的图像；In step S101, images of different perceptual qualities are generated by different super-resolution image generation methods;
具体的，可以选用一张或者多张低分辨率图像，可以用不同的超分辨率图像生成方法，对每张图像生成不同的感知质量的图像。所述超分辨率图像生成方法可以包括像素损失的超分辨率方法、传统的生成对抗超分辨率图像生成方法或基于卷积神经网络的超分辨率图像生成方法等。Specifically, one or more low-resolution images may be selected, and different super-resolution image generation methods may be used to generate images of different perceptual qualities for each image. The super-resolution image generation method may include a pixel-loss super-resolution method, a conventional generation-resistant super-resolution image generation method, or a super-resolution image generation method based on a convolutional neural network, and the like.
对于同一图像，采用两种不同的超分辨率图像生成方法生成不同感知质量的图像时，可以选取其中的两个不同感知质量的图像，作为图像对。For the same image, when two different super-resolution image generation methods are used to generate images with different perceptual qualities, two images with different perceptual qualities can be selected as image pairs.
其中，用于生成不同感知质量的图像的低分辨率图像的训练数据集，可以根据需求进行变换和拓展，即可以选择不同的先验信息的训练数据。Among them, the training data set for generating low-resolution images of images with different perceptual qualities can be transformed and expanded according to requirements, that is, training data with different prior information can be selected.
在步骤S102中，通过感知质量评价指标计算不同感知质量的图像的感知质量评价分数；In step S102, a perceptual quality evaluation score of images of different perceptual quality is calculated through a perceptual quality evaluation index;
为了便于对图像通过感知质量评价指标进行计算，可以将不同感知质量的图像裁剪为图像块，选择同一图像内容且具有不同知感知质量的图像块构成图像对，通过感知质量评价指标计算图像对的感知质量评价分数。其中，所述感知质量评价指标可以包括无参考图像质量评价（英文全称为No Reference Image Quality Assessment, 英文简称为NR-IQA）, 例如自然图像评价（英文全称为Natural image quality evaluator，英文简称为NIQE）等。通过感知质量评价指标的评价，可以获得同一个图像内容上不同感知质量的图像对的感知质量评价分数，可以将所获得的感知质量评价分数作为图像对中的图像的标签。In order to facilitate the calculation of the image through the perceptual quality evaluation index, images of different perceptual quality can be cropped into image blocks, image blocks with the same image content and different perceptual quality can be selected to form image pairs, and the image pair can be calculated through the perceptual quality evaluation index. Perceived quality evaluation score. Wherein, the perceptual quality evaluation index may include no reference image quality evaluation (English is referred to as No Reference Image Quality Assessment, referred to as NR-IQA in English, for example, natural image evaluation (English is called Natural image quality evaluator, referred to as NIQE in English), etc. Through the evaluation of the perceptual quality evaluation index, the perceptual quality evaluation score of the image pair with different perceptual quality on the same image content can be obtained, and the obtained perceptual quality evaluation score can be used as the label of the image in the image pair.
当然，在使用图像对时，还可以对图像对的差异性进行评估，比如可以设置阈值来评估图像对的差异是否足够大，如果图像对的差异大于预设的阈值，则可以用于后续步骤的排序估计网络的训练。Of course, when using image pairs, you can also evaluate the difference of image pairs. For example, you can set a threshold to evaluate whether the difference between image pairs is large enough. If the difference between image pairs is greater than the preset threshold, it can be used in subsequent steps. The ranking of the estimated network training.
在步骤S103中，根据所计算的感知质量评价分数对排序估计网络进行训练；In step S103, train the ranking estimation network according to the calculated perceptual quality evaluation score;
对于所生成的不同感知质量的图像，对应有不同感知质量评价分数的标签。将不同感知质量的图像作为输入向量，作为标签的感知质量评价分数作为标签作为输出向量，根据多个所述输入向量及对应的所述输出向量作为训练数据，训练作为排序估计网络的卷积神经网络，从而使得训练后的排序估计网络可以对超分辨率图像进行排序估计，输出超分辨率图像的排序分数。其中，对所述排序估计网络进行训练的步骤具体可以如图2所示，包括：For the generated images with different perceptual qualities, corresponding labels with different perceptual quality evaluation scores are corresponding. Use images with different perceptual qualities as input vectors, perceptual quality evaluation scores as labels as tags as output vectors, and train convolutional nerves as ranking estimation networks based on multiple input vectors and corresponding output vectors as training data Network, so that the trained ranking estimation network can perform ranking estimation on the super-resolution images, and output the ranking score of the super-resolution images. Wherein, the step of training the ranking estimation network may be specifically shown in FIG. 2 and includes:
在步骤S201中，选择作为排序估计网络的卷积神经网络计算所述图像的排序分数；In step S201, the convolutional neural network selected as the ranking estimation network calculates the ranking score of the image;
对于排序估计网络，可以选取适当的卷积神经网络模型对图像进行排序分数的计算。由于排序估计任务和分类任务具有一定程度上的相关性，可以参考经典的分类网络进行设计，例如VGG-Net(Visual Geometry Group, Network)。排序估计网络可以通过一个基本卷积神经网络的双生结构来实现，如图3所示，所述卷积神经网络包括具有参数共享的特征提取模块、池化模块、全连接模块和排序损失函数模块，其中特征提取模块可以采用VGG-16网络。For the ranking estimation network, an appropriate convolutional neural network model can be selected to calculate the ranking score of the image. Since the sorting estimation task and the classification task have a certain degree of correlation, you can refer to the classic classification network for design, such as VGG-Net (Visual Geometry Group, Network). The ranking estimation network can be implemented by a twin structure of a basic convolutional neural network. As shown in FIG. 3, the convolutional neural network includes a feature extraction module with parameter sharing, a pooling module, a fully connected module, and a ranking loss function module , Where the feature extraction module can use VGG-16 network.
在步骤S202中，根据感知质量评价指标计算的评价分数和所述排序分数，得到所述卷积神经网络的排序误差；In step S202, according to the evaluation score calculated by the perceptual quality evaluation index and the ranking score, the ranking error of the convolutional neural network is obtained;
由图3可知，具有相同内容的不同感知质量的图像1和图像2分别输入到特征提取模块，然后经过池化层和全连接层分别得到其对应的排序分数1和排序分数2，然后利用排序分数1和排序分数2使用排序损失函数来进行整体网络的参数更新。As can be seen from FIG. 3, images 1 and 2 with different perceptual qualities of the same content are input to the feature extraction module, and then through the pooling layer and the fully connected layer to obtain their corresponding ranking score 1 and ranking score 2, respectively, and then use the ranking Score 1 and ranking score 2 use the ranking loss function to update the parameters of the overall network.
在步骤S203中，通过反向传播算法更新所述卷积神经网络的参数。In step S203, the parameters of the convolutional neural network are updated by a back propagation algorithm.
对于同一个图像内容下的两幅不同感知质量的图像，卷积神经网络可以输出他们的排序分数。一般的，感知质量评价指标，例如NIQE越小则说明图像的感知质量越好，同一个图像内容下排序分数越小，则说明图像的感知质量越好。根据感知质量评价指标通过排序损失（margin-ranking loss）可以得到排序误差。通过反射传播算法（Backpropagation  Algorithm）可以将网络最终的排序误差传递到卷积神经网络的每一层，通过误差传递实现网络参数的训练和更新，最终使得排序误差最小化。For two images of different perceptual quality under the same image content, the convolutional neural network can output their ranking scores. Generally, the perceptual quality evaluation index, for example, the smaller the NIQE, the better the perceptual quality of the image, and the smaller the ranking score under the same image content, the better the perceptual quality of the image. According to the perceptual quality evaluation index, the ranking loss (margin-ranking loss) can get the sorting error. Through reflection propagation algorithm (Backpropagation) Algorithm) can transfer the final sorting error of the network to each layer of the convolutional neural network, and realize the training and update of network parameters through error transfer, and finally minimize the sorting error.
在步骤S104中，根据训练后的排序估计网络计算生成对抗网络的生成器所生成的图像的排序内容损失，根据所述排序内容损失指导生成对抗网络的生成器进行训练；In step S104, calculate the sequence content loss of the image generated by the generator of the confrontation network according to the trained sequence estimation network, and guide the generator of the generation of the confrontation network according to the sequence content loss for training;
对生成器进行训练的过程，可以如图4所示，包括：The process of training the generator, as shown in Figure 4, includes:
在步骤S401中，通过生成器生成高分辨率图像；In step S401, a high-resolution image is generated by the generator;
对于对抗网络生成器的训练，可以选用真实的超分辨率图像和对应的低分辨率图像作为训练样本，为了便于对图像进行计算，可以对训练样本中的同一对图像裁剪为具有相同内容的图像块，通过对图像块进行训练计算。为了对生成器进行训练，可以根据样本中的低分辨率图像或低分辨率图像的图像块，由所述生成器生成高分辨率图像。For the training of the adversarial network generator, you can use real super-resolution images and corresponding low-resolution images as training samples. To facilitate the calculation of the images, you can crop the same pair of images in the training sample into images with the same content. Block, through training calculation on the image block. In order to train the generator, the generator may generate a high-resolution image according to the low-resolution image or the image block of the low-resolution image in the sample.
在步骤S402中，根据所述训练后的排序估计网络计算所述高分辨率图像的排序内容损失；In step S402, the ranked content loss of the high-resolution image is calculated according to the trained ranking estimation network;
根据之前所训练后的排序估计网络，计算所述高分辨率图像的排序分数，根据所述排序分数得到所述高分辨率图像的排序内容损失。Calculate the ranking score of the high-resolution image according to the previously trained ranking estimation network, and obtain the ranking content loss of the high-resolution image according to the ranking score.
在步骤S403中，根据所述排序内容损失对生成器进行训练。In step S403, the generator is trained according to the sorted content loss.
将所述排序内容损失反馈至生成器，对所述生成器进行训练和更新。Feedback the sorted content loss to the generator, and train and update the generator.
优选的一种实施方式中，所述根据所述排序内容损失对生成器进行训练的步骤还包括：In a preferred embodiment, the step of training the generator according to the sorted content loss further includes:
通过判别器判断所生成的高分辨率图像与真实高分辨率图像的误差；Determine the error between the generated high-resolution image and the real high-resolution image through the discriminator;
通过VGG网络的特征空间计算生成的高分辨率图像与真实图像高分辨率图像的感知损失误差；The perceptual loss error of the high-resolution image generated by the feature space calculation of the VGG network and the real image high-resolution image;
联合所述排序内容损失、所述判别器判断的误差和所述感知损失误差，对生成器进行训练。The generator is trained by combining the sorted content loss, the discriminator error and the perceptual loss error.
在排序内容损失的基础上，结合判别器判别的误差，以及VGG网络的特征空间的感知损失误差，可以更为有效的对所述生成器进行训练，提高生成器的训练效率，以及提高超分辨率图像生成的真实度。Based on the loss of sorted content, combined with the discriminant error of the discriminator and the perceptual loss error of the feature space of the VGG network, the generator can be trained more effectively, the training efficiency of the generator can be improved, and the super-resolution can be improved. The realism of the rate image generation.
在步骤S105中，根据训练后的生成器对低分辨率图像进行超分辨率图像的重构。In step S105, super-resolution image reconstruction is performed on the low-resolution image according to the trained generator.
利用已训练完成的生成器，可以对低分辨率图像进行超分辨率的实现。Using the trained generator, you can implement super-resolution on low-resolution images.
由于本申请能够使用排序估计网络在具体的感知质量评价标准上直接优化进而得到更好的图像感知质量。表现为现有的超分辨率图像方法多为在均方误差或者特征空间上的误差来对超分辨率模型进行优化。在具体的感知质量评价标准上，对于此类方法的改进，是十分耗时耗力的，因为无论是对均方误差或特征空间上的误差优化都不能直接表示在具体感知质量上的提升。本申请提出了使用排序估计网络来约束超分辨率模型直接在具体的感知质量上进行优化，可以降低研发时间成本并且能够显著的提升重建高分辨率图像的感知质量。Because this application can use the ranking estimation network to directly optimize on specific perceptual quality evaluation standards to obtain better image perceptual quality. It is manifested that the existing super-resolution image methods mostly use the error in mean square error or feature space to optimize the super-resolution model. In the specific perceptual quality evaluation standard, the improvement of such methods is very time-consuming and laborious, because neither the mean square error nor the error optimization in the feature space can directly represent the improvement in specific perceptual quality. This application proposes to use a ranking estimation network to constrain the super-resolution model to directly optimize the specific perceptual quality, which can reduce the cost of development time and can significantly improve the perceptual quality of reconstructed high-resolution images.
并且，针对具体的图像感知质量，本发明能够拥有更强的拓展性。表现为排序估计网络可以学习具有不同先验信息的训练数据，能够灵活的根据需求来进行拓展。通过训练数据中不同感知质量数据的排序方式和自身的统计特性等，可以学习到不同的先验信息。具有不同先验信息的排序估计网络可以约束生成器生成出具有不同特点的超分辨率重建图像。Moreover, the present invention can have stronger expansibility for specific image perception quality. The performance is that the ranking estimation network can learn training data with different prior information, and can be flexibly expanded according to needs. Different prior information can be learned through the ordering of different perceptual quality data in training data and its own statistical characteristics. The ranking estimation network with different prior information can constrain the generator to generate super-resolution reconstruction images with different characteristics.
图5为本申请实施例提供的一种超排序生成对抗超分辨率图像网络的示意图，其主要由生成器，判别器，特征提取器和排序估计器组成，其中特征提取器和排序估计器可以为训练好的模型，可以只进行损失计算不进行模型的参数更新。由示意图可知，低分率图像可以通过生成器生成重构高分辨率图像，然后判别器可以通过原始高分辨率图像和重构的高分辨率图像计算对抗损失，特征提取器可以分别对重构高分辨率图像和原始高分辨率图像在特征空间上计算误差来得到感知损失误差，进一步，重构的高分辨率图像可以通过排序估计器计算其在具体感知质量评价标准下的排序分数进而得到排序内容损失。最后使用对抗损失，感知损失误差和排序内容损失来联合训练生成器。FIG. 5 is a schematic diagram of a super-rank generation anti-super-resolution image network provided by an embodiment of the present application, which is mainly composed of a generator, a discriminator, a feature extractor, and a rank estimator, where the feature extractor and rank estimator can For the trained model, only the loss calculation can be performed without updating the model parameters. It can be seen from the schematic diagram that the low-resolution image can be generated by the generator to generate a reconstructed high-resolution image, and then the discriminator can calculate the original high-resolution image and the reconstructed high-resolution image to combat loss, and the feature extractor can separately reconstruct The high-resolution image and the original high-resolution image calculate the error in the feature space to obtain the perceptual loss error. Furthermore, the reconstructed high-resolution image can be obtained by calculating the ranking score under the specific perceptual quality evaluation standard through the ranking estimator. Loss of sorted content. Finally, it uses joint loss, perceptual loss error and ordered content loss to jointly train the generator.
图5a为申请实施例提供的一种超分辨率图像的重构效果对比示意图，其中，左一为低分辨率图像，左二为基于像素损失的超分辨率方法生成的超分辨率图像，右二为传统的生成对抗超分辨率方法生成的超分辨率图像，右一为本申请所述超分辨率图像的重构方法所生成的超分辨率图像。根据实验结果可知，本申请所述超分辨率图像重构方法，能够从低分辨率图像中生成出更加真实的图像。5a is a schematic diagram of a comparison of reconstruction effects of a super-resolution image provided by an embodiment of the application, where the first left is a low-resolution image, the second left is a super-resolution image generated by a super-resolution method based on pixel loss, and the right The second is the traditional generation of the super-resolution image generated by the super-resolution resolution method, and the first one is the super-resolution image generated by the reconstruction method of the super-resolution image described in this application. According to the experimental results, the super-resolution image reconstruction method described in this application can generate a more realistic image from a low-resolution image.
图6为本申请实施例提供的一种超分辨率图像的重构装置的结构示意图，详述如下：FIG. 6 is a schematic structural diagram of a super-resolution image reconstruction device provided by an embodiment of the present application, and the details are as follows:
所述超分辨率图像的重构装置包括：The super-resolution image reconstruction device includes:
图像生成单元601，用于通过不同的超分辨率图像生成方法生成不同感知质量的图像；The image generating unit 601 is used to generate images with different perceptual qualities through different super-resolution image generating methods;
评价分数计算单元602，用于通过感知质量评价指标计算不同感知质量的图像的感知质量评价分数；The evaluation score calculation unit 602 is used to calculate the perception quality evaluation scores of images with different perception qualities through the perception quality evaluation indicators;
排序估计网络训练单元603，用于根据所计算的感知质量评价分数对排序估计网络进行训练；The ranking estimation network training unit 603 is used to train the ranking estimation network according to the calculated perceptual quality evaluation score;
生成器训练单元604，用于根据训练后的排序估计网络计算生成对抗网络的生成器所生成的图像的排序内容损失，根据所述排序内容损失指导生成对抗网络的生成器进行训练；The generator training unit 604 is configured to calculate the sequence content loss of the image generated by the generator of the confrontation network according to the trained sequence estimation network, and guide the generator of the generation of the confrontation network according to the sequence content loss for training;
重构单元605，用于根据训练后的生成器对低分辨率图像进行超分辨率图像的重构。The reconstruction unit 605 is configured to reconstruct the super-resolution image of the low-resolution image according to the trained generator.
优选的，所述评价分数计算单元包括：Preferably, the evaluation score calculation unit includes:
裁剪子单元，用于将所述不同感知质量的图像裁剪为图像块；A cropping subunit, configured to crop the images of different perceptual qualities into image blocks;
评价子单元，用于选择同一图像内容且具有不同感知质量的图像块构成图像对，通过无参考图像评价指标获取不同感知质量的图像对的感知质量评价分数。The evaluation subunit is used to select image blocks of the same image content and having different perceptual qualities to form an image pair, and to obtain the perceptual quality evaluation scores of image pairs of different perceptual qualities through a non-reference image evaluation index.
图6所述的高分辨率图像的重构装置，与图1所述的高分辨率图像的重构方法对应。The high-resolution image reconstruction device described in FIG. 6 corresponds to the high-resolution image reconstruction method described in FIG. 1.
图7是本申请一实施例提供的高分辨率图像的重构设备的示意图。如图7所示，该实施例的高分辨率图像的重构设备7包括：处理器70、存储器71以及存储在所述存储器71中并可在所述处理器70上运行的计算机程序72，例如高分辨率图像的重构程序。所述处理器70执行所述计算机程序72时实现上述各个高分辨率图像的重构方法实施例中的步骤。或者，所述处理器70执行所述计算机程序72时实现上述各装置实施例中各模块/单元的功能。7 is a schematic diagram of a high-resolution image reconstruction device provided by an embodiment of the present application. As shown in FIG. 7, the high-resolution image reconstruction device 7 of this embodiment includes: a processor 70, a memory 71, and a computer program 72 stored in the memory 71 and executable on the processor 70, For example, high-resolution image reconstruction procedures. When the processor 70 executes the computer program 72, the steps in the above embodiments of the method for reconstructing each high-resolution image are implemented. Alternatively, when the processor 70 executes the computer program 72, the functions of the modules/units in the foregoing device embodiments are realized.
示例性的，所述计算机程序72可以被分割成一个或多个模块/单元，所述一个或者多个模块/单元被存储在所述存储器71中，并由所述处理器70执行，以完成本申请。所述一个或多个模块/单元可以是能够完成特定功能的一系列计算机程序指令段，该指令段用于描述所述计算机程序72在所述高分辨率图像的重构设备7中的执行过程。例如，所述计算机程序72可以被分割成：Exemplarily, the computer program 72 may be divided into one or more modules/units, and the one or more modules/units are stored in the memory 71 and executed by the processor 70 to complete This application. The one or more modules/units may be a series of computer program instruction segments capable of performing specific functions, which are used to describe the execution process of the computer program 72 in the high-resolution image reconstruction device 7 . For example, the computer program 72 may be divided into:
图像生成单元，用于通过不同的超分辨率图像生成方法生成不同感知质量的图像；Image generation unit for generating images of different perceptual qualities through different super-resolution image generation methods;
评价分数计算单元，用于通过感知质量评价指标计算不同感知质量的图像的感知质量评价分数；The evaluation score calculation unit is used to calculate the perception quality evaluation score of images with different perception qualities through the perception quality evaluation index;
排序估计网络训练单元，用于根据所计算的感知质量评价分数对排序估计网络进行训练；The ranking estimation network training unit is used to train the ranking estimation network according to the calculated perceptual quality evaluation score;
生成器训练单元，用于根据训练后的排序估计网络计算生成对抗网络的生成器所生成的图像的排序内容损失，根据所述排序内容损失指导生成对抗网络的生成器进行训练；The generator training unit is configured to calculate the sequence content loss of the image generated by the generator of the confrontation network according to the trained sequence estimation network, and guide the generator of the confrontation network generation training according to the sequence content loss;
重构单元，用于根据训练后的生成器对低分辨率图像进行超分辨率图像的重构。The reconstruction unit is used to reconstruct the super-resolution image of the low-resolution image according to the trained generator.
所述高分辨率图像的重构设备7可以是桌上型计算机、笔记本、掌上电脑及云端服务器等计算设备。所述高分辨率图像的重构设备可包括，但不仅限于，处理器70、存储器71。本领域技术人员可以理解，图7仅仅是高分辨率图像的重构设备7的示例，并不构成对高分辨率图像的重构设备7的限定，可以包括比图示更多或更少的部件，或者组合某些部件，或者不同的部件，例如所述高分辨率图像的重构设备还可以包括输入输出设备、网络接入设备、总线等。The reconstruction device 7 of the high-resolution image may be a computing device such as a desktop computer, a notebook, a palmtop computer and a cloud server. The high-resolution image reconstruction device may include, but is not limited to, the processor 70 and the memory 71. Those skilled in the art may understand that FIG. 7 is only an example of the high-resolution image reconstruction device 7 and does not constitute a limitation on the high-resolution image reconstruction device 7, and may include more or less than the illustration. Components, or a combination of certain components, or different components, for example, the high-resolution image reconstruction device may further include an input and output device, a network access device, a bus, and the like.
所称处理器70可以是中央处理单元(Central Processing Unit，CPU)，还可以是其他通用处理器、数字信号处理器 (Digital Signal Processor，DSP)、专用集成电路 (Application Specific Integrated Circuit，ASIC)、现成可编程门阵列 (Field-Programmable Gate Array，FPGA) 或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件等。通用处理器可以是微处理器或者该处理器也可以是任何常规的处理器等。The so-called processor 70 may be a central processing unit (Central Processing Unit (CPU), can also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (Application Specific Integrated Circuit (ASIC), ready-made programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
所述存储器71可以是所述高分辨率图像的重构设备7的内部存储单元，例如高分辨率图像的重构设备7的硬盘或内存。所述存储器71也可以是所述高分辨率图像的重构设备7的外部存储设备，例如所述高分辨率图像的重构设备7上配备的插接式硬盘，智能存储卡（Smart Media Card, SMC），安全数字（Secure Digital, SD）卡，闪存卡（Flash Card）等。进一步地，所述存储器71还可以既包括所述高分辨率图像的重构设备7的内部存储单元也包括外部存储设备。所述存储器71用于存储所述计算机程序以及所述高分辨率图像的重构设备所需的其他程序和数据。所述存储器71还可以用于暂时地存储已经输出或者将要输出的数据。The memory 71 may be an internal storage unit of the high-resolution image reconstruction device 7, for example, a hard disk or a memory of the high-resolution image reconstruction device 7. The memory 71 may also be an external storage device of the high-resolution image reconstruction device 7, for example, a plug-in hard disk equipped on the high-resolution image reconstruction device 7, a smart memory card (Smart Media Card , SMC), Secure Digital (SD) card, flash memory card (Flash Card) etc. Further, the memory 71 may also include both the internal storage unit of the high-resolution image reconstruction device 7 and the external storage device. The memory 71 is used to store the computer program and other programs and data required by the high-resolution image reconstruction device. The memory 71 can also be used to temporarily store data that has been or will be output.
所属领域的技术人员可以清楚地了解到，为了描述的方便和简洁，仅以上述各功能单元、模块的划分进行举例说明，实际应用中，可以根据需要而将上述功能分配由不同的功能单元、模块完成，即将所述装置的内部结构划分成不同的功能单元或模块，以完成以上描述的全部或者部分功能。实施例中的各功能单元、模块可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中，上述集成的单元既可以采用硬件的形式实现，也可以采用软件功能单元的形式实现。另外，各功能单元、模块的具体名称也只是为了便于相互区分，并不用于限制本申请的保护范围。上述***中单元、模块的具体工作过程，可以参考前述方法实施例中的对应过程，在此不再赘述。Those skilled in the art can clearly understand that, for convenience and conciseness of description, only the above-mentioned division of each functional unit and module is used as an example for illustration. In practical applications, the above-mentioned functions may be allocated by different functional units, Module completion means that the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments may be integrated into one processing unit, or each unit may exist alone physically, or two or more units may be integrated into one unit. The above integrated unit may use hardware It can also be implemented in the form of software functional units. In addition, the specific names of the functional units and modules are only for the purpose of distinguishing each other, and are not intended to limit the protection scope of the present application. For the specific working processes of the units and modules in the above system, reference may be made to the corresponding processes in the foregoing method embodiments, which will not be repeated here.
在上述实施例中，对各个实施例的描述都各有侧重，某个实施例中没有详述或记载的部分，可以参见其它实施例的相关描述。In the above embodiments, the description of each embodiment has its own emphasis. For a part that is not detailed or recorded in an embodiment, you can refer to the related descriptions of other embodiments.
本领域普通技术人员可以意识到，结合本文中所公开的实施例描述的各示例的单元及算法步骤，能够以电子硬件、或者计算机软件和电子硬件的结合来实现。这些功能究竟以硬件还是软件方式来执行，取决于技术方案的特定应用和设计约束条件。专业技术人员可以对每个特定的应用来使用不同方法来实现所描述的功能，但是这种实现不应认为超出本申请的范围。Those of ordinary skill in the art may realize that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in hardware or software depends on the specific application of the technical solution and design constraints. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
在本申请所提供的实施例中，应该理解到，所揭露的装置/终端设备和方法，可以通过其它的方式实现。例如，以上所描述的装置/终端设备实施例仅仅是示意性的，例如，所述模块或单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如多个单元或组件可以结合或者可以集成到另一个***，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通讯连接可以是通过一些接口，装置或单元的间接耦合或通讯连接，可以是电性，机械或其它的形式。In the embodiments provided in this application, it should be understood that the disclosed device/terminal device and method may be implemented in other ways. For example, the device/terminal device embodiments described above are only schematic. For example, the division of the module or unit is only a logical function division, and in actual implementation, there may be another division manner, such as multiple units Or components can be combined or integrated into another system, or some features can be ignored or not implemented. In addition, the displayed or discussed mutual coupling or direct coupling or communication connection may be indirect coupling or communication connection through some interfaces, devices or units, and may be electrical, mechanical or other forms.
所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的目的。The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
另外，在本申请各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。上述集成的单元既可以采用硬件的形式实现，也可以采用软件功能单元的形式实现。In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units may be integrated into one unit. The above integrated unit can be implemented in the form of hardware or software function unit.
所述集成的模块/单元如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个计算机可读取存储介质中。基于这样的理解，本申请实现上述实施例方法中的全部或部分流程，也可以通过计算机程序来指令相关的硬件来完成，所述的计算机程序可存储于一计算机可读存储介质中，该计算机程序在被处理器执行时，可实现上述各个方法实施例的步骤。。其中，所述计算机程序包括计算机程序代码，所述计算机程序代码可以为源代码形式、对象代码形式、可执行文件或某些中间形式等。所述计算机可读介质可以包括：能够携带所述计算机程序代码的任何实体或装置、记录介质、U盘、移动硬盘、磁碟、光盘、计算机存储器、只读存储器（ROM，Read-Only Memory）、随机存取存储器（RAM，Random Access Memory）、电载波信号、电信信号以及软件分发介质等。需要说明的是，所述计算机可读介质包含的内容可以根据司法管辖区内立法和专利实践的要求进行适当的增减，例如在某些司法管辖区，根据立法和专利实践，计算机可读介质不包括是电载波信号和电信信号。If the integrated module/unit is implemented in the form of a software functional unit and sold or used as an independent product, it may be stored in a computer-readable storage medium. Based on this understanding, the present application can implement all or part of the processes in the methods of the above embodiments, or it can be completed by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium. When the program is executed by the processor, the steps of the foregoing method embodiments may be implemented. . Wherein, the computer program includes computer program code, and the computer program code may be in a source code form, an object code form, an executable file, or some intermediate form, etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a U disk, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signals, telecommunications signals, and software distribution media. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in jurisdictions. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media Excluded are electrical carrier signals and telecommunications signals.
以上所述实施例仅用以说明本申请的技术方案，而非对其限制；尽管参照前述实施例对本申请进行了详细的说明，本领域的普通技术人员应当理解：其依然可以对前述各实施例所记载的技术方案进行修改，或者对其中部分技术特征进行等同替换；而这些修改或者替换，并不使相应技术方案的本质脱离本申请各实施例技术方案的精神和范围，均应包含在本申请的保护范围之内。The above-mentioned embodiments are only used to illustrate the technical solutions of the present application, not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, persons of ordinary skill in the art should understand that they can still implement the foregoing The technical solutions described in the examples are modified, or some of the technical features are equivalently replaced; and these modifications or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present application. Within the scope of protection of this application.

Claims (10)

  1. 一种超分辨率图像的重构方法，其特征在于，所述超分辨率图像的重构方法包括：A super-resolution image reconstruction method, characterized in that the super-resolution image reconstruction method includes:
    通过不同的超分辨率图像生成方法生成不同感知质量的图像；Generate images with different perceptual qualities through different super-resolution image generation methods;
    通过感知质量评价指标计算不同感知质量的图像的感知质量评价分数；Calculate the perceptual quality evaluation score of images with different perceptual quality through the perceptual quality evaluation index;
    根据所计算的感知质量评价分数对排序估计网络进行训练；Train the ranking estimation network according to the calculated perceptual quality evaluation score;
    根据训练后的排序估计网络计算生成对抗网络的生成器所生成的图像的排序内容损失，根据所述排序内容损失指导生成对抗网络的生成器进行训练；Calculating the sequence content loss of the image generated by the generator of the confrontation network according to the trained sequence estimation network, and instructing the generator of generating the confrontation network according to the sequence content loss for training;
    根据训练后的生成器对低分辨率图像进行超分辨率图像的重构。The low-resolution image is reconstructed according to the trained generator.
  2. 根据权利要求1所述的超分辨率图像的重构方法，其特征在于，所述通过感知质量评价指标计算不同感知质量的图像的感知质量评价分数的步骤包括：The method for reconstructing a super-resolution image according to claim 1, wherein the step of calculating a perceptual quality evaluation score of images of different perceptual quality through a perceptual quality evaluation index includes:
    将所述不同感知质量的图像裁剪为图像块；Crop the images of different perceptual qualities into image blocks;
    选择同一图像内容且具有不同感知质量的图像块构成图像对，通过无参考图像评价指标获取不同感知质量的图像对的感知质量评价分数。Image blocks with the same image content and different perceptual qualities are selected to form image pairs, and the perceptual quality evaluation scores of image pairs with different perceptual qualities are obtained through the non-reference image evaluation index.
  3. 根据权利要求2所述的超分辨率图像的重构方法，其特征在于，所述根据所计算的感知质量评价分数对排序估计网络进行训练的步骤包括：The method for reconstructing a super-resolution image according to claim 2, wherein the step of training the ranking estimation network according to the calculated perceptual quality evaluation score includes:
    选择作为排序估计网络的卷积神经网络计算所述图像的排序分数；Selecting a convolutional neural network as the ranking estimation network to calculate the ranking score of the image;
    根据感知质量评价指标计算的评价分数和所述排序分数，得到所述卷积神经网络的排序误差；According to the evaluation score calculated by the perceptual quality evaluation index and the ranking score, the ranking error of the convolutional neural network is obtained;
    通过反向传播算法更新所述卷积神经网络的参数。The parameters of the convolutional neural network are updated by a back propagation algorithm.
  4. 根据权利要求1所述的超分辨率图像的重构方法，其特征在于，所述根据训练后的排序估计网络计算生成对抗网络的生成器所生成的图像的排序内容损失，根据所述排序内容损失指导生成对抗网络的生成器进行训练的步骤包括：The method for reconstructing a super-resolution image according to claim 1, characterized in that, based on the trained ranking estimation network, the ranking content loss of the image generated by the generator against the network is calculated and calculated according to the ranking content The steps for the loss-guided generation adversarial network generator training include:
    通过生成器生成高分辨率图像；Generate high-resolution images through the generator;
    根据所述训练后的排序估计网络计算所述高分辨率图像的排序内容损失；Calculating the ranking content loss of the high-resolution image according to the trained ranking estimation network;
    根据所述排序内容损失对生成器进行训练。The generator is trained according to the sorted content loss.
  5. 根据权利要求4所述的超分辨率图像的重构方法，其特征在于，所述根据所述排序内容损失对生成器进行训练的步骤还包括：The method for reconstructing a super-resolution image according to claim 4, wherein the step of training the generator according to the loss of sorted content further comprises:
    通过判别器判断所生成的高分辨率图像与真实高分辨率图像的误差；Determine the error between the generated high-resolution image and the real high-resolution image through the discriminator;
    通过VGG网络的特征空间计算生成的高分辨率图像与真实图像高分辨率图像的感知损失误差；The perceptual loss error of the high-resolution image generated by the feature space calculation of the VGG network and the real image high-resolution image;
    联合所述排序内容损失、所述判别器判断的误差和所述感知损失误差，对生成器进行训练。The generator is trained by combining the sorted content loss, the discriminator error and the perceptual loss error.
  6. 根据权利要求1所述的超分辨率图像的重构方法，其特征在于，在所述根据所计算的感知质量评价分数对排序估计网络进行训练的步骤之前，所述方法还包括：The method for reconstructing a super-resolution image according to claim 1, wherein before the step of training the ranking estimation network according to the calculated perceptual quality evaluation score, the method further comprises:
    计算不同感知质量的图像的差异，选择所述图像的差异大于预设的阈值的图像。Calculate the difference of images with different perceptual qualities, and select an image whose difference between the images is greater than a preset threshold.
  7. 一种超分辨率图像的重构装置，其特征在于，所述超分辨率图像的重构装置包括：A super-resolution image reconstruction device, characterized in that the super-resolution image reconstruction device includes:
    图像生成单元，用于通过不同的超分辨率图像生成方法生成不同感知质量的图像；Image generation unit for generating images of different perceptual qualities through different super-resolution image generation methods;
    评价分数计算单元，用于通过感知质量评价指标计算不同感知质量的图像的感知质量评价分数；The evaluation score calculation unit is used to calculate the perception quality evaluation score of images with different perception qualities through the perception quality evaluation index;
    排序估计网络训练单元，用于根据所计算的感知质量评价分数对排序估计网络进行训练；The ranking estimation network training unit is used to train the ranking estimation network according to the calculated perceptual quality evaluation score;
    生成器训练单元，用于根据训练后的排序估计网络计算生成对抗网络的生成器所生成的图像的排序内容损失，根据所述排序内容损失指导生成对抗网络的生成器进行训练；The generator training unit is configured to calculate the sequence content loss of the image generated by the generator of the confrontation network according to the trained sequence estimation network, and guide the generator of the confrontation network generation training according to the sequence content loss;
    重构单元，用于根据训练后的生成器对低分辨率图像进行超分辨率图像的重构。The reconstruction unit is used to reconstruct the super-resolution image of the low-resolution image according to the trained generator.
  8. 根据权利要求7所述的超分辨率图像的重构装置，其特征在于，所述评价分数计算单元包括：The apparatus for reconstructing a super-resolution image according to claim 7, wherein the evaluation score calculation unit includes:
    裁剪子单元，用于将所述不同感知质量的图像裁剪为图像块；A cropping subunit, configured to crop the images of different perceptual qualities into image blocks;
    评价子单元，用于选择同一图像内容且具有不同感知质量的图像块构成图像对，通过无参考图像评价指标获取不同感知质量的图像对的感知质量评价分数。The evaluation subunit is used to select image blocks of the same image content and having different perceptual qualities to form an image pair, and to obtain the perceptual quality evaluation scores of image pairs of different perceptual qualities through a non-reference image evaluation index.
  9. 一种高分辨率图像的重构设备，包括存储器、处理器以及存储在所述存储器中并可在所述处理器上运行的计算机程序，其特征在于，所述处理器执行所述计算机程序时实现如权利要求1至6任一项所述高分辨率图像的重构方法的步骤。A high-resolution image reconstruction device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that when the processor executes the computer program The steps of implementing the high-resolution image reconstruction method according to any one of claims 1 to 6.
  10. 一种计算机可读存储介质，所述计算机可读存储介质存储有计算机程序，其特征在于，所述计算机程序被处理器执行时实现如权利要求1至6任一项所述高分辨率图像的重构方法的步骤。A computer-readable storage medium storing a computer program, characterized in that when the computer program is executed by a processor, a high-resolution image according to any one of claims 1 to 6 is realized Refactoring method steps.
PCT/CN2019/130826 2019-01-12 2019-12-31 Super-resolution image reconstruction method, apparatus and device WO2020143513A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201910029164.XA CN109949219B (en) 2019-01-12 2019-01-12 Reconstruction method, device and equipment of super-resolution image
CN201910029164.X 2019-01-12

Publications (1)

Publication Number Publication Date
WO2020143513A1 true WO2020143513A1 (en) 2020-07-16

Family

ID=67007311

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/130826 WO2020143513A1 (en) 2019-01-12 2019-12-31 Super-resolution image reconstruction method, apparatus and device

Country Status (2)

Country Link
CN (1) CN109949219B (en)
WO (1) WO2020143513A1 (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112837232A (en) * 2021-01-13 2021-05-25 山东省科学院海洋仪器仪表研究所 Underwater image enhancement and detail recovery method

Families Citing this family (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109949219B (en) * 2019-01-12 2021-03-26 深圳先进技术研究院 Reconstruction method, device and equipment of super-resolution image
CN110363792A (en) * 2019-07-19 2019-10-22 广东工业大学 A kind of method for detecting change of remote sensing image based on illumination invariant feature extraction
CN112365553B (en) * 2019-07-24 2022-05-20 北京新唐思创教育科技有限公司 Human body image generation model training, human body image generation method and related device
CN110490802B (en) * 2019-08-06 2021-01-19 北京观微科技有限公司 Super-resolution-based satellite image airplane target model identification method
CN112446835B (en) * 2019-09-04 2024-06-18 华为技术有限公司 Image restoration method, image restoration network training method, device and storage medium
CN110958722B (en) * 2019-11-29 2023-05-09 洛阳理工学院 Wireless image sensor network energy saving method based on super-resolution technology under cloud assistance
CN111402137B (en) * 2020-03-20 2023-04-18 南京信息工程大学 Depth attention coding and decoding single image super-resolution algorithm based on perception loss guidance
CN111932444B (en) * 2020-07-16 2023-09-19 中国石油大学(华东) Face attribute editing method based on generation countermeasure network and information processing terminal

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180075581A1 (en) * 2016-09-15 2018-03-15 Twitter, Inc. Super resolution using a generative adversarial network
CN108550115A (en) * 2018-04-25 2018-09-18 中国矿业大学 A kind of image super-resolution rebuilding method
CN108921788A (en) * 2018-06-20 2018-11-30 华北电力大学 Image super-resolution method, device and storage medium based on deep layer residual error CNN
CN109034365A (en) * 2018-07-06 2018-12-18 电子科技大学 The training method and device of deep learning model
CN109949219A (en) * 2019-01-12 2019-06-28 深圳先进技术研究院 A kind of reconstructing method of super-resolution image, device and equipment

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10210635B2 (en) * 2016-06-29 2019-02-19 Siemens Medical Solutions Usa, Inc. Reconstruction quality assessment with local non-uniformity in nuclear imaging
CN106683048B (en) * 2016-11-30 2020-09-01 浙江宇视科技有限公司 Image super-resolution method and device
CN107194872B (en) * 2017-05-02 2019-08-20 武汉大学 Remote sensed image super-resolution reconstruction method based on perception of content deep learning network
CN108830796B (en) * 2018-06-20 2021-02-02 重庆大学 Hyperspectral image super-resolution reconstruction method based on spectral-spatial combination and gradient domain loss
CN109003265B (en) * 2018-07-09 2022-02-11 嘉兴学院 No-reference image quality objective evaluation method based on Bayesian compressed sensing

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180075581A1 (en) * 2016-09-15 2018-03-15 Twitter, Inc. Super resolution using a generative adversarial network
CN108550115A (en) * 2018-04-25 2018-09-18 中国矿业大学 A kind of image super-resolution rebuilding method
CN108921788A (en) * 2018-06-20 2018-11-30 华北电力大学 Image super-resolution method, device and storage medium based on deep layer residual error CNN
CN109034365A (en) * 2018-07-06 2018-12-18 电子科技大学 The training method and device of deep learning model
CN109949219A (en) * 2019-01-12 2019-06-28 深圳先进技术研究院 A kind of reconstructing method of super-resolution image, device and equipment

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
SIWEI HUANG: "Research on Super-Resolution Image Reconstruction Algorithm Based on Deep Learning", CHINA MASTER’S THESES FULL-TEXT DATABASE, 15 November 2018 (2018-11-15), pages 1 - 84, XP055718913 *

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112837232A (en) * 2021-01-13 2021-05-25 山东省科学院海洋仪器仪表研究所 Underwater image enhancement and detail recovery method

Also Published As

Publication number Publication date
CN109949219B (en) 2021-03-26
CN109949219A (en) 2019-06-28

Similar Documents

Publication Publication Date Title
WO2020143513A1 (en) Super-resolution image reconstruction method, apparatus and device
WO2020199693A1 (en) Large-pose face recognition method and apparatus, and device
EP3828765A1 (en) Human body detection method and apparatus, computer device, and storage medium
CN111325851B (en) Image processing method and device, electronic equipment and computer readable storage medium
US20220084163A1 (en) Target image generation method and apparatus, server, and storage medium
US20230120253A1 (en) Method and apparatus for generating virtual character, electronic device and readable storage medium
CN114339409B (en) Video processing method, device, computer equipment and storage medium
CN113327278A (en) Three-dimensional face reconstruction method, device, equipment and storage medium
US20220358675A1 (en) Method for training model, method for processing video, device and storage medium
CN115147265B (en) Avatar generation method, apparatus, electronic device, and storage medium
JP2023545189A (en) Image processing methods, devices, and electronic equipment
CN115564642A (en) Image conversion method, image conversion device, electronic apparatus, storage medium, and program product
CN110717405B (en) Face feature point positioning method, device, medium and electronic equipment
CN114792355A (en) Virtual image generation method and device, electronic equipment and storage medium
CN113610958A (en) 3D image construction method and device based on style migration and terminal
CN115984949B (en) Low-quality face image recognition method and equipment with attention mechanism
WO2022236802A1 (en) Method and apparatus for reconstructing object model, and terminal device and storage medium
EP4086853A2 (en) Method and apparatus for generating object model, electronic device and storage medium
CN113298931B (en) Reconstruction method and device of object model, terminal equipment and storage medium
CN115775300A (en) Reconstruction method of human body model, training method and device of human body reconstruction model
CN113610856B (en) Method and device for training image segmentation model and image segmentation
CN112785498B (en) Pathological image superscore modeling method based on deep learning
CN113139490B (en) Image feature matching method and device, computer equipment and storage medium
CN112508772B (en) Image generation method, device and storage medium
CN117197063B (en) Point cloud quality evaluation method based on multi-view projection and transducer model and related products

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19909469

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19909469

Country of ref document: EP

Kind code of ref document: A1

122 Ep: pct application non-entry in european phase

Ref document number: 19909469

Country of ref document: EP

Kind code of ref document: A1