WO2023072072A1 - Blurred image generating method and apparatus, and network model training method and apparatus - Google Patents

Blurred image generating method and apparatus, and network model training method and apparatus Download PDF

Info

Publication number
WO2023072072A1
WO2023072072A1 PCT/CN2022/127384 CN2022127384W WO2023072072A1 WO 2023072072 A1 WO2023072072 A1 WO 2023072072A1 CN 2022127384 W CN2022127384 W CN 2022127384W WO 2023072072 A1 WO2023072072 A1 WO 2023072072A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
images
blur kernel
blurred
model
Prior art date
Application number
PCT/CN2022/127384
Other languages
French (fr)
Chinese (zh)
Inventor
董航
Original Assignee
北京字跳网络技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 北京字跳网络技术有限公司 filed Critical 北京字跳网络技术有限公司
Publication of WO2023072072A1 publication Critical patent/WO2023072072A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/77—Retouching; Inpainting; Scratch removal
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/60—Image enhancement or restoration using machine learning, e.g. neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/73—Deblurring; Sharpening
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning

Definitions

  • the present application relates to the technical field of image processing, in particular to a blurred image generation method, a network model training method and a device.
  • Video repair task is the key business in video quality enhancement.
  • the most widely used method of video repair is: obtain a training data set, use the obtained training data set to train the network model to obtain a video repair network model, and finally perform video repair through the video repair network model. Since the training data set directly determines the model performance of the obtained video inpainting network model, how to obtain a training data set that is more consistent with the real data has become one of the research hotspots in this field.
  • the method commonly used in the related art is: acquire a high-quality image set including multiple clear images, then use degraded methods such as Bicubic downsampling to generate blurred images corresponding to each clear image in the high-quality image set, and finally combine the high-quality image set
  • degraded methods such as Bicubic downsampling
  • the clear images in and the corresponding blurred images are used as the training data set.
  • the blurred image obtained by degrading methods such as Bicubic downsampling is very different from the real blurred image. Since the blurred images in the training data set acquired through the training data set generation method in the related art are very different from the real blurred images, the performance of the trained video inpainting network model is very unsatisfactory.
  • the present application provides a blurred image generation method, network model training method and device, which are used to solve the problem that the blurred image in the training data set obtained in the related art is very different from the real blurred image.
  • the embodiments of the present application provide a method for generating a blurred image, including:
  • the first image collection includes a plurality of images with a resolution smaller than the first threshold;
  • the second image set includes a plurality of images with a resolution greater than a second threshold
  • Each image in the second image set is degraded through a blur kernel corresponding to each image in the second image set, and a blurred image corresponding to each image in the second image set is acquired.
  • the embodiments of the present application provide a network model training method, including:
  • the set of sample images including a plurality of sample images with a resolution greater than a threshold resolution
  • the blurred image corresponding to each sample image in the sample image set is obtained;
  • An image inpainting network model for inpainting blurred images is trained by the training data set.
  • an embodiment of the present application provides a device for generating a blurred image, including:
  • An acquisition unit configured to acquire the blur kernel of each image in the first image set; generate a blur kernel pool, the first image set includes a plurality of images with a resolution smaller than a first threshold;
  • a selection unit configured to select a blur kernel corresponding to each image in a second image set from the blur kernel pool; the second image set includes a plurality of images with a resolution greater than a second threshold;
  • a processing unit configured to degrade each image in the second image set through a blur kernel corresponding to each image in the second image set, and acquire a blurred image corresponding to each image in the second image set.
  • the embodiment of the present application provides a network model training device, including:
  • an acquiring unit configured to acquire a sample image set, the sample image set including a plurality of sample images with a resolution greater than a threshold resolution
  • a processing unit configured to obtain a blurred image corresponding to each sample image in the sample image set by using the blurred image generation method described in any one of the first aspect
  • a generation unit configured to generate a training data set according to each sample image in the sample image set and the blurred image corresponding to each sample image
  • the training unit is configured to train a preset network model through the training data set, and obtain an image repair network model for repairing blurred images.
  • an embodiment of the present application provides an electronic device, including: a memory and a processor, the memory is used to store a computer program; the processor is used to enable the electronic device to implement the first Aspect or the method described in any embodiment of the first aspect.
  • the embodiments of the present application provide a computer-readable storage medium.
  • the computing device implements the method described in the first aspect or any implementation manner of the first aspect.
  • an embodiment of the present application provides a computer program product, which enables the computer to implement the method described in the first aspect or any implementation manner of the first aspect when the computer program product is run on a computer.
  • the blurred image generation method provided in the embodiment of the present application first obtains the blur kernels of each image in the first image set of multiple images with a resolution smaller than the first threshold to generate a blur kernel pool, and then selects a blur kernel pool including multiple A blur kernel corresponding to each image in the second image set of images with a resolution greater than the second threshold, and then degrades each image in the second image set through the blur kernel corresponding to each image in the second image set quality, acquiring blurred images corresponding to each image in the second image set.
  • the embodiment of the present application can solve the problem that the blurred image acquired in the related art is very different from the real blurred image. question.
  • Fig. 1 is one of the flow charts of the steps of the blurred image generation method provided by the embodiment of the present application;
  • FIG. 2 is the second schematic diagram of the data flow of the blurred image generation method provided by the embodiment of the present application.
  • FIG. 3 is a model frame diagram of a blurred image generation method provided by an embodiment of the present application.
  • FIG. 4 is a flow chart of the steps of the network model training method provided by the embodiment of the present application.
  • FIG. 5 is a schematic structural diagram of a blurred image generation device provided in an embodiment of the present application.
  • FIG. 6 is a schematic structural diagram of a network model training device provided in an embodiment of the present application.
  • FIG. 7 is a schematic diagram of a hardware structure of an electronic device provided by an embodiment of the present application.
  • words such as “exemplary” or “for example” are used as examples, illustrations or illustrations. Any embodiment or design scheme described as “exemplary” or “for example” in the embodiments of the present application shall not be interpreted as being more preferred or more advantageous than other embodiments or design schemes. Rather, the use of words such as “exemplary” or “such as” is intended to present related concepts in a concrete manner.
  • the meaning of "plurality” refers to two or more.
  • the embodiment of the present application provides a method for generating a blurred image.
  • the method for generating a blurred image includes the following steps:
  • the first set of images includes a plurality of images with resolutions smaller than a first threshold.
  • the implementation process of the above step S11 may include: obtaining a plurality of low-resolution images with a resolution smaller than the first threshold to form a first image set, and then extracting the blur kernel of each low-resolution image in the first image set, Finally, all the extracted fuzzy kernels are combined into a fuzzy kernel pool.
  • the first threshold in the embodiment of the present application can be determined according to the usage scenario of the video repair network model obtained from the final training. If the video repair network model obtained from the final training is used to repair a video with lower definition, the first The threshold is set smaller, and if the video inpainting network model finally trained is used to repair a video with higher definition, the first threshold can be set larger.
  • the first image set in the embodiment of the present application may consist of multiple mutually independent images, or may be multiple video frames obtained by sampling image frames from the same video, or may be a set of images in the same video All the image frames are not limited in this embodiment of the present application, as long as the resolutions of the images in the first image set are all smaller than the first threshold.
  • the second image set includes a plurality of images with a resolution greater than a second threshold.
  • one blur kernel is selected from the blur kernel pool as the corresponding blur kernel.
  • the above step S13 (degrade each image in the second image set through the blur kernel corresponding to each image in the second image set, and obtain the corresponding blur kernel of each image in the second image set blurred image) including:
  • the first formula is:
  • I x Deg(J x , K x )+N
  • J x represents the image in the second image collection
  • I x represents the blurred image corresponding to the image J x in the second image collection
  • K x represents the blur kernel corresponding to the image J x in the second image collection
  • Deg(J x , K x ) means that J x is degraded by K x
  • N means additional noise added to I x .
  • the blurred image generation method provided in the embodiment of the present application first obtains the blur kernels of each image in the first image set of multiple images with a resolution smaller than the first threshold to generate a blur kernel pool, and then selects a blur kernel pool including multiple A blur kernel corresponding to each image in the second image set of images with a resolution greater than the second threshold, and then degrades each image in the second image set through the blur kernel corresponding to each image in the second image set quality, acquiring blurred images corresponding to each image in the second image set.
  • the embodiment of the present application can solve the problem that the blurred image acquired in the related art is very different from the real blurred image. question.
  • step S11 obtaining the blur kernel of each image in the first image set
  • step S11 may include the following steps:
  • both the first noise and the second noise satisfy normal distribution.
  • the DIP model is a network model constructed by using the network structure itself to capture a large amount of low-level image statistical prior information; random noise is used as the input of the DIP model, and as the number of iterations of the DIP model increases, the DIP model can output the corresponding Therefore, the first image can be output by the DIP model after the first noise is input into the DIP model in the above step S22.
  • a reversible network can be constructed, and the process from random noise to the corresponding fuzzy kernel can be obtained through pre-training.
  • the trained FKP model is input with a normal distribution of noise, it can get to a real fuzzy kernel. Therefore, the above step S23 After inputting the second noise into the FKP model, the real blur kernel predicted by the FKP model can be obtained.
  • the above step S24 (degrading the first image through the predictive blur kernel to obtain a second image) includes:
  • the first formula is:
  • J y represents the first image
  • I y represents the second image
  • k represents the prediction blur kernel
  • DegDeg(J y , k) represents the operation of degrading the first image through k
  • N represents the additional input to the second image Additional noise added.
  • the image restoration task generally does not need to increase the resolution of the blurred image, it is not necessary to down-sample the first image in the process of degrading the first image through the predictive blur kernel to obtain the second image .
  • an L1loss constraint may be performed on the second image and the target image, so as to determine whether the convergence condition is satisfied.
  • step S25 if the second image and the target image do not meet the convergence condition, then perform the following step S26, and if the second image and the target image meet the convergence condition, then perform the following step S27 .
  • the predictive blur kernel output by the FKP model degrades the first image through the predictive blur kernel, acquires a second image, and judges whether a convergence condition is satisfied based on the second image and the target image. That is, as shown in FIG. 2, after updating the model parameters and/or model input, return to step S22.
  • said updating model parameters and/or model inputs includes:
  • model parameters of the DIP model and/or the second noise are updated during the training process, but the model parameters of the FKP model or the model input (first noise) of the DIP model are not updated.
  • the blur kernel of each image in the first image set can be obtained, and then the blur kernel pool in the above embodiment is generated.
  • the implementation process of acquiring the target image of the first image set shown in FIG. 2 includes:
  • the first image g(z x , ⁇ g ) is degraded by predicting the blur kernel k(z k , ⁇ k ), to obtain the second image P.
  • an implementation of the above step S12 (selecting the blur kernel corresponding to each image in the second image set from the blur kernel pool) includes:
  • an implementation of the above step S12 (selecting the blur kernel corresponding to each image in the second image set from the blur kernel pool) includes the following steps a to d:
  • Step a Divide the images in the first image set into multiple first sub-image sets based on the scene of the image.
  • images with the same image scene in the first image set are divided into a first sub-image set, so as to obtain multiple first sub-image sets.
  • the first set of images consists of image frames of a first video.
  • the image-based scene divides the images in the first image set into a plurality of first sub-image sets, comprising:
  • scene transition detection can be performed on the video, so that the video can be divided into a plurality of first video segments, and then each first video segment is extracted
  • the image frames of the segment are divided into a first sub-image set, so that the images in the first image set are divided into multiple first sub-image sets according to the scenes of the images.
  • Step b Divide blur kernels of images belonging to the same first sub-image set into a blur kernel group.
  • the blur kernels of the two images belong to the same blur kernel group, and if the two images belong to different the first sub-image set, the blur kernels of the two images belong to different blur kernel groups.
  • Step c dividing the images in the second image set into multiple second sub-image sets based on the scene of the image.
  • images with the same image scene in the second image set are divided into a second sub-image set, so as to obtain multiple second sub-image sets.
  • the second set of images consists of image frames of a second video.
  • the image-based scene divides the images in the second image set into a plurality of second sub-image sets, comprising:
  • scene transition detection can be performed on the video, thereby dividing the video into a plurality of second video segments, and then extracting each second video
  • the image frames of the segment are divided into a second sub-image set, so that the images in the second image set are divided into multiple second sub-image sets according to the scenes of the images.
  • Step d For images belonging to the same second sub-image set, randomly select a corresponding blur kernel from the same blur kernel group.
  • the above embodiments can further reduce or avoid the inconsistency of adjacent videos, make the acquired blurred image more temporally consistent, and further make the acquired blurred image more consistent with the real blurred image.
  • the embodiment of the present application also provides a network model training method.
  • the network model training method includes the following steps: S41 to S44:
  • the set of sample images includes a plurality of sample images with a resolution greater than a threshold resolution.
  • the sample image set may be an image set composed of image frames in a piece of high-definition video.
  • the implementation manner of acquiring the blurred image corresponding to each sample image in the sample image set is: acquiring the blurred image corresponding to each sample image in the sample image set through the blurred image generation method provided in any one of the above embodiments.
  • the sample image set is used as the second image set, and the method for generating a blurred image provided by the above embodiment is executed to obtain a blurred image corresponding to each sample image in the sample image set.
  • each sample image in the sample image set is a high-resolution image with a resolution greater than the second threshold
  • the blurred image corresponding to each sample image in the sample image set is generated by degrading each sample image in the sample image set Low-resolution images, therefore, according to each sample image in the sample image set and the blurred image corresponding to each sample image in the sample image set, multiple high-resolution images and multiple high-resolution images corresponding to each A training dataset of low-resolution images.
  • the training data set generated in the embodiment of the present application is the training data set of the image inpainting network model used for inpainting blurred images.
  • the blur kernel used to degrade the images in the sample image set in the embodiment of the present application is the blur kernel of the real image in the first image set
  • the blur kernel corresponding to each sample image in the sample image set The blurred image obtained by downgrading each sample image in the sample image set is more consistent with the real blurred image, so the embodiment of the present application can solve the problem that the blurred image obtained in the related art is very different from the real blurred image , thereby improving the performance of the video inpainting network model.
  • the embodiment of the present application also provides a blurred image generation device and a network model training device
  • the device embodiment corresponds to the aforementioned method embodiment, for the convenience of reading, the device
  • the embodiment does not repeat the details of the foregoing method embodiments one by one, but it should be clear that the device in this embodiment can correspondingly implement all the content in the foregoing method embodiments.
  • FIG. 5 is a schematic structural diagram of the blurred image generating device. As shown in FIG. 5, the blurred image generating device 500 includes:
  • An acquisition unit 51 configured to acquire the blur kernel of each image in the first image set; generate a blur kernel pool, the first image set includes a plurality of images with a resolution smaller than a first threshold;
  • the selection unit 52 is configured to select a blur kernel corresponding to each image in the second image set from the blur kernel pool; the second image set includes a plurality of images with a resolution greater than a second threshold;
  • the processing unit 53 is configured to degrade each image in the second image set through the blur kernel corresponding to each image in the second image set, and obtain a blurred image 5 corresponding to each image in the second image set .
  • the acquisition unit 51 is specifically configured to randomly generate the first noise and the second noise corresponding to the target image in the first image set; the first noise and the second noise both satisfy normal distribution; input the first noise into the depth image prior DIP model, and obtain the first image output by the DIP model; input the second noise into the flow-based kernel prior FKP model, and obtain the FKP model
  • the output predictive blur kernel degrade the first image through the predictive blur kernel to obtain a second image; judge whether the convergence condition is satisfied based on the second image and the target image; if not, update the model parameters and/or model input, and after updating the model parameters and/or the model input, judge whether the reacquired second image and the target image meet the convergence condition until the second image and the target image The convergence condition is met; if so, the predicted blur kernel output by the FKP model is determined as the blur kernel of the target image.
  • the acquiring unit 51 is specifically configured to update the model parameters of the DIP model and/or the model input of the FKP model.
  • the selecting unit 52 is specifically configured to, for each image in the second image set, randomly select a corresponding blur kernel from the blur kernel pool.
  • the selecting unit 52 is specifically configured to divide the images in the first image set into multiple first sub-image sets based on the scene of the images; divide the images belonging to the same first sub-image set
  • the fuzzy kernel of the image is divided into a fuzzy kernel group
  • the image in the second image set is divided into a plurality of second sub-image sets based on the scene of the image; for the images belonging to the same second sub-image set, from A corresponding blur kernel is randomly selected from the same blur kernel group.
  • the first set of images consists of image frames of a first video
  • the second set of images consists of image frames of a second video
  • the selecting unit 52 is specifically configured to divide the first video into a plurality of first video clips based on the scene of the image, and divide the image frames of each of the first video clips into a first sub-image set , and divide the second video into a plurality of second video clips based on the scene of the image, and divide the image frames of each of the second video clips into a second sub-image set.
  • the device for generating a blurred image provided in this embodiment can execute the method for generating a blurred image provided in the above method embodiment, and its implementation principle and technical effect are similar, and details will not be repeated here.
  • FIG. 6 is a schematic structural diagram of the blurred image generation device.
  • the network model training device 600 includes:
  • An acquisition unit 61 configured to acquire a set of sample images, the set of sample images including a plurality of sample images with a resolution greater than a threshold resolution;
  • a processing unit 62 configured to obtain a blurred image corresponding to each sample image in the sample image set through the blurred image generation method described in any one of the above embodiments;
  • a generating unit 63 configured to generate a training data set according to each sample image in the sample image set and the blurred image corresponding to each sample image;
  • the training unit 64 is configured to train a preset network model through the training data set, and obtain an image inpainting network model for inpainting blurred images.
  • the network model training device provided in this embodiment can execute the network model training method provided in the above method embodiment, and its implementation principle and technical effect are similar, and will not be repeated here.
  • FIG. 7 is a schematic structural diagram of an electronic device provided by an embodiment of the present application.
  • the electronic device provided by this embodiment includes: a memory 71 and a processor 72, the memory 71 is used to store computer programs; the processing The device 72 is configured to execute the methods provided in the above embodiments when calling a computer program.
  • an embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the computing device implements the above-mentioned embodiment provided method.
  • an embodiment of the present application further provides a computer program product, which enables the computing device to implement the method provided in the foregoing embodiments when the computer program product is run on a computer.
  • the embodiments of the present application may be provided as methods, systems, or computer program products. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media having computer-usable program code embodied therein.
  • the processor can be a central processing unit (Central Processing Unit, CPU), or other general-purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), off-the-shelf programmable Field-Programmable Gate Array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.
  • a general-purpose processor may be a microprocessor, or the processor may be any conventional processor, or the like.
  • Memory may include non-permanent storage in computer readable media, in the form of random access memory (RAM) and/or nonvolatile memory such as read only memory (ROM) or flash RAM.
  • RAM random access memory
  • ROM read only memory
  • flash RAM flash random access memory
  • Computer-readable media includes both volatile and non-volatile, removable and non-removable storage media.
  • the storage medium may store information by any method or technology, and the information may be computer-readable instructions, data structures, program modules, or other data.
  • Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Flash memory or other memory technology, Compact Disc Read-Only Memory (CD-ROM), Digital Versatile Disc (DVD) or other optical storage, A magnetic tape cartridge, disk storage or other magnetic storage device or any other non-transmission medium that can be used to store information that can be accessed by a computing device.
  • computer readable media excludes transitory computer readable media, such as modulated data signals and carrier waves.

Landscapes

  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Image Processing (AREA)

Abstract

The present application provides a blurred image generating method and apparatus, and a network model training method and apparatus. The method comprises: obtaining a blur kernel of each image in a first image set, and generating a blur kernel pool, the first image set comprising a plurality of images each having a resolution less than a first threshold; selecting a blur kernel corresponding to each image in a second image set from the blur kernel pool, the second image set comprising a plurality of images each having a resolution greater than a second threshold; and performing quality reduction on the images in the second image set by means of the blur kernels corresponding to the images to obtain blurred images corresponding to the images in the second image set.

Description

一种模糊图像生成方法、网络模型训练方法及装置A fuzzy image generation method, network model training method and device
本申请要求于2021年10月26日提交的，申请名称为“一种模糊图像生成方法、网络模型训练方法及装置”的、中国专利申请号为“202111250062.4”的优先权，该中国专利申请的全部内容通过引用结合在本申请中。This application claims the priority of the Chinese patent application number "202111250062.4" filed on October 26, 2021, with the title of "A Fuzzy Image Generation Method, Network Model Training Method and Device", and the Chinese patent application The entire contents are incorporated by reference in this application.
技术领域technical field
本申请涉及图像处理技术领域，尤其涉及一种模糊图像生成方法、网络模型训练方法及装置。The present application relates to the technical field of image processing, in particular to a blurred image generation method, a network model training method and a device.
背景技术Background technique
视频修复任务是视频画质增强中的重点业务。目前使用最为广泛的一种视频修复方式为：获取训练数据集，使用获取的训练数据集对网络模型进行训练获取视频修复网络模型，最终通过视频修复网络模型进行视频修复。由于训练数据集直接决定获取的视频修复网络模型的模型性能，因此如何获取与真实数据更加吻合的训练数据集已成为本领域的研究热点之一。Video repair task is the key business in video quality enhancement. Currently, the most widely used method of video repair is: obtain a training data set, use the obtained training data set to train the network model to obtain a video repair network model, and finally perform video repair through the video repair network model. Since the training data set directly determines the model performance of the obtained video inpainting network model, how to obtain a training data set that is more consistent with the real data has become one of the research hotspots in this field.
相关技术中普遍采用的方法为：获取包括多个清晰图像的高质量图像集合，然后采用Bicubic降采样等降质方式生成高质量图像集合中各个清晰图像对应的模糊图像，最后将高质量图像集合中的清晰图像以及对应的模糊图像作为训练数据集。然而，采用Bicubic降采样等降质方式获取的模糊图像与真实的模糊图像的差异很大。由于通过相关技术中的训练数据集生成方法获取的训练数据集中的模糊图像与真实的模糊图像的差异很大，因此训练得到的视频修复网络模型的性能非常不理想。The method commonly used in the related art is: acquire a high-quality image set including multiple clear images, then use degraded methods such as Bicubic downsampling to generate blurred images corresponding to each clear image in the high-quality image set, and finally combine the high-quality image set The clear images in and the corresponding blurred images are used as the training data set. However, the blurred image obtained by degrading methods such as Bicubic downsampling is very different from the real blurred image. Since the blurred images in the training data set acquired through the training data set generation method in the related art are very different from the real blurred images, the performance of the trained video inpainting network model is very unsatisfactory.
发明内容Contents of the invention
有鉴于此，本申请提供了一种模糊图像生成方法、网络模型训练方法及装置，用于解决相关技术中获取的训练数据集中模糊图像与真实的模糊图像的差异很大的问题。In view of this, the present application provides a blurred image generation method, network model training method and device, which are used to solve the problem that the blurred image in the training data set obtained in the related art is very different from the real blurred image.
为了实现上述目的，本申请实施例提供技术方案如下：In order to achieve the above purpose, the embodiment of the present application provides the following technical solutions:
第一方面，本申请的实施例提供了一种模糊图像生成方法，包括：In the first aspect, the embodiments of the present application provide a method for generating a blurred image, including:
获取第一图像集合中各个图像的模糊核，生成模糊核池；所述第一图像集合包括多个分辨率小于第一阈值的图像；Acquiring the blur kernels of each image in the first image collection to generate a blur kernel pool; the first image collection includes a plurality of images with a resolution smaller than the first threshold;
从所述模糊核池中选取第二图像集合中的各个图像对应的模糊核；所述第二图像集合包括多个分辨率大于第二阈值的图像；Selecting a blur kernel corresponding to each image in a second image set from the blur kernel pool; the second image set includes a plurality of images with a resolution greater than a second threshold;
通过所述第二图像集合中的各个图像对应的模糊核对所述第二图像集合中的各个图像进行降质，获取所述第二图像集合中各个图像对应的模糊图像。Each image in the second image set is degraded through a blur kernel corresponding to each image in the second image set, and a blurred image corresponding to each image in the second image set is acquired.
第二方面，本申请的实施例提供了一种网络模型训练方法，包括：In a second aspect, the embodiments of the present application provide a network model training method, including:
获取样本图像集合，所述样本图像集合包括多个分辨率大于阈值分辨率的样本图像；Acquiring a set of sample images, the set of sample images including a plurality of sample images with a resolution greater than a threshold resolution;
通过第一方面任一项所述的模糊图像生成方法，获取所述样本图像集合中各个样本图像 对应的模糊图像；By the fuzzy image generation method described in any one of the first aspect, the blurred image corresponding to each sample image in the sample image set is obtained;
根据所述样本图像集合中的各个样本图像以及各个样本图像对应的模糊图像，生成训练数据集；generating a training data set according to each sample image in the sample image set and the blurred image corresponding to each sample image;
通过所述训练数据集对用于对模糊图像进行修复的图像修复网络模型进行训练。An image inpainting network model for inpainting blurred images is trained by the training data set.
第三方面，本申请的实施例提供了一种模糊图像生成装置，包括：In a third aspect, an embodiment of the present application provides a device for generating a blurred image, including:
获取单元，用于获取第一图像集合中各个图像的模糊核；生成模糊核池，所述第一图像集合包括多个分辨率小于第一阈值的图像；An acquisition unit, configured to acquire the blur kernel of each image in the first image set; generate a blur kernel pool, the first image set includes a plurality of images with a resolution smaller than a first threshold;
选取单元，用于从所述模糊核池中选取第二图像集合中的各个图像对应的模糊核；所述第二图像集合包括多个分辨率大于第二阈值的图像；A selection unit, configured to select a blur kernel corresponding to each image in a second image set from the blur kernel pool; the second image set includes a plurality of images with a resolution greater than a second threshold;
处理单元，用于通过所述第二图像集合中的各个图像对应的模糊核对所述第二图像集合中的各个图像进行降质，获取所述第二图像集合中各个图像对应的模糊图像。A processing unit, configured to degrade each image in the second image set through a blur kernel corresponding to each image in the second image set, and acquire a blurred image corresponding to each image in the second image set.
第四方面，本申请实施例提供了一种网络模型训练装置，包括：In a fourth aspect, the embodiment of the present application provides a network model training device, including:
获取单元，用于获取样本图像集合，所述样本图像集合包括多个分辨率大于阈值分辨率的样本图像；an acquiring unit, configured to acquire a sample image set, the sample image set including a plurality of sample images with a resolution greater than a threshold resolution;
处理单元，用于通过第一方面任一项所述的模糊图像生成方法，获取所述样本图像集合中各个样本图像对应的模糊图像；A processing unit, configured to obtain a blurred image corresponding to each sample image in the sample image set by using the blurred image generation method described in any one of the first aspect;
生成单元，用于根据所述样本图像集合中的各个样本图像以及各个样本图像对应的模糊图像，生成训练数据集；A generation unit, configured to generate a training data set according to each sample image in the sample image set and the blurred image corresponding to each sample image;
训练单元，用于通过所述训练数据集对预设网络模型进行训练，获取用于对模糊图像进行修复的图像修复网络模型。The training unit is configured to train a preset network model through the training data set, and obtain an image repair network model for repairing blurred images.
第五方面，本申请实施例提供了一种电子设备，包括：存储器和处理器，所述存储器用于存储计算机程序；所述处理器用于在调用计算机程序时，使得所述电子设备实现第一方面或第一方面任一实施方式所述的方法。In a fifth aspect, an embodiment of the present application provides an electronic device, including: a memory and a processor, the memory is used to store a computer program; the processor is used to enable the electronic device to implement the first Aspect or the method described in any embodiment of the first aspect.
第六方面，本申请实施例提供一种计算机可读存储介质，当所述计算机程序被计算设备执行时，使得所述计算设备实现第一方面或第一方面任一实施方式所述的方法。In a sixth aspect, the embodiments of the present application provide a computer-readable storage medium. When the computer program is executed by a computing device, the computing device implements the method described in the first aspect or any implementation manner of the first aspect.
第七方面，本申请实施例提供一种计算机程序产品，当所述计算机程序产品在计算机上运行时，使得所述计算机实现第一方面或第一方面任一实施方式所述的方法。In a seventh aspect, an embodiment of the present application provides a computer program product, which enables the computer to implement the method described in the first aspect or any implementation manner of the first aspect when the computer program product is run on a computer.
本申请实施例提供的模糊图像生成方法首先获取多个分辨率小于第一阈值的图像的第一图像集合中各个图像的模糊核生成模糊核池，然后从所述模糊核池中选取包括多个分辨率大于第二阈值的图像的第二图像集合中的各个图像对应的模糊核，再通过所述第二图像集合中的各个图像对应的模糊核对所述第二图像集合中的各个图像进行降质，获取所述第二图像集合中各个图像对应的模糊图像。由于本申请实施例中用于对第二图像集合中的图像进行降 质的模糊核为第一图像集合中的真实图像的模糊核，因此通过所述第二图像集合中的各个图像对应的模糊核对所述第二图像集合中的各个图像进行降质获取的模糊图像与真实的模糊图像更加吻合，因此本申请实施例可以解决相关技术中获取的模糊图像与真实的模糊图像的差异很大的问题。The blurred image generation method provided in the embodiment of the present application first obtains the blur kernels of each image in the first image set of multiple images with a resolution smaller than the first threshold to generate a blur kernel pool, and then selects a blur kernel pool including multiple A blur kernel corresponding to each image in the second image set of images with a resolution greater than the second threshold, and then degrades each image in the second image set through the blur kernel corresponding to each image in the second image set quality, acquiring blurred images corresponding to each image in the second image set. Since the blur kernel used to degrade the images in the second image set in the embodiment of the present application is the blur kernel of the real image in the first image set, the blurring corresponding to each image in the second image set The blurred image obtained by checking each image in the second image set through downgrading is more consistent with the real blurred image. Therefore, the embodiment of the present application can solve the problem that the blurred image acquired in the related art is very different from the real blurred image. question.
附图说明Description of drawings
此处的附图被并入说明书中并构成本说明书的一部分，示出了符合本申请的实施例，并与说明书一起用于解释本申请的原理。The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the application and together with the description serve to explain the principles of the application.
为了更清楚地说明本申请实施例或相关技术中的技术方案，下面将对实施例或相关技术描述中所需要使用的附图作简单地介绍，显而易见地，对于本领域普通技术人员而言，在不付出创造性劳动性的前提下，还可以根据这些附图获得其他的附图。In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following will briefly introduce the drawings that need to be used in the descriptions of the embodiments or related technologies. Obviously, for those of ordinary skill in the art, Other drawings can also be obtained from these drawings without any creative effort.
图1为本申请实施例提供的模糊图像生成方法的步骤流程图之一；Fig. 1 is one of the flow charts of the steps of the blurred image generation method provided by the embodiment of the present application;
图2为本申请实施例提供的模糊图像生成方法的数据流示意图之二；FIG. 2 is the second schematic diagram of the data flow of the blurred image generation method provided by the embodiment of the present application;
图3为本申请实施例提供的模糊图像生成方法模型框架图；FIG. 3 is a model frame diagram of a blurred image generation method provided by an embodiment of the present application;
图4为本申请实施例提供的网络模型训练方法点的步骤流程图；FIG. 4 is a flow chart of the steps of the network model training method provided by the embodiment of the present application;
图5为本申请实施例提供的模糊图像生成装置的结构示意图；FIG. 5 is a schematic structural diagram of a blurred image generation device provided in an embodiment of the present application;
图6为本申请实施例提供的网络模型训练装置的结构示意图FIG. 6 is a schematic structural diagram of a network model training device provided in an embodiment of the present application
图7为本申请实施例提供的电子设备的硬件结构示意图。FIG. 7 is a schematic diagram of a hardware structure of an electronic device provided by an embodiment of the present application.
具体实施方式Detailed ways
为了能够更清楚地理解本申请的上述目的、特征和优点，下面将对本申请的方案进行进一步描述。需要说明的是，在不冲突的情况下，本申请的实施例及实施例中的特征可以相互组合。In order to better understand the above purpose, features and advantages of the present application, the solution of the present application will be further described below. It should be noted that, in the case of no conflict, the embodiments of the present application and the features in the embodiments can be combined with each other.
在下面的描述中阐述了很多具体细节以便于充分理解本申请，但本申请还可以采用其他不同于在此描述的方式来实施；显然，说明书中的实施例只是本申请的一部分实施例，而不是全部的实施例。In the following description, a lot of specific details have been set forth in order to fully understand the present application, but the present application can also be implemented in other ways different from those described here; obviously, the embodiments in the description are only a part of the present application, and Not all examples.
在本申请实施例中，“示例性的”或者“例如”等词用于表示作例子、例证或说明。本申请实施例中被描述为“示例性的”或者“例如”的任何实施例或设计方案不应被解释为比其它实施例或设计方案更优选或更具优势。确切而言，使用“示例性的”或者“例如”等词旨在以具体方式呈现相关概念。此外，在本申请实施例的描述中，除非另有说明，“多个”的含义是指两个或两个以上。In the embodiments of the present application, words such as "exemplary" or "for example" are used as examples, illustrations or illustrations. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of the present application shall not be interpreted as being more preferred or more advantageous than other embodiments or design schemes. Rather, the use of words such as "exemplary" or "such as" is intended to present related concepts in a concrete manner. In addition, in the description of the embodiments of the present application, unless otherwise specified, the meaning of "plurality" refers to two or more.
本申请实施例提供了一种模糊图像生成方法，参照图1所示，该模糊图像生成方法包括如下步骤：The embodiment of the present application provides a method for generating a blurred image. Referring to FIG. 1, the method for generating a blurred image includes the following steps:
S11、获取第一图像集合中各个图像的模糊核，生成模糊核池。S11. Obtain blur kernels of each image in the first image set, and generate a blur kernel pool.
其中，所述第一图像集合包括多个分辨率小于第一阈值的图像。Wherein, the first set of images includes a plurality of images with resolutions smaller than a first threshold.
具体的，上述步骤S11的实现过程可以包括：获取多个分辨率小于第一阈值的低清晰度图像组成第一图像集合，然后提取第一图像集合中的每一个低清晰度图像的模糊核，最后将提取的全部模糊核组成模糊核池。Specifically, the implementation process of the above step S11 may include: obtaining a plurality of low-resolution images with a resolution smaller than the first threshold to form a first image set, and then extracting the blur kernel of each low-resolution image in the first image set, Finally, all the extracted fuzzy kernels are combined into a fuzzy kernel pool.
本申请实施例中的第一阈值可以根据最终训练得到的视频修复网络模型的使用场景确定，若最终训练得到的视频修复网络模型用于对较低清晰度的视频进行修复，则可以将第一阈值设置的较小，而若最终训练得到的视频修复网络模型用于对较高清晰度的视频进行修复，则可以将第一阈值设置的较大。The first threshold in the embodiment of the present application can be determined according to the usage scenario of the video repair network model obtained from the final training. If the video repair network model obtained from the final training is used to repair a video with lower definition, the first The threshold is set smaller, and if the video inpainting network model finally trained is used to repair a video with higher definition, the first threshold can be set larger.
需要说明的是，本申请实施例中的第一图像集合可以由多张相互独立的图像组成，也可以为从同一视频中进行图像帧采样获取的多个视频帧，还可以为同一视频中的全部图像帧，本申请实施例对此不做限定，以第一图像集合中的图像的分辨率均小于第一阈值为准。It should be noted that the first image set in the embodiment of the present application may consist of multiple mutually independent images, or may be multiple video frames obtained by sampling image frames from the same video, or may be a set of images in the same video All the image frames are not limited in this embodiment of the present application, as long as the resolutions of the images in the first image set are all smaller than the first threshold.
S12、从所述模糊核池中选取第二图像集合中的各个图像对应的模糊核。S12. Select a blur kernel corresponding to each image in the second image set from the blur kernel pool.
其中，所述第二图像集合包括多个分辨率大于第二阈值的图像。Wherein, the second image set includes a plurality of images with a resolution greater than a second threshold.
即，针对第二图像集合中的每一张图像，分别从模糊核池中选取一个模糊核作为对应的模糊核。That is, for each image in the second image set, one blur kernel is selected from the blur kernel pool as the corresponding blur kernel.
S13、通过所述第二图像集合中的各个图像对应的模糊核对所述第二图像集合中的各个图像进行降质，获取所述第二图像集合中各个图像对应的模糊图像。S13. Degrade each image in the second image set by using a blur kernel corresponding to each image in the second image set, and acquire a blurred image corresponding to each image in the second image set.
在一些实施方式中，上述步骤S13(通过所述第二图像集合中的各个图像对应的模糊核对所述第二图像集合中的各个图像进行降质，获取所述第二图像集合中各个图像对应的模糊图像)包括：In some implementations, the above step S13 (degrade each image in the second image set through the blur kernel corresponding to each image in the second image set, and obtain the corresponding blur kernel of each image in the second image set blurred image) including:
通过所述第二图像集合中的各个图像对应的模糊核和第一公式对所述第二图像集合中的各个图像进行降质，获取所述第二图像集合中各个图像对应的模糊图像；Degrade each image in the second image set by using the blur kernel corresponding to each image in the second image set and the first formula, and obtain a blurred image corresponding to each image in the second image set;
所述第一公式为：The first formula is:
I x＝Deg(J x，K x)+N I x =Deg(J x , K x )+N
其中，J x表示第二图像集合中的图像，I x表示第二图像集合中的图像J x对应的模糊图像，K x表示第二图像集合中的图像J x对应的模糊核，Deg(J x，K x)表示通过K x对J x进行降质运算，N表示额外向I x中额外添加的噪声。 Wherein, J x represents the image in the second image collection, I x represents the blurred image corresponding to the image J x in the second image collection, K x represents the blur kernel corresponding to the image J x in the second image collection, Deg(J x , K x ) means that J x is degraded by K x , and N means additional noise added to I x .
本申请实施例提供的模糊图像生成方法首先获取多个分辨率小于第一阈值的图像的第一图像集合中各个图像的模糊核生成模糊核池，然后从所述模糊核池中选取包括多个分辨率大于第二阈值的图像的第二图像集合中的各个图像对应的模糊核，再通过所述第二图像集合中的各个图像对应的模糊核对所述第二图像集合中的各个图像进行降质，获取所述第二图像集合中各个图像对应的模糊图像。由于本申请实施例中用于对第二图像集合中的图像进行降 质的模糊核为第一图像集合中的真实图像的模糊核，因此通过所述第二图像集合中的各个图像对应的模糊核对所述第二图像集合中的各个图像进行降质获取的模糊图像与真实的模糊图像更加吻合，因此本申请实施例可以解决相关技术中获取的模糊图像与真实的模糊图像的差异很大的问题。The blurred image generation method provided in the embodiment of the present application first obtains the blur kernels of each image in the first image set of multiple images with a resolution smaller than the first threshold to generate a blur kernel pool, and then selects a blur kernel pool including multiple A blur kernel corresponding to each image in the second image set of images with a resolution greater than the second threshold, and then degrades each image in the second image set through the blur kernel corresponding to each image in the second image set quality, acquiring blurred images corresponding to each image in the second image set. Since the blur kernel used to degrade the images in the second image set in the embodiment of the present application is the blur kernel of the real image in the first image set, the blurring corresponding to each image in the second image set The blurred image obtained by checking each image in the second image set through downgrading is more consistent with the real blurred image. Therefore, the embodiment of the present application can solve the problem that the blurred image acquired in the related art is very different from the real blurred image. question.
在一些实施方式中，参照图2所示，上述实施例中的步骤S11(获取第一图像集合中各个图像的模糊核)的实现方式可以包括如下步骤：In some implementations, as shown in FIG. 2, the implementation of step S11 (obtaining the blur kernel of each image in the first image set) in the above-mentioned embodiment may include the following steps:
S21、随机生成所述第一图像集合中的目标图像对应的第一噪声和第二噪声。S21. Randomly generate first noise and second noise corresponding to the target image in the first image set.
其中，所述第一噪声和第二噪声均满足正态分布。Wherein, both the first noise and the second noise satisfy normal distribution.
S22、将所述第一噪声输入深度图像先验(Deep Image Prior，DIP)模型，并获取所述DIP模型输出的第一图像。S22. Input the first noise into a deep image prior (Deep Image Prior, DIP) model, and acquire a first image output by the DIP model.
具体的，由于网络结构本身能够抓取大量低层级的图像统计先验信息，从而实现特征提取，因此仅在单张损坏图像上进行反复迭代，也同样能学习到图像的先验信息，进而完成图像修复。DIP模型是一个利用利用网络结构本身能够抓取大量低层级的图像统计先验信息，而构建的网络模型；将随机噪声作为DIP模型的输入，随着DIP模型迭代次数增加，DIP模型可输出对应的高质量图像，因此上述步骤S22将第一噪声输入DIP模型后可以DIP模型输出的第一图像。Specifically, since the network structure itself can capture a large amount of low-level image statistical prior information to realize feature extraction, only repeated iterations on a single damaged image can also learn the prior information of the image, and then complete Image restoration. The DIP model is a network model constructed by using the network structure itself to capture a large amount of low-level image statistical prior information; random noise is used as the input of the DIP model, and as the number of iterations of the DIP model increases, the DIP model can output the corresponding Therefore, the first image can be output by the DIP model after the first noise is input into the DIP model in the above step S22.
S23、将所述第二噪声输入基于流的核先验(Flow-based Kernel Prior，FKP模型，并获取所述FKP模型输出的预测模糊核。S23. Input the second noise into a Flow-based Kernel Prior (Flow-based Kernel Prior, FKP model, and obtain a prediction blur kernel output by the FKP model.
具体的，可以构建一个可逆网络，并通过预训练得到随机噪声到对应的模糊核的过程，训练得到的FKP模型只要输入一个正太分布的噪声，即可以到一个真实的模糊核，因此上述步骤S23将第二噪声输入FKP模型后，可以获取FKP模型预测的真实模糊核。Specifically, a reversible network can be constructed, and the process from random noise to the corresponding fuzzy kernel can be obtained through pre-training. As long as the trained FKP model is input with a normal distribution of noise, it can get to a real fuzzy kernel. Therefore, the above step S23 After inputting the second noise into the FKP model, the real blur kernel predicted by the FKP model can be obtained.
S24、通过所述预测模糊核对所述第一图像进行降质，获取第二图像。S24. Degrade the first image by using the predictive blur kernel to acquire a second image.
在一些实施方式中，上述步骤S24(通过所述预测模糊核对所述第一图像进行降质，获取第二图像)包括：In some implementation manners, the above step S24 (degrading the first image through the predictive blur kernel to obtain a second image) includes:
通过预测模糊核和第一公式对第一图像进行降质，获取第二图像；Degrading the first image by predicting the blur kernel and the first formula to obtain the second image;
所述第一公式为：The first formula is:
I y＝Deg(J y，k)+N I y =Deg(J y ,k)+N
其中，J y表示第一图像，I y表示第二图像，k表示预测模糊核，DegDeg(J y，k)表示通过k对第一图像进行降质的运算，N表示额外向第二图像中额外添加的噪声。 Among them, J y represents the first image, I y represents the second image, k represents the prediction blur kernel, DegDeg(J y , k) represents the operation of degrading the first image through k, and N represents the additional input to the second image Additional noise added.
需要说明的是，由于图像修复任务一般无需提升模糊图像的分辨率，因此在通过所述预测模糊核对所述第一图像进行降质获取第二图像的过程中，无需对第一图像进行降采样。It should be noted that since the image restoration task generally does not need to increase the resolution of the blurred image, it is not necessary to down-sample the first image in the process of degrading the first image through the predictive blur kernel to obtain the second image .
S25、基于所述第二图像和所述目标图像判断是否满足收敛条件。S25. Determine whether a convergence condition is met based on the second image and the target image.
具体的，可以对第二图像和目标图像进行L1loss约束，从而判断是否满足收敛条件。Specifically, an L1loss constraint may be performed on the second image and the target image, so as to determine whether the convergence condition is satisfied.
在上述步骤S25中，若所述第二图像和所述目标图像不满足收敛条件，则执行下述步骤S26，若所述第二图像和所述目标图像满足收敛条件，则执行下述步骤S27。In the above step S25, if the second image and the target image do not meet the convergence condition, then perform the following step S26, and if the second image and the target image meet the convergence condition, then perform the following step S27 .
S26、更新模型参数和/或模型输入，在更新所述模型参数和/或所述模型输入后，判断重新获取的第二图像和所述目标图像是否满足收敛条件，直至所述第二图像和所述目标图像满足所述收敛条件。S26. Update model parameters and/or model inputs. After updating the model parameters and/or the model inputs, judge whether the reacquired second image and the target image meet the convergence condition until the second image and the The target image satisfies the convergence condition.
即，在更新模型参数和/或模型输入后，重新将所述第一噪声输入DIP模型，并获取所述DIP模型输出的第一图像，将所述第二噪声输入FKP模型，并获取所述FKP模型输出的预测模糊核，通过所述预测模糊核对所述第一图像进行降质，获取第二图像，以及基于所述第二图像和所述目标图像判断是否满足收敛条件。即，如图2所示，在更新模型参数和/或模型输入后，返回步骤S22。That is, after updating the model parameters and/or model input, re-input the first noise into the DIP model, and obtain the first image output by the DIP model, input the second noise into the FKP model, and obtain the The predictive blur kernel output by the FKP model degrades the first image through the predictive blur kernel, acquires a second image, and judges whether a convergence condition is satisfied based on the second image and the target image. That is, as shown in FIG. 2, after updating the model parameters and/or model input, return to step S22.
在一些实施方式中，所述更新模型参数和/或模型输入，包括：In some embodiments, said updating model parameters and/or model inputs includes:
更新所述DIP模型的模型参数和/或所述FKP模型的模型输入。Updating model parameters of the DIP model and/or model inputs of the FKP model.
即，训练过程中只会对DIP模型的模型参数和/或第二噪声进行更新，而不会对FKP模型的模型参数或DIP模型的模型输入(第一噪声)进行更新。That is, only the model parameters of the DIP model and/or the second noise are updated during the training process, but the model parameters of the FKP model or the model input (first noise) of the DIP model are not updated.
S27、将所述FKP模型输出的预测模糊核确定为所述目标图像的模糊核。S27. Determine the predicted blur kernel output by the FKP model as the blur kernel of the target image.
逐一将第一图像集合中各个图像作为上述实施例中的目标图像即可获取第一图像集合中各个图像的模糊核，进而生成上述实施例中的模糊核池。By using each image in the first image set as the target image in the above embodiment one by one, the blur kernel of each image in the first image set can be obtained, and then the blur kernel pool in the above embodiment is generated.
进一步的，参照图3所示，图2所示的获取第一图像集合的目标图像的实现过程包括：Further, referring to FIG. 3 , the implementation process of acquiring the target image of the first image set shown in FIG. 2 includes:
将第一噪声z x输入DIP模型，获取DIP模型输出的第一图像g(z x,θ g)。 Input the first noise z x into the DIP model, and obtain the first image g(z x ,θ g ) output by the DIP model.
将第二噪声z k输入FKP模型，获取FKP模型输出的预测模糊核k(z k,θ k)。 Input the second noise z k into the FKP model, and obtain the predicted blur kernel k(z k , θ k ) output by the FKP model.
通过预测模糊核k(z k,θ k)对第一图像g(z x,θ g)进行降质，获取第二图像P。 The first image g(z x , θ g ) is degraded by predicting the blur kernel k(z k , θ k ), to obtain the second image P.
对第二图像P和目标图像LR进行损失约束，并判断是否满足收敛条件。Perform loss constraints on the second image P and the target image LR, and judge whether the convergence condition is satisfied.
在一些实施方式中，上述步骤S12(从所述模糊核池中选取第二图像集合中的各个图像对应的模糊核)的一种实现方式包括：In some embodiments, an implementation of the above step S12 (selecting the blur kernel corresponding to each image in the second image set from the blur kernel pool) includes:
针对所述第二图像集合中的每一个图像，随机从所述模糊核池中选取对应的模糊核。For each image in the second image set, randomly select a corresponding blur kernel from the blur kernel pool.
即，随机从所述模糊核池中为第二图像集合中的每一个图像选取对应的模糊核。That is, randomly select a corresponding blur kernel for each image in the second image set from the blur kernel pool.
在一些实施方式中，上述步骤S12(从所述模糊核池中选取第二图像集合中的各个图像对应的模糊核)的一种实现方式包括如下步骤a至步骤d：In some embodiments, an implementation of the above step S12 (selecting the blur kernel corresponding to each image in the second image set from the blur kernel pool) includes the following steps a to d:
步骤a、基于图像的场景将所述第一图像集合中的图像划分为多个第一子图像集合。Step a: Divide the images in the first image set into multiple first sub-image sets based on the scene of the image.
即，将第一图像集合中图像场景相同的图像划分为一个第一子图像集合，从而得到多个第一子图像集合。That is, images with the same image scene in the first image set are divided into a first sub-image set, so as to obtain multiple first sub-image sets.
在一些实施例中所述第一图像集合由第一视频的图像帧组成。所述基于图像的场景将所述第一图像集合中的图像划分为多个第一子图像集合，包括：In some embodiments the first set of images consists of image frames of a first video. The image-based scene divides the images in the first image set into a plurality of first sub-image sets, comprising:
基于图像的场景将所述第一视频划分为多个第一视频片段，并分别将各个所述第一视频片段的图像帧划分一个所述第一子图像集合。Divide the first video into a plurality of first video clips based on the image scene, and divide the image frames of each of the first video clips into a first sub-image set.
即，当第一图像集合中的图像为某一视频中的视频帧时，可以对该视频进行场景转换检测，从而将该视频划分为多个第一视频片段，然后在提取每一个第一视频片段的图像帧划作为一个第一子图像集合，从而将第一图像集合中图像按照图像的场景划分为多个第一子图像集合。That is, when the images in the first image set are video frames in a certain video, scene transition detection can be performed on the video, so that the video can be divided into a plurality of first video segments, and then each first video segment is extracted The image frames of the segment are divided into a first sub-image set, so that the images in the first image set are divided into multiple first sub-image sets according to the scenes of the images.
步骤b、将属于同一所述第一子图像集合的图像的模糊核，划分为一个模糊核组。Step b. Divide blur kernels of images belonging to the same first sub-image set into a blur kernel group.
即，对于第一图像集合中的任意两张图像，若该两张图像属于同一第一子图像集合，则该两张图像的模糊核属于同一模糊核组，而若该两张图像属于不同的第一子图像集合，则该两张图像的模糊核属于不同的模糊核组。That is, for any two images in the first image set, if the two images belong to the same first sub-image set, the blur kernels of the two images belong to the same blur kernel group, and if the two images belong to different the first sub-image set, the blur kernels of the two images belong to different blur kernel groups.
步骤c、基于图像的场景将所述第二图像集合中的图像划分为多个第二子图像集合。Step c, dividing the images in the second image set into multiple second sub-image sets based on the scene of the image.
即，将第二图像集合中图像场景相同的图像划分为一个第二子图像集合，从而得到多个第二子图像集合。That is, images with the same image scene in the second image set are divided into a second sub-image set, so as to obtain multiple second sub-image sets.
在一些实施例中，所述第二图像集合由第二视频的图像帧组成。所述基于图像的场景将所述第二图像集合中的图像划分为多个第二子图像集合，包括：In some embodiments, the second set of images consists of image frames of a second video. The image-based scene divides the images in the second image set into a plurality of second sub-image sets, comprising:
基于图像的场景将所述第二视频划分为多个第二视频片段，并分别将各个所述第二视频片段的图像帧划分一个所述第二子图像集合。Divide the second video into a plurality of second video clips based on the image scene, and divide the image frames of each of the second video clips into a second sub-image set.
即，当第二图像集合中的图像为某一视频中的视频帧时，可以对该视频进行场景转换检测，从而将该视频划分为多个第二视频片段，然后在提取每一个第二视频片段的图像帧划作为一个第二子图像集合，从而将第二图像集合中图像按照图像的场景划分为多个第二子图像集合。That is, when the images in the second image collection are video frames in a certain video, scene transition detection can be performed on the video, thereby dividing the video into a plurality of second video segments, and then extracting each second video The image frames of the segment are divided into a second sub-image set, so that the images in the second image set are divided into multiple second sub-image sets according to the scenes of the images.
步骤d、对属于同一所述第二子图像集合的图像，从同一所述模糊核组中随机选取对应的模糊核。Step d. For images belonging to the same second sub-image set, randomly select a corresponding blur kernel from the same blur kernel group.
在上述实施例中，由于图像场景相同的图像帧只会从同一模糊核组中随机选取模糊核进行降质操作，因此相比于对第二图像集合中的图像均随机选取对应的模糊核，上述实施例可以进一步减少或避免相邻视频的不一致，使获取的模糊图像更加具有时间□致性，进而使获取的模糊图像与真实的模糊图像更加吻合。In the above embodiment, since image frames with the same image scene will only randomly select blur kernels from the same blur kernel group for degrading operation, compared to randomly selecting corresponding blur kernels for images in the second image set, The above embodiments can further reduce or avoid the inconsistency of adjacent videos, make the acquired blurred image more temporally consistent, and further make the acquired blurred image more consistent with the real blurred image.
基于同一发明构思，本申请实施例还提供了一种网络模型训练方法，具体的，参照图4所示，该网络模型训练方法包括如下步骤：S41至S44:Based on the same inventive concept, the embodiment of the present application also provides a network model training method. Specifically, as shown in FIG. 4, the network model training method includes the following steps: S41 to S44:
S41、获取样本图像集合。S41. Acquire a sample image set.
其中，所述样本图像集合包括多个分辨率大于阈值分辨率的样本图像。Wherein, the set of sample images includes a plurality of sample images with a resolution greater than a threshold resolution.
示例性的，所述样本图像集合可以为一段高清视频中的图像帧组成的图像集合。Exemplarily, the sample image set may be an image set composed of image frames in a piece of high-definition video.
S42、获取所述样本图像集合中各个样本图像对应的模糊图像。S42. Obtain a blurred image corresponding to each sample image in the sample image set.
其中，获取所述样本图像集合中各个样本图像对应的模糊图像的实现方式为：通过上述任一实施例提供的模糊图像生成方法，获取所述样本图像集合中各个样本图像对应的模糊图像。Wherein, the implementation manner of acquiring the blurred image corresponding to each sample image in the sample image set is: acquiring the blurred image corresponding to each sample image in the sample image set through the blurred image generation method provided in any one of the above embodiments.
即，将所述样本图像集合作为的第二图像集合，执行上述实施例提供的模糊图像生成方法，以获取所述样本图像集合中各个样本图像对应的模糊图像。That is, the sample image set is used as the second image set, and the method for generating a blurred image provided by the above embodiment is executed to obtain a blurred image corresponding to each sample image in the sample image set.
S43、根据所述样本图像集合中的各个样本图像以及各个样本图像对应的模糊图像，生成训练数据集。S43. Generate a training data set according to each sample image in the sample image set and the blurred image corresponding to each sample image.
具体的，样本图像集合中的各个样本图像均为分辨率大于第二阈值的高分辨率图像，样本图像集合中各个样本图像对应的模糊图像为对样本图像集合中各个样本图像进行降质生成的低分辨率图像，因此根据所述样本图像集合中的各个样本图像以及所述样本图像集合中各个样本图像对应的模糊图像可以生成包括多个高分辨率图像以及多个高分辨率图像分别对应的低分辨率图像的训练数据集。Specifically, each sample image in the sample image set is a high-resolution image with a resolution greater than the second threshold, and the blurred image corresponding to each sample image in the sample image set is generated by degrading each sample image in the sample image set Low-resolution images, therefore, according to each sample image in the sample image set and the blurred image corresponding to each sample image in the sample image set, multiple high-resolution images and multiple high-resolution images corresponding to each A training dataset of low-resolution images.
S44、通过所述训练数据集对用于对模糊图像进行修复的图像修复网络模型进行训练。S44. Using the training data set, train an image inpainting network model for inpainting blurred images.
即，本申请实施例中生成的训练数据集为用于对模糊图像进行修复的图像修复网络模型的训练数据集。That is, the training data set generated in the embodiment of the present application is the training data set of the image inpainting network model used for inpainting blurred images.
由于本申请实施例中用于对样本图像集合中的图像进行降质的模糊核为第一图像集合中的真实图像的模糊核，因此通过所述样本图像集合中的各个样本图像对应的模糊核对所述样本图像集合中的各个样本图像进行降质获取的模糊图像与真实的模糊图像更加吻合，因此本申请实施例可以解决相关技术中获取的模糊图像与真实的模糊图像的差异很大的问题，进而提升视频修复网络模型的性能。Since the blur kernel used to degrade the images in the sample image set in the embodiment of the present application is the blur kernel of the real image in the first image set, the blur kernel corresponding to each sample image in the sample image set The blurred image obtained by downgrading each sample image in the sample image set is more consistent with the real blurred image, so the embodiment of the present application can solve the problem that the blurred image obtained in the related art is very different from the real blurred image , thereby improving the performance of the video inpainting network model.
基于同一发明构思，作为对上述方法的实现，本申请实施例还提供了一种模糊图像生成装置和一种网络模型训练装置，该装置实施例与前述方法实施例对应，为便于阅读，本装置实施例不再对前述方法实施例中的细节内容进行逐一赘述，但应当明确，本实施例中的装置能够对应实现前述方法实施例中的全部内容。Based on the same inventive concept, as the implementation of the above method, the embodiment of the present application also provides a blurred image generation device and a network model training device, the device embodiment corresponds to the aforementioned method embodiment, for the convenience of reading, the device The embodiment does not repeat the details of the foregoing method embodiments one by one, but it should be clear that the device in this embodiment can correspondingly implement all the content in the foregoing method embodiments.
本申请实施例提供了一种模糊图像生成装置，图5为该模糊图像生成装置的结构示意图，如图5所示，该模糊图像生成装置500包括：The embodiment of the present application provides a blurred image generating device. FIG. 5 is a schematic structural diagram of the blurred image generating device. As shown in FIG. 5, the blurred image generating device 500 includes:
获取单元51，用于获取第一图像集合中各个图像的模糊核；生成模糊核池，所述第一图像集合包括多个分辨率小于第一阈值的图像；An acquisition unit 51, configured to acquire the blur kernel of each image in the first image set; generate a blur kernel pool, the first image set includes a plurality of images with a resolution smaller than a first threshold;
选取单元52，用于从所述模糊核池中选取第二图像集合中的各个图像对应的模糊核； 所述第二图像集合包括多个分辨率大于第二阈值的图像；The selection unit 52 is configured to select a blur kernel corresponding to each image in the second image set from the blur kernel pool; the second image set includes a plurality of images with a resolution greater than a second threshold;
处理单元53，用于通过所述第二图像集合中的各个图像对应的模糊核对所述第二图像集合中的各个图像进行降质，获取所述第二图像集合中各个图像对应的模糊图像5。The processing unit 53 is configured to degrade each image in the second image set through the blur kernel corresponding to each image in the second image set, and obtain a blurred image 5 corresponding to each image in the second image set .
在一些实施方式中，所述获取单元51，具体用于随机生成所述第一图像集合中的目标图像对应的第一噪声和第二噪声；所述第一噪声和第二噪声均满足正态分布；将所述第一噪声输入深度图像先验DIP模型，并获取所述DIP模型输出的第一图像；将所述第二噪声输入基于流的核先验FKP模型，并获取所述FKP模型输出的预测模糊核；通过所述预测模糊核对所述第一图像进行降质，获取第二图像；基于所述第二图像和所述目标图像判断是否满足收敛条件；若否，则更新模型参数和/或模型输入，并在更新所述模型参数和/或所述模型输入后，判断重新获取的第二图像和所述目标图像是否满足收敛条件，直至所述第二图像和所述目标图像满足所述收敛条件；若是，则将所述FKP模型输出的预测模糊核确定为所述目标图像的模糊核。In some implementations, the acquisition unit 51 is specifically configured to randomly generate the first noise and the second noise corresponding to the target image in the first image set; the first noise and the second noise both satisfy normal distribution; input the first noise into the depth image prior DIP model, and obtain the first image output by the DIP model; input the second noise into the flow-based kernel prior FKP model, and obtain the FKP model The output predictive blur kernel; degrade the first image through the predictive blur kernel to obtain a second image; judge whether the convergence condition is satisfied based on the second image and the target image; if not, update the model parameters and/or model input, and after updating the model parameters and/or the model input, judge whether the reacquired second image and the target image meet the convergence condition until the second image and the target image The convergence condition is met; if so, the predicted blur kernel output by the FKP model is determined as the blur kernel of the target image.
在一些实施方式中，所述获取单元51，具体用于更新所述DIP模型的模型参数和/或所述FKP模型的模型输入。In some implementations, the acquiring unit 51 is specifically configured to update the model parameters of the DIP model and/or the model input of the FKP model.
在一些实施方式中，所述选取单元52，具体用于针对所述第二图像集合中的每一个图像，随机从所述模糊核池中选取对应的模糊核。In some implementation manners, the selecting unit 52 is specifically configured to, for each image in the second image set, randomly select a corresponding blur kernel from the blur kernel pool.
在一些实施方式中，所述选取单元52，具体用于基于图像的场景将所述第一图像集合中的图像划分为多个第一子图像集合；将属于同一所述第一子图像集合的图像的模糊核，划分为一个模糊核组；基于图像的场景将所述第二图像集合中的图像划分为多个第二子图像集合；对属于同一所述第二子图像集合的图像，从同一所述模糊核组中随机选取对应的模糊核。In some implementations, the selecting unit 52 is specifically configured to divide the images in the first image set into multiple first sub-image sets based on the scene of the images; divide the images belonging to the same first sub-image set The fuzzy kernel of the image is divided into a fuzzy kernel group; the image in the second image set is divided into a plurality of second sub-image sets based on the scene of the image; for the images belonging to the same second sub-image set, from A corresponding blur kernel is randomly selected from the same blur kernel group.
在一些实施方式中，所述第一图像集合由第一视频的图像帧组成，所述第二图像集合由第二视频的图像帧组成；In some embodiments, the first set of images consists of image frames of a first video, and the second set of images consists of image frames of a second video;
所述选取单元52，具体用于基于图像的场景将所述第一视频划分为多个第一视频片段，并分别将各个所述第一视频片段的图像帧划分一个所述第一子图像集合，以及基于图像的场景将所述第二视频划分为多个第二视频片段，并分别将各个所述第二视频片段的图像帧划分一个所述第二子图像集合。The selecting unit 52 is specifically configured to divide the first video into a plurality of first video clips based on the scene of the image, and divide the image frames of each of the first video clips into a first sub-image set , and divide the second video into a plurality of second video clips based on the scene of the image, and divide the image frames of each of the second video clips into a second sub-image set.
本实施例提供的模糊图像生成装置可以执行上述方法实施例提供的模糊图像生成方法，其实现原理与技术效果类似，此处不再赘述。The device for generating a blurred image provided in this embodiment can execute the method for generating a blurred image provided in the above method embodiment, and its implementation principle and technical effect are similar, and details will not be repeated here.
本申请实施例提供了一种网络模型训练装置，图6为该模糊图像生成装置的结构示意图，如图6所示，该网络模型训练装置600包括：The embodiment of the present application provides a network model training device. FIG. 6 is a schematic structural diagram of the blurred image generation device. As shown in FIG. 6, the network model training device 600 includes:
获取单元61，用于获取样本图像集合，所述样本图像集合包括多个分辨率大于阈值分辨率的样本图像；An acquisition unit 61, configured to acquire a set of sample images, the set of sample images including a plurality of sample images with a resolution greater than a threshold resolution;
处理单元62，用于通过上述任一实施例所述的模糊图像生成方法，获取所述样本图像集合中各个样本图像对应的模糊图像；A processing unit 62, configured to obtain a blurred image corresponding to each sample image in the sample image set through the blurred image generation method described in any one of the above embodiments;
生成单元63，用于根据所述样本图像集合中的各个样本图像以及各个样本图像对应的模糊图像，生成训练数据集；A generating unit 63, configured to generate a training data set according to each sample image in the sample image set and the blurred image corresponding to each sample image;
训练单元64，用于通过所述训练数据集对预设网络模型进行训练，获取用于对模糊图像进行修复的图像修复网络模型。The training unit 64 is configured to train a preset network model through the training data set, and obtain an image inpainting network model for inpainting blurred images.
本实施例提供的网络模型训练装置可以执行上述方法实施例提供的网络模型训练方法，其实现原理与技术效果类似，此处不再赘述。The network model training device provided in this embodiment can execute the network model training method provided in the above method embodiment, and its implementation principle and technical effect are similar, and will not be repeated here.
基于同一发明构思，本申请实施例还提供了一种电子设备。图7为本申请实施例提供的电子设备的结构示意图，如图7所示，本实施例提供的电子设备包括：存储器71和处理器72，所述存储器71用于存储计算机程序；所述处理器72用于在调用计算机程序时执行上述实施例提供的方法。Based on the same inventive concept, the embodiment of the present application also provides an electronic device. FIG. 7 is a schematic structural diagram of an electronic device provided by an embodiment of the present application. As shown in FIG. 7 , the electronic device provided by this embodiment includes: a memory 71 and a processor 72, the memory 71 is used to store computer programs; the processing The device 72 is configured to execute the methods provided in the above embodiments when calling a computer program.
基于同一发明构思，本申请实施例还提供了一种计算机可读存储介质，该计算机可读存储介质上存储有计算机程序，当计算机程序被处理器执行时，使得所述计算设备实现上述实施例提供的方法。Based on the same inventive concept, an embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the computing device implements the above-mentioned embodiment provided method.
基于同一发明构思，本申请实施例还提供了一种计算机程序产品，当所述计算机程序产品在计算机上运行时，使得所述计算设备实现上述实施例提供的方法。Based on the same inventive concept, an embodiment of the present application further provides a computer program product, which enables the computing device to implement the method provided in the foregoing embodiments when the computer program product is run on a computer.
本领域技术人员应明白，本申请的实施例可提供为方法、***、或计算机程序产品。因此，本申请可采用完全硬件实施例、完全软件实施例、或结合软件和硬件方面的实施例的形式。而且，本申请可采用在一个或多个其中包含有计算机可用程序代码的计算机可用存储介质上实施的计算机程序产品的形式。Those skilled in the art should understand that the embodiments of the present application may be provided as methods, systems, or computer program products. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media having computer-usable program code embodied therein.
处理器可以是中央处理单元(Central Processing Unit，CPU)，还可以是其他通用处理器、数字信号处理器(Digital Signal Processor，DSP)、专用集成电路(Application Specific Integrated Circuit，ASIC)、现成可编程门阵列(Field-Programmable Gate Array，FPGA)或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件等。通用处理器可以是微处理器或者该处理器也可以是任何常规的处理器等。The processor can be a central processing unit (Central Processing Unit, CPU), or other general-purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), off-the-shelf programmable Field-Programmable Gate Array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor, or the processor may be any conventional processor, or the like.
存储器可能包括计算机可读介质中的非永久性存储器，随机存取存储器(RAM)和/或非易失性内存等形式，如只读存储器(ROM)或闪存(flash RAM)。存储器是计算机可读介质的示例。Memory may include non-permanent storage in computer readable media, in the form of random access memory (RAM) and/or nonvolatile memory such as read only memory (ROM) or flash RAM. The memory is an example of a computer readable medium.
计算机可读介质包括永久性和非永久性、可移动和非可移动存储介质。存储介质可以由任何方法或技术来实现信息存储，信息可以是计算机可读指令、数据结构、程序的模块或其他数据。计算机的存储介质的例子包括，但不限于相变内存(PRAM)、静态随机存取存储器 (SRAM)、动态随机存取存储器(DRAM)、其他类型的随机存取存储器(RAM)、只读存储器(ROM)、电可擦除可编程只读存储器(EEPROM)、快闪记忆体或其他内存技术、只读光盘只读存储器(CD-ROM)、数字多功能光盘(DVD)或其他光学存储、磁盒式磁带，磁盘存储或其他磁性存储设备或任何其他非传输介质，可用于存储可以被计算设备访问的信息。根据本文中的界定，计算机可读介质不包括暂存电脑可读媒体(transitory media)，如调制的数据信号和载波。Computer-readable media includes both volatile and non-volatile, removable and non-removable storage media. The storage medium may store information by any method or technology, and the information may be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Flash memory or other memory technology, Compact Disc Read-Only Memory (CD-ROM), Digital Versatile Disc (DVD) or other optical storage, A magnetic tape cartridge, disk storage or other magnetic storage device or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media excludes transitory computer readable media, such as modulated data signals and carrier waves.
最后应说明的是：以上各实施例仅用以说明本申请的技术方案，而非对其限制；尽管参照前述各实施例对本申请进行了详细的说明，本领域的普通技术人员应当理解：其依然可以对前述各实施例所记载的技术方案进行修改，或者对其中部分或者全部技术特征进行等同替换；而这些修改或者替换，并不使相应技术方案的本质脱离本申请各实施例技术方案的范围。Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and are not intended to limit it; although the application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: It is still possible to modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the technical solutions of the various embodiments of the present application. scope.

Claims (12)

  1. 一种模糊图像生成方法，其包括：A method for generating blurred images, comprising:
    获取第一图像集合中各个图像的模糊核，生成模糊核池；所述第一图像集合包括多个分辨率小于第一阈值的图像；Acquiring the blur kernels of each image in the first image collection to generate a blur kernel pool; the first image collection includes a plurality of images with a resolution smaller than the first threshold;
    从所述模糊核池中选取第二图像集合中的各个图像对应的模糊核；所述第二图像集合包括多个分辨率大于第二阈值的图像；Selecting a blur kernel corresponding to each image in a second image set from the blur kernel pool; the second image set includes a plurality of images with a resolution greater than a second threshold;
    通过所述第二图像集合中的各个图像对应的模糊核对所述第二图像集合中的各个图像进行降质，获取所述第二图像集合中各个图像对应的模糊图像。Each image in the second image set is degraded through a blur kernel corresponding to each image in the second image set, and a blurred image corresponding to each image in the second image set is acquired.
  2. 根据权利要求1所述的方法，其中，所述获取第一图像集合中各个图像的模糊核，包括：The method according to claim 1, wherein said obtaining the blur kernel of each image in the first image collection comprises:
    随机生成所述第一图像集合中的目标图像对应的第一噪声和第二噪声；所述第一噪声和第二噪声均满足正态分布；Randomly generating first noise and second noise corresponding to the target image in the first image set; both the first noise and the second noise satisfy a normal distribution;
    将所述第一噪声输入深度图像先验DIP模型，并获取所述DIP模型输出的第一图像；Input the first noise into the depth image prior DIP model, and obtain the first image output by the DIP model;
    将所述第二噪声输入基于流的核先验FKP模型，并获取所述FKP模型输出的预测模糊核；Inputting the second noise into a flow-based kernel prior FKP model, and obtaining a predictive blur kernel output by the FKP model;
    通过所述预测模糊核对所述第一图像进行降质，获取第二图像；Degrading the first image through the predictive blur kernel to obtain a second image;
    基于所述第二图像和所述目标图像判断是否满足收敛条件；judging whether a convergence condition is met based on the second image and the target image;
    若否，则更新模型参数和/或模型输入，并在更新所述模型参数和/或所述模型输入后，判断重新获取的第二图像和所述目标图像是否满足收敛条件，直至所述第二图像和所述目标图像满足所述收敛条件；If not, update the model parameters and/or model input, and after updating the model parameters and/or the model input, judge whether the reacquired second image and the target image meet the convergence condition until the first The second image and the target image satisfy the convergence condition;
    若是，则将所述FKP模型输出的预测模糊核确定为所述目标图像的模糊核。If yes, the predicted blur kernel output by the FKP model is determined as the blur kernel of the target image.
  3. 根据权利要求2所述的方法，其中，所述更新模型参数和/或模型输入，包括：The method according to claim 2, wherein said updating model parameters and/or model inputs comprises:
    更新所述DIP模型的模型参数和/或所述FKP模型的模型输入。Updating model parameters of the DIP model and/or model inputs of the FKP model.
  4. 根据权利要求1所述的方法，其中，所述从所述模糊核池中选取第二图像集合中的各个图像对应的模糊核，包括：The method according to claim 1, wherein said selecting a blur kernel corresponding to each image in the second image set from the blur kernel pool comprises:
    针对所述第二图像集合中的每一个图像，随机从所述模糊核池中选取对应的模糊核。For each image in the second image set, randomly select a corresponding blur kernel from the blur kernel pool.
  5. 根据权利要求1所述的方法，其中，所述从所述模糊核池中选取第二图像集合中的各个图像对应的模糊核，包括：The method according to claim 1, wherein said selecting a blur kernel corresponding to each image in the second image set from the blur kernel pool includes:
    基于图像的场景将所述第一图像集合中的图像划分为多个第一子图像集合；dividing the images in the first image set into a plurality of first sub-image sets based on the image scene;
    将属于同一所述第一子图像集合的图像的模糊核，划分为一个模糊核组；dividing the blur kernels of the images belonging to the same first sub-image set into a blur kernel group;
    基于图像的场景将所述第二图像集合中的图像划分为多个第二子图像集合；dividing the images in the second image set into a plurality of second sub-image sets based on the image scene;
    对属于同一所述第二子图像集合的图像，从同一所述模糊核组中随机选取对应的模糊核。For images belonging to the same set of second sub-images, a corresponding blur kernel is randomly selected from the same blur kernel group.
  6. 根据权利要求5所述的方法，其中，所述第一图像集合由第一视频的图像帧组成，所述第二图像集合由第二视频的图像帧组成；The method according to claim 5, wherein the first set of images consists of image frames of a first video, and the second set of images consists of image frames of a second video;
    所述基于图像的场景将所述第一图像集合中的图像划分为多个第一子图像集合，包括：基于图像的场景将所述第一视频划分为多个第一视频片段，并分别将各个所述第一视频片段的图像帧划分一个所述第一子图像集合；The image-based scenario divides the images in the first image set into a plurality of first sub-image sets, including: dividing the first video into a plurality of first video clips in the image-based scenario, and respectively The image frames of each of the first video segments are divided into a set of the first sub-images;
    所述基于图像的场景将所述第一图像集合中的图像划分为多个第二子图像集合，包括：基于图像的场景将所述第二视频划分为多个第二视频片段，并分别将各个所述第二视频片段的图像帧划分一个所述第二子图像集合。The image-based scene divides the images in the first image set into a plurality of second sub-image sets, including: dividing the second video into a plurality of second video clips in the image-based scene, and respectively Each image frame of the second video segment is divided into one second sub-image set.
  7. 一种网络模型训练方法，其包括：A network model training method, comprising:
    获取样本图像集合，所述样本图像集合包括多个分辨率大于阈值分辨率的样本图像；Acquiring a set of sample images, the set of sample images including a plurality of sample images with a resolution greater than a threshold resolution;
    通过权利要求1-6任一项所述的模糊图像生成方法，获取所述样本图像集合中各个样本图像对应的模糊图像；Obtaining a blurred image corresponding to each sample image in the sample image set by the method for generating a blurred image according to any one of claims 1-6;
    根据所述样本图像集合中的各个样本图像以及各个样本图像对应的模糊图像，生成训练数据集；generating a training data set according to each sample image in the sample image set and the blurred image corresponding to each sample image;
    通过所述训练数据集对用于对模糊图像进行修复的图像修复网络模型进行训练。An image inpainting network model for inpainting blurred images is trained by the training data set.
  8. 一种模糊图像生成装置，其包括：A device for generating a blurred image, comprising:
    获取单元，用于获取第一图像集合中各个图像的模糊核；生成模糊核池，所述第一图像集合包括多个分辨率小于第一阈值的图像；An acquisition unit, configured to acquire the blur kernel of each image in the first image set; generate a blur kernel pool, the first image set includes a plurality of images with a resolution smaller than a first threshold;
    选取单元，用于从所述模糊核池中选取第二图像集合中的各个图像对应的模糊核；所述第二图像集合包括多个分辨率大于第二阈值的图像；A selection unit, configured to select a blur kernel corresponding to each image in a second image set from the blur kernel pool; the second image set includes a plurality of images with a resolution greater than a second threshold;
    处理单元，用于通过所述第二图像集合中的各个图像对应的模糊核对所述第二图像集合中的各个图像进行降质，获取所述第二图像集合中各个图像对应的模糊图像。A processing unit, configured to degrade each image in the second image set through a blur kernel corresponding to each image in the second image set, and acquire a blurred image corresponding to each image in the second image set.
  9. 一种网络模型训练装置，其包括：A network model training device, comprising:
    获取单元，用于获取样本图像集合，所述样本图像集合包括多个分辨率大于阈值分辨率的样本图像；an acquiring unit, configured to acquire a sample image set, the sample image set including a plurality of sample images with a resolution greater than a threshold resolution;
    处理单元，用于通过权利要求1-6任一项所述的模糊图像生成方法，获取所述样本图像集合中各个样本图像对应的模糊图像；A processing unit, configured to obtain a blurred image corresponding to each sample image in the sample image set through the method for generating a blurred image according to any one of claims 1-6;
    生成单元，用于根据所述样本图像集合中的各个样本图像以及各个样本图像对应的模糊图像，生成训练数据集；A generation unit, configured to generate a training data set according to each sample image in the sample image set and the blurred image corresponding to each sample image;
    训练单元，用于通过所述训练数据集对预设网络模型进行训练，获取用于对模糊图像进行修复的图像修复网络模型。The training unit is configured to train a preset network model through the training data set, and obtain an image repair network model for repairing blurred images.
  10. 一种电子设备，其包括：存储器和处理器，所述存储器用于存储计算机程序；所述 处理器用于在调用计算机程序时，使得所述电子设备实现权利要求1-7任一项所述的方法。An electronic device, comprising: a memory and a processor, the memory is used to store a computer program; the processor is used to enable the electronic device to implement the method described in any one of claims 1-7 when calling the computer program method.
  11. 一种计算机可读存储介质，其中，所述计算机可读存储介质上存储有计算机程序，当所述计算机程序被计算设备执行时，使得所述计算设备实现权利要求1-7任一项所述的方法。A computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a computing device, the computing device realizes any one of claims 1-7. Methods.
  12. 一种计算机程序产品，其中，当所述计算机程序产品在计算机上运行时，使得所述计算机实现如权利要求1-7任一项所述的方法。A computer program product, wherein, when the computer program product is run on a computer, the computer is made to implement the method according to any one of claims 1-7.
PCT/CN2022/127384 2021-10-26 2022-10-25 Blurred image generating method and apparatus, and network model training method and apparatus WO2023072072A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202111250062.4A CN116029911A (en) 2021-10-26 2021-10-26 Fuzzy image generation method, network model training method and device
CN202111250062.4 2021-10-26

Publications (1)

Publication Number Publication Date
WO2023072072A1 true WO2023072072A1 (en) 2023-05-04

Family

ID=86069300

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/127384 WO2023072072A1 (en) 2021-10-26 2022-10-25 Blurred image generating method and apparatus, and network model training method and apparatus

Country Status (2)

Country Link
CN (1) CN116029911A (en)
WO (1) WO2023072072A1 (en)

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20170357871A1 (en) * 2016-06-10 2017-12-14 Apple Inc. Hierarchical Sharpness Evaluation
CN108364262A (en) * 2018-01-11 2018-08-03 深圳大学 A kind of restored method of blurred picture, device, equipment and storage medium
CN113240581A (en) * 2021-04-09 2021-08-10 辽宁工程技术大学 Real world image super-resolution method for unknown fuzzy kernel

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20170357871A1 (en) * 2016-06-10 2017-12-14 Apple Inc. Hierarchical Sharpness Evaluation
CN108364262A (en) * 2018-01-11 2018-08-03 深圳大学 A kind of restored method of blurred picture, device, equipment and storage medium
CN113240581A (en) * 2021-04-09 2021-08-10 辽宁工程技术大学 Real world image super-resolution method for unknown fuzzy kernel

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
WANG, LAN; KONG, XIANGYI; ZHANG, HAITAO: "Image Super-Resolution Based on Blur Kernel Correction with Unknown Degradation Method", COMPUTER ENGINEERING AND APPLICATIONS, HUABEI JISUAN JISHU YANJIUSUO, CN, vol. 21, no. 58, 6 July 2021 (2021-07-06), CN , pages 232 - 242, XP009545734, ISSN: 1002-8331, DOI: 10.3778/j.issn.1002-8331.2103-0530 *

Also Published As

Publication number Publication date
CN116029911A (en) 2023-04-28

Similar Documents

Publication Publication Date Title
Reda et al. Film: Frame interpolation for large motion
US11449966B2 (en) Real-time video ultra resolution
CN108122197B (en) Image super-resolution reconstruction method based on deep learning
Zhu et al. GAN‐Based Image Super‐Resolution with a Novel Quality Loss
WO2021233006A1 (en) Image processing model training method and apparatus, image processing method and apparatus, and device
EP3341911B1 (en) Image upscaling
CN102576454B (en) Image deblurring using a spatial image prior
CN111784570A (en) Video image super-resolution reconstruction method and device
WO2022077978A1 (en) Video processing method and video processing apparatus
US8391626B2 (en) Learning of coefficients for motion deblurring by pixel classification and constraint condition weight computation
US11741579B2 (en) Methods and systems for deblurring blurry images
Jiang et al. Deep edge map guided depth super resolution
Kokaram et al. Motion‐based frame interpolation for film and television effects
JP2011250013A (en) Image quality evaluation method, image quality evaluation apparatus, and program
US20230138053A1 (en) Systems and methods for optical flow estimation
WO2021179954A1 (en) Video processing method and apparatus, device, and storage medium
Hung et al. Image interpolation using convolutional neural networks with deep recursive residual learning
WO2023072072A1 (en) Blurred image generating method and apparatus, and network model training method and apparatus
US10819983B1 (en) Determining a blurriness score for screen capture videos
Zhu et al. Eednet: enhanced encoder-decoder network for autoisp
US20220327663A1 (en) Video Super-Resolution using Deep Neural Networks
Zhang et al. Video Superresolution Reconstruction Using Iterative Back Projection with Critical‐Point Filters Based Image Matching
TWI417810B (en) Image enhancement method, image enhancement apparaus and image processing circuit
CN113610031A (en) Video processing method and video processing device
KR102325898B1 (en) System and method for multi-domain image restoration

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22885949

Country of ref document: EP

Kind code of ref document: A1