CN115829865A - Image completion method, system, device and storage medium based on model prior - Google Patents
Image completion method, system, device and storage medium based on model prior Download PDFInfo
- Publication number
- CN115829865A CN115829865A CN202211438318.9A CN202211438318A CN115829865A CN 115829865 A CN115829865 A CN 115829865A CN 202211438318 A CN202211438318 A CN 202211438318A CN 115829865 A CN115829865 A CN 115829865A
- Authority
- CN
- China
- Prior art keywords
- model
- image
- training
- completion
- sample
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 74
- 238000012549 training Methods 0.000 claims abstract description 255
- 230000006870 function Effects 0.000 claims description 35
- 238000012545 processing Methods 0.000 claims description 34
- 238000012216 screening Methods 0.000 claims description 6
- 238000004590 computer program Methods 0.000 claims description 3
- 230000000694 effects Effects 0.000 description 11
- 238000010586 diagram Methods 0.000 description 5
- 238000005457 optimization Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000013145 classification model Methods 0.000 description 1
- 238000009792 diffusion process Methods 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 230000000873 masking effect Effects 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 230000009469 supplementation Effects 0.000 description 1
Images
Landscapes
- Image Processing (AREA)
Abstract
The embodiment of the application discloses an image completion method, system, equipment and storage medium based on model prior. According to the technical scheme provided by the embodiment of the application, missing images and the specified completion area information of the missing images are obtained; and inputting the missing image and the information of the specified completion area into a pre-constructed image completion model, and outputting a completion image of the missing image. The image completion model takes model parameters when the pre-training model converges as initial model parameters, and performs image completion training based on training samples of the pre-training model and prior samples generated by the pre-training model until the image completion model converges. The pre-training model carries out image completion training in advance based on training samples, and after the pre-training model converges, a priori sample is generated based on the pre-training model. By adopting the technical means, more real and various image completion results can be provided, and the diversity of the image completion model completion images is improved.
Description
Technical Field
The embodiment of the application relates to the technical field of computer vision, in particular to an image completion method, system, equipment and storage medium based on model prior.
Background
At present, the image completion technology is widely applied to various scenes such as short videos, live broadcasts and the like. Image inpainting technology (image inpainting) is an important technical direction in the field of computer vision, and is used for inpainting invisible or missing parts of an image, meanwhile, the whole image is reasonable, no sense of incongruity exists, and the inpainted image is as real as possible. The existing image completion technology is mainly realized based on an image completion model, and the image completion model is constructed based on a generation countermeasure network. By training the image completion model based on the generated countermeasure network, some areas are randomly masked for completion during training, thereby realizing image completion.
However, the existing image completion model obtained by simply performing model training through masking image partial regions is relatively single and lacks diversity. The obtained completion image has relatively poor completion effect, and the completion image is easy to have a sense of incongruity.
Disclosure of Invention
The embodiment of the application provides an image completion method, system, equipment and storage medium based on model prior, which can improve the diversity of the completion image, improve the image completion effect and solve the technical problems of single and inconsistent image completion effect of the existing image completion model.
In a first aspect, an embodiment of the present application provides an image completion method based on model prior, including:
acquiring a missing image and specified completion area information of the missing image;
inputting the missing image and the specified completion region information into a pre-constructed image completion model, outputting a completion image of the missing image, using a model parameter when the pre-training model is converged as an initial model parameter by the image completion model, and performing image completion training on the basis of a training sample of the pre-training model and a prior sample generated by the pre-training model until the image completion model is converged;
the pre-training model carries out image completion training in advance based on the training samples, and after the pre-training model converges, a priori sample is generated based on the pre-training model and comprises a noise-added image and a plurality of de-noising images obtained by image completion of the corresponding noise-added image.
In a second aspect, an embodiment of the present application provides an image completion system based on model prior, including:
the acquisition module is configured to acquire a missing image and the specified completion area information of the missing image;
the image completion model takes model parameters during the convergence of the pre-training model as initial model parameters, and carries out image completion training based on training samples of the pre-training model and prior samples generated by the pre-training model until the image completion model converges; the pre-training model carries out image completion training in advance based on the training sample, and after the pre-training model is converged, a prior sample is generated based on the pre-training model and comprises a noise-added image and a plurality of de-noising images obtained by image completion of the corresponding noise-added image.
In a third aspect, an embodiment of the present application provides an image completion apparatus based on model prior, including:
a memory and one or more processors;
the memory configured to store one or more programs;
when executed by the one or more processors, cause the one or more processors to implement the model prior-based image completion method of the first aspect.
In a fourth aspect, embodiments of the present application provide a computer-readable storage medium having stored thereon computer-executable instructions that, when executed by a computer processor, are configured to perform the model prior-based image completion method according to the first aspect.
In a fifth aspect, the present application provides a computer program product containing instructions that, when executed on a computer or processor, cause the computer or processor to perform the model prior-based image completion method according to the first aspect.
According to the method and the device, missing images and the specified completion area information of the missing images are obtained; and inputting the missing image and the information of the specified completion area into a pre-constructed image completion model, and outputting a completion image of the missing image. The image completion model takes model parameters of the pre-training model during convergence as initial model parameters, and performs image completion training based on training samples of the pre-training model and prior samples generated by the pre-training model until the image completion model converges. The pre-training model carries out image completion training in advance based on the training samples, and after the pre-training model converges, the prior samples are generated based on the pre-training model. Here, the prior sample includes a noisy image and a plurality of denoised images obtained by performing image completion on the noisy image. By adopting the technical means, the image completion model training is carried out through the model parameters, the training samples and the prior samples provided by the pre-training model, and the obtained image completion model can provide more real and various image completion results. The optimization training is carried out on the basis of the pre-training model, the diversity of the image completion model completion images is further improved, image completion flaws and the sense of incongruity are reduced, and the completion images are more real and stable.
Drawings
Fig. 1 is a flowchart of an image completion method based on model prior provided in an embodiment of the present application;
FIG. 2 is a training flow diagram of a pre-training model according to an embodiment of the present application;
FIG. 3 is a schematic diagram of image denoising and denoising processing according to an embodiment of the present application;
FIG. 4 is a flow chart of the training of the image completion model according to the embodiment of the present application;
fig. 5 is a schematic structural diagram of an image completion system based on model prior provided in an embodiment of the present application;
fig. 6 is a schematic structural diagram of an image completion apparatus based on model prior according to an embodiment of the present application.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, specific embodiments of the present application will be described in detail with reference to the accompanying drawings. It is to be understood that the specific embodiments described herein are merely illustrative of the application and are not limiting of the application. It should be further noted that, for the convenience of description, only some but not all of the relevant portions of the present application are shown in the drawings. Before discussing exemplary embodiments in more detail, it should be noted that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although a flowchart may describe the operations (or steps) as a sequential process, many of the operations can be performed in parallel, concurrently or simultaneously. In addition, the order of the operations may be re-arranged. The process may be terminated when its operations are completed, but may have additional steps not included in the figure. The processes may correspond to methods, functions, procedures, subroutines, and the like.
The application provides an image completion method based on model prior, aims at carrying out image completion model training through model parameters, training samples and prior samples provided by a pre-training model, and improves the diversity of completion images on the basis of the pre-training model, so that the completion images generated by the image completion model are more real and stable. For the conventional image completion scheme, when image completion is performed, one method is to fill the completion information into the missing region step by step from the outside of the missing position by using an interpolation mode, and this mode can well complete a small region, but the completion effect is not good for a large region. And the other method is to adopt an image completion method based on the generation of a countermeasure network, generate a countermeasure model through training, and randomly shield some areas for image completion during training, so as to realize image completion. Although this method can fill a large area, the filling image is relatively single and has insufficient diversity. Therefore, the image completion method based on the model prior is provided to solve the technical problems that the existing image completion model is single in image completion effect and not harmonious.
Example (b):
fig. 1 shows a flowchart of an image completion method based on model prior provided in an embodiment of the present application, where the image completion method based on model prior provided in this embodiment may be executed by an image completion device based on model prior, the image completion device based on model prior may be implemented in a software and/or hardware manner, and the image completion device based on model prior may be formed by two or more physical entities or may be formed by one physical entity. Generally, the image completion device based on model prior may be a computer, an image processing server, a mobile phone, a tablet or other processing device.
The following description will be given taking the model prior-based image completion apparatus as an example of a subject that performs the model prior-based image completion method. Referring to fig. 1, the image completion method based on model prior specifically includes:
s110, acquiring a missing image and appointed completion area information of the missing image;
s120, inputting the missing image and the information of the specified completion area into a pre-constructed image completion model, outputting a completion image of the missing image, wherein the image completion model takes model parameters when the pre-training model converges as initial model parameters, and performs image completion training on the basis of a training sample of the pre-training model and a prior sample generated by the pre-training model until the image completion model converges; the pre-training model carries out image completion training in advance based on the training sample, and after the pre-training model is converged, a prior sample is generated based on the pre-training model and comprises a noise-added image and a plurality of de-noising images obtained by image completion of the corresponding noise-added image.
According to the image completion method based on model prior, when image completion is carried out, image completion processing is carried out on an image to be completed through a pre-trained image completion model, so that the missing part of the image is completed. Defining an image to be complemented as a missing image, and an image subjected to image complementing processing as a complementing image, inputting the missing image into the image complementing model, predicting a part of missing area of the missing image through the image complementing model, and further outputting a corresponding complementing image.
In addition, in order to enable the completion images output by the image completion model to be more various and real, the prior information of the pre-training model is used for training the image completion model, so that the image completion model has the basic image completion capability of the pre-training model and the capability of generating various completion images.
Before this, a pre-training model, which is a diffusion model, is first trained. A variety of a priori samples are then generated using the pre-trained model. The prior sample comprises a noise-added image and a plurality of de-noised images obtained by image completion of the corresponding noise-added image, and each noise-added image and the de-noised images obtained by image completion processing corresponding to the noise-added image form a data pair for subsequent image completion model training. The model parameters of the pre-training model are used as initial model parameters of the image completion model, training samples of the pre-training model and generated prior samples are input into the image completion model, and the image completion model is trained until the model converges. Because one noisy image in the prior sample comprises a plurality of denoised images, the diversity of image completion processing results can be embodied. And a priori sample which can reflect the diversity of the image completion processing result is added on the basis of the training sample subsequently, so that the diversity of the image completion model obtained by training is stronger, and the image completion effect is better. And the image completion model further performs image completion optimization training by taking the model parameters after the pre-training model training convergence as initial model parameters, so that the image completion result of the model has fewer flaws, the image completion is more real, and the model processing speed is higher.
Specifically, referring to fig. 2, the training process of the pre-training model includes:
s101, carrying out noise adding treatment on a training sample based on a pre-training model to obtain a noise adding sample;
s102, performing image completion on the noise-added sample based on the pre-training model to obtain a de-noised sample, calculating a first model loss function according to the de-noised sample, and iteratively adjusting model parameters of the pre-training model based on the first model loss function until the pre-training model is converged.
The training process of the pre-training model comprises the noise adding and denoising process of the training sample. Such asAs shown in fig. 3, the noise adding process is a forward process, and is configured to add image noise to a sample image of a training sample, so that a missing region is formed on the sample image, and is used for performing subsequent image completion training to perform prediction completion on the missing region. The denoising process is the inverse process of the denoising process, and the original sample image is obtained by denoising and predicting the denoised image and continuously reasoning. Referring to FIG. 3, x 0 To x T And the processing process of the sample image from time 0 to time T, namely a noise adding process, wherein noise is added to the real sample image continuously. x is the number of T To x 0 Representing the processing process of the sample image from time T to 0, namely the denoising process, wherein denoising reasoning is continuously carried out in the process, so that an initial sample image, namely the sample image x at the time point 0 is predicted and obtained 0 。
Wherein, add the noise processing and obtain the sample of making a noise to training sample based on the model of training in advance, include:
adding Gaussian distribution noise to training images of training samples based on a pre-training model at each designated moment in a set time period to obtain Gaussian noise distribution probability of each training image, and taking the Gaussian noise distribution probability of each training image as a noise adding sample;
the image completion is carried out on the noise adding sample based on the pre-training model to obtain a noise removing sample, and the method comprises the following steps:
and sequentially predicting the real-time noise distribution probability of each appointed time in a set time period based on the Gaussian noise distribution probability of each training image and the set estimation parameters, and taking the prediction result of the first appointed time in the set time period as a denoising sample.
Taking a training image as shown in fig. 3 as an example, noise adding or denoising processing is performed at a specified time within a set time period, so that a real-time noise distribution probability of any specified time t can be obtained:
and specifying the total gaussian noise distribution probability from time 0 to T, i.e. the gaussian noise distribution probability of the training image:
wherein β represents a hyper-parameter which is a variance of a predetermined Gaussian distribution, and β may be 1 e-4 To 1 e-2 T represents a specified time, the time step length thereof is fixed, N represents Gaussian distribution, T represents the total time length of a set time period, x t Representing the noisy processed image at time t.
By giving a sample image x0, based on the noise adding process, gaussian distribution noise is added at each appointed time to obtain x 1 ，x 2 …x T And as the time step is continuously increased, finally obtaining the image x containing the Gaussian noise distribution information of the sample image T 。
Wherein for a given x 0 Then, time t has:
here, α s And alpha t Representing a step size hyperparameter, I represents a normal gaussian distribution,
α t ＝1-β t 。
further, since the noise adding process is to continuously add the gaussian distribution noise to the specified time according to the set time period, the real-time noise distribution probability q (x) is not directly and theoretically inferred in the noise removing process t-1 |x t ). According to the embodiment of the application, the real-time noise distribution probability of each noise-added image at each appointed time in a set time period is predicted based on the noise-added original text after the noise-added processing through a set estimation parameter theta, namely the real-time noise distribution probability predicted at any appointed time t:
p θ (x t-1 |x t ):＝N(x t-1 ；μ θ (x t ,t),∑ θ (x t ,t)) (1)
then the total predicted gaussian noise distribution probability from the given time 0 to T is expressed as:
then at a given sample image x 0 Can be obtained by a Bayesian formula:
and can obtain:
where μ represents the predicted probability mean and ε represents the predicted noise distribution.
In the process of denoising, the Gaussian noise of each step can pass through x t And t predicts ε θ (x t T), and then the probability mean value mu (x) can be obtained according to the formula (4) t T) in combination with equations (1), (2) and (3), a predicted real-time noise distribution probability q (x) can be obtained t-1 |x t ) Thereby predicting the image x t-1 。
Based on the principle, the pre-training model is subjected to noise adding and noise removing training, during training, the selected step length T is 1000, beta is linearly increased from 0.0004 to 0.002, the learning rate is initially set to 0.00001, and then attenuation is carried out according to the training times. And during training, randomly shielding the sample image for noise adding, performing image completion training based on the training flow, and performing loss function calculation on the noise-removed sample predicted in the training process. And defining a loss function of the pre-training model as a first model loss function, and iteratively adjusting model parameters of the pre-training model based on the first model loss function until the pre-training model converges. The first model loss function adopts an L1 loss function, and is expressed as:
Loss＝abs(M(x)–N(u,σ))
where M (x) represents the probability distribution of the image reality, N represents the gaussian distribution, u represents the mean of the noise, and σ represents the variance of the noise.
Further, based on a trained pre-training model, various data pairs, i.e., prior samples, are generated using the pre-training model, wherein generating the prior samples based on the pre-training model comprises:
inputting the noise-added images into a pre-training model, performing image completion of the noise-added images for a set number of times based on the pre-training model to obtain a corresponding number of noise-added images, and taking the set number of noise-added images and the corresponding noise-added images as prior samples.
The noisy image may be a noisy image obtained by subjecting a sample image of the training sample to noise processing by a pre-training model, or an image subjected to occlusion processing may be input to the pre-training model as the noisy image. For the same noise image, various reasonable completion effects also need to be generated, so that a plurality of different noise-removed images are obtained, and a plurality of data pairs with non-unique results are constructed, so that the diversity of data is increased. It can be understood that for the model image completion processing, only image completion is required to be reasonable, similar answers are not unique, and the data set is more consistent with actual reasoning characteristics. In addition, the position of the completion, the size of the completion, and the pattern shape need to be random, so that the more similar completion data is generated, the higher the diversity of the data is.
Further, because the quality of the image subjected to model random completion is relatively difficult to guarantee, some de-noised images with poor effects exist. In order to improve the training effect of the image completion model and make the completed image more real and stable, the prior artifact needs to be cleaned. Optionally, in the embodiment of the present application, based on a prior sample generated by a pre-training model, a completion processing score of each denoised image is calculated based on the denoised image and the corresponding denoised image, and data screening is performed on the prior sample based on the completion processing score.
The method comprises the steps of training a randomly generated graph to generate a confrontation model, and training an evaluation capability of a discriminator to generate the denoising effect of the graph. And using the trained arbiter for generating the confrontation model for data screening of the prior sample. The denoised image is input into the discriminator, the discriminator outputs corresponding completion processing scores according to the denoising quality, and partial data with relatively high completion processing scores are screened according to the completion processing scores to perform image completion model training.
Optionally, the generated completion processing scores are sorted based on the completion processing scores of the denoised images, and a completion processing score distribution curve is drawn. And selecting a threshold according to the distribution curve, storing the denoised image higher than the threshold, and constructing a final prior sample for training an image completion model. In practical application, a two-classification model can be trained for data screening of a priori sample. The specific data screening method is not fixedly limited in the embodiment of the present application, and is not described herein again.
Further, based on the screened prior sample, the training sample of the pre-training model and the model parameters during model convergence are combined to perform the training of the image completion model. Referring to fig. 4, the training process of the image completion model includes:
s103, inputting a training sample and a prior sample into an image completion model by taking a model parameter during the convergence of a pre-training model as an initial model parameter;
and S104, performing image completion training by using the training sample and the prior sample based on the image completion model, calculating a second model loss function based on the training result, and iteratively adjusting initial model parameters based on the second model loss function until the image completion model converges.
The model parameters of the pre-training model during convergence are used as initial model parameters, so that the convergence of the image completion model is faster, and the model training efficiency is improved. In addition, the pre-training model has certain image completion capability, so that the model is further trained on the basis, the image completion model can inherit the advantages of the original model, the image completion effect of the model is optimized, and the generated completion image is more real and stable. Meanwhile, the model can be trained through the prior sample, so that the model has the capability of generating various completion images, and the diversity of the completion images is improved. In addition, through various sample training, the image completion model can adapt to the image completion of different missing images, the image completion can be carried out more efficiently compared with a pre-training model, and the time consumption of the model is in the millisecond level.
And carrying out model training based on prior information provided by the pre-training model, and calculating a loss function according to a completion image generated by the image completion model in the training process. Defining the loss function as a second model loss function, wherein the second model loss function comprises a perception loss function and an L1 loss function. The perceptual loss function is expressed as:
Perceptual loss＝E((VGG(x)-VGG(M(x)))2)
the L1 loss function is expressed as:
L1_loss＝E(x–M(x))
where X denotes a real image and M (X) denotes a predicted image.
And during training, an Adam optimizer is adopted, the learning rate is set to be 0.0001, a second loss function is calculated according to the training result of the image completion model, and initial model parameters are continuously adjusted in an iterative mode according to the value of the second loss function until the image completion model is converged, so that model training is completed.
And then based on the trained image completion model, acquiring the missing image to be completed and the specified completion region information of the missing image, inputting the missing image and the specified completion region information into a pre-constructed image completion model, performing image completion processing based on the image completion model, predicting the image information at the specified completion region, and fusing the predicted completion region back to the missing image to obtain the final completion image.
In practical application, for example, a user inputs a to-be-completed image, specifies an area to be completed, and performs image completion processing through an image completion model, so as to output a completed image. By inputting a section of video and designating the area needing to be supplemented in the video, the image information of the supplemented area is predicted frame by frame through the image supplementation model, the image can be restored frame by frame, and the supplemented video is output.
Obtaining the missing image and the specified completion area information of the missing image; and inputting the missing image and the information of the specified completion area into a pre-constructed image completion model, and outputting a completion image of the missing image. The image completion model takes model parameters when the pre-training model converges as initial model parameters, and performs image completion training based on training samples of the pre-training model and prior samples generated by the pre-training model until the image completion model converges. The pre-training model carries out image completion training in advance based on the training samples, and after the pre-training model converges, the prior samples are generated based on the pre-training model. Here, the prior sample includes a noisy image and a plurality of denoised images obtained by image completion corresponding to the noisy image. By adopting the technical means, the image completion model training is carried out through the model parameters, the training samples and the prior samples provided by the pre-training model, and the obtained image completion model can provide more real and various image completion results. The optimization training is carried out on the basis of the pre-training model, the diversity of the image completion model completion image is further improved, the image completion flaws and the sense of incongruity are reduced, and the completion image is more real and stable.
On the basis of the foregoing embodiment, fig. 5 is a schematic structural diagram of an image completion system based on model prior provided by the present application. Referring to fig. 5, the image completion system based on model prior provided in this embodiment specifically includes: an acquisition module 21 and a completion module 22.
The obtaining module 21 is configured to obtain a missing image and specified completion region information of the missing image;
the completion module 22 is configured to input the missing image and the information of the specified completion region into a pre-constructed image completion model, and output a completion image of the missing image, wherein the image completion model takes a model parameter when the pre-training model converges as an initial model parameter, and performs image completion training based on a training sample of the pre-training model and a prior sample generated by the pre-training model until the image completion model converges; the pre-training model carries out image completion training in advance based on the training sample, and after the pre-training model is converged, a prior sample is generated based on the pre-training model and comprises a noise-added image and a plurality of de-noising images obtained by image completion of the corresponding noise-added image.
Specifically, the training process of the pre-training model comprises the following steps:
carrying out noise adding treatment on the training sample based on a pre-training model to obtain a noise adding sample;
and performing image completion on the noise-added sample based on the pre-training model to obtain a de-noised sample, calculating a first model loss function according to the de-noised sample, and iteratively adjusting model parameters of the pre-training model based on the first model loss function until the pre-training model is converged.
Wherein, add the noise processing and obtain the sample of making a noise to training sample based on the model of training in advance, include:
adding Gaussian distribution noise to training images of training samples based on a pre-training model at each designated moment in a set time period to obtain Gaussian noise distribution probability of each training image, and taking the Gaussian noise distribution probability of each training image as a noise adding sample;
the image completion is carried out on the noise adding sample based on the pre-training model to obtain a noise removing sample, and the method comprises the following steps:
and sequentially predicting the real-time noise distribution probability of each appointed time in a set time period based on the Gaussian noise distribution probability of each training image and set estimation parameters, and taking the prediction result of the first appointed time in the set time period as a denoising sample.
Generating prior samples based on a pre-trained model, comprising:
inputting the noise-added images into a pre-training model, performing image completion of the noise-added images for a set number of times based on the pre-training model to obtain a corresponding number of noise-added images, and taking the set number of noise-added images and the corresponding noise-added images as prior samples.
And calculating a completion processing score of each denoised image based on the denoised image and the corresponding denoised image, and carrying out data screening on the prior sample based on the completion processing score.
Specifically, the training process of the image completion model includes:
taking a model parameter when a pre-training model converges as an initial model parameter, and inputting a training sample and a prior sample into an image completion model;
and performing image completion training by using the training sample and the prior sample based on the image completion model, calculating a second model loss function based on the training result, and iteratively adjusting the initial model parameters based on the second model loss function until the image completion model is converged.
Wherein the second model loss function comprises a perceptual loss function and an L1 loss function.
Obtaining the missing image and the specified completion area information of the missing image; and inputting the missing image and the information of the specified completion area into a pre-constructed image completion model, and outputting a completion image of the missing image. The image completion model takes model parameters when the pre-training model converges as initial model parameters, and performs image completion training based on training samples of the pre-training model and prior samples generated by the pre-training model until the image completion model converges. The pre-training model carries out image completion training in advance based on the training samples, and after the pre-training model converges, the prior samples are generated based on the pre-training model. Here, the prior sample includes a noisy image and a plurality of denoised images obtained by image completion corresponding to the noisy image. By adopting the technical means, the image completion model training is carried out through the model parameters, the training samples and the prior samples provided by the pre-training model, and the obtained image completion model can provide more real and various image completion results. The optimization training is carried out on the basis of the pre-training model, the diversity of the image completion model completion image is further improved, the image completion flaws and the sense of incongruity are reduced, and the completion image is more real and stable.
The image completion system based on the model prior provided by the embodiment of the application can be configured to execute the image completion method based on the model prior provided by the embodiment, and has corresponding functions and beneficial effects.
On the basis of the above practical example, an embodiment of the present application further provides an image completion apparatus based on model prior, with reference to fig. 6, the image completion apparatus based on model prior includes: a processor 31, a memory 32, a communication module 33, an input device 34, and an output device 35. The memory 32, which is a computer-readable storage medium, may be configured to store a software program, a computer-executable program, and modules, such as program instructions/modules corresponding to the model prior-based image completion method according to any embodiment of the present application (for example, an obtaining module and a completion module in the model prior-based image completion system). The communication module 33 is configured to perform data transmission. The processor 31 executes software programs, instructions and modules stored in the memory so as to execute various functional applications of the device and data processing, namely, implement the model prior-based image completion method. The input device 34 may be configured to receive input numeric or character information and to generate key signal inputs relating to user settings and function controls of the apparatus. The output device 35 may include a display device such as a display screen. The image completion device based on model prior provided above can be configured to execute the image completion method based on model prior provided above, and has corresponding functions and beneficial effects.
On the basis of the above embodiments, the present application further provides a computer-readable storage medium storing computer-executable instructions that, when executed by a computer processor, are configured to perform a model prior-based image completion method, and the storage medium may be any of various types of memory devices or storage devices. Of course, the computer-readable storage medium provided in this embodiment of the present application has computer-executable instructions that are not limited to the model prior-based image completion method described above, and may also perform related operations in the model prior-based image completion method provided in any embodiment of the present application.
On the basis of the foregoing embodiments, the present application further provides a computer program product, which is stored in a storage medium and includes several instructions to enable a computer device, a mobile terminal, or a processor therein to execute all or part of the steps of the model prior-based image completion method according to the embodiments of the present application.
Claims (10)
1. An image completion method based on model prior, comprising:
acquiring a missing image and specified completion area information of the missing image;
inputting the missing image and the specified completion region information into a pre-constructed image completion model, and outputting a completion image of the missing image, wherein the image completion model takes model parameters when a pre-training model converges as initial model parameters, and performs image completion training based on a training sample of the pre-training model and a prior sample generated by the pre-training model until the image completion model converges;
the pre-training model carries out image completion training in advance based on the training sample, and after the pre-training model converges, the prior sample is generated based on the pre-training model and comprises a noise adding image and a plurality of denoising images obtained by carrying out image completion corresponding to the noise adding image.
2. The model prior-based image completion method of claim 1, wherein the training procedure of the pre-trained model comprises:
based on the pre-training model, carrying out noise adding processing on the training sample to obtain a noise adding sample;
and performing image completion on the noise adding sample based on the pre-training model to obtain a de-noised sample, calculating a first model loss function according to the de-noised sample, and iteratively adjusting model parameters of the pre-training model based on the first model loss function until the pre-training model is converged.
3. The model prior-based image completion method according to claim 2, wherein the denoising the training samples based on the pre-training model to obtain denoised samples comprises:
adding Gaussian distribution noise to the training images of the training samples based on the pre-training model at each designated moment in a set time period to obtain Gaussian noise distribution probability of each training image, and taking the Gaussian noise distribution probability of each training image as the noise adding sample;
the image completion of the noise adding sample based on the pre-training model to obtain a noise removing sample comprises the following steps:
and sequentially predicting the real-time noise distribution probability of each appointed time in the set time period based on the Gaussian noise distribution probability of each training image and the set estimation parameters, and taking the prediction result of the first appointed time in the set time period as the denoising sample.
4. The model prior-based image completion method of claim 1, wherein the generating the prior samples based on the pre-trained model comprises:
inputting the noise images into the pre-training model, performing image completion of the noise images for a set number of times based on the pre-training model to obtain a corresponding number of the noise images, and taking the noise images and the corresponding noise images in the set number as the prior samples.
5. The model prior-based image completion method of claim 4, further comprising, after generating the prior samples based on the pre-trained model:
and calculating a completion processing score of each denoised image based on the denoised image and the corresponding denoised image, and performing data screening on the prior sample based on the completion processing score.
6. The model prior-based image completion method of claim 1, wherein the training procedure of the image completion model comprises:
taking the model parameter when the pre-training model converges as an initial model parameter, and inputting the training sample and the prior sample into the image completion model;
and performing image completion training by using the training sample and the prior sample based on the image completion model, calculating a second model loss function based on a training result, and iteratively adjusting the initial model parameters based on the second model loss function until the image completion model converges.
7. An image completion system based on model priors, comprising:
the acquisition module is configured to acquire a missing image and the specified completion area information of the missing image;
the completion module is configured to input the missing image and the specified completion region information into a pre-constructed image completion model and output a completion image of the missing image, the image completion model takes model parameters when a pre-training model converges as initial model parameters, and image completion training is carried out on the basis of a training sample of the pre-training model and a prior sample generated by the pre-training model until the image completion model converges; the pre-training model carries out image completion training in advance based on the training sample, and after the pre-training model converges, the prior sample is generated based on the pre-training model and comprises a noise adding image and a plurality of denoising images obtained by carrying out image completion corresponding to the noise adding image.
8. An image completion apparatus based on model prior, comprising:
a memory and one or more processors;
the memory configured to store one or more programs;
when executed by the one or more processors, cause the one or more processors to implement the model prior-based image completion method of any of claims 1-6.
9. A computer-readable storage medium having stored thereon computer-executable instructions that, when executed by a computer processor, are configured to perform the model prior-based image completion method of any of claims 1-6.
10. A computer program product comprising instructions which, when run on a computer or processor, cause the computer or processor to carry out the model prior based image inpainting method of any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211438318.9A CN115829865A (en) | 2022-11-16 | 2022-11-16 | Image completion method, system, device and storage medium based on model prior |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211438318.9A CN115829865A (en) | 2022-11-16 | 2022-11-16 | Image completion method, system, device and storage medium based on model prior |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115829865A true CN115829865A (en) | 2023-03-21 |
Family
ID=85528657
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211438318.9A Pending CN115829865A (en) | 2022-11-16 | 2022-11-16 | Image completion method, system, device and storage medium based on model prior |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115829865A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116228605A (en) * | 2023-05-09 | 2023-06-06 | 深圳大学 | Image complement method, device, computer equipment and storage medium |
-
2022
- 2022-11-16 CN CN202211438318.9A patent/CN115829865A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116228605A (en) * | 2023-05-09 | 2023-06-06 | 深圳大学 | Image complement method, device, computer equipment and storage medium |
| CN116228605B (en) * | 2023-05-09 | 2023-09-08 | 深圳大学 | Image complement method, device, computer equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109859147B (en) | Real image denoising method based on generation of antagonistic network noise modeling | |
| CN109658344B (en) | Image denoising method, device and equipment based on deep learning and storage medium | |
| CN108986050B (en) | Image and video enhancement method based on multi-branch convolutional neural network | |
| Dorta et al. | Structured uncertainty prediction networks | |
| CN109035189B (en) | Infrared and weak visible light image fusion method based on Cauchy fuzzy function | |
| CN107133923B (en) | Fuzzy image non-blind deblurring method based on adaptive gradient sparse model | |
| CN113238189B (en) | Sound source identification method and system based on array measurement and sparse prior information | |
| CN114341979A (en) | Method and apparatus for voice source separation based on convolutional neural network | |
| Ng et al. | Wavelet restoration of medical pulse-echo ultrasound images in an EM framework | |
| CN110728636A (en) | Monte Carlo rendering image denoising model, method and device based on generative confrontation network | |
| US20240062343A1 (en) | Image Restoration Method and Apparatus, Image Restoration Device and Storage Medium | |
| CN112912758A (en) | Method and system for adaptive beamforming of ultrasound signals | |
| Liu et al. | Learning hadamard-product-propagation for image dehazing and beyond | |
| CN113610989B (en) | Method and device for training style migration model and method and device for style migration | |
| CN115829865A (en) | Image completion method, system, device and storage medium based on model prior | |
| CN111724370A (en) | Multi-task non-reference image quality evaluation method and system based on uncertainty and probability | |
| CN113256508A (en) | Improved wavelet transform and convolution neural network image denoising method | |
| CN111968047A (en) | Adaptive optical image blind restoration method based on generating type countermeasure network | |
| CN112509144A (en) | Face image processing method and device, electronic equipment and storage medium | |
| CN110570375B (en) | Image processing method, device, electronic device and storage medium | |
| Wang et al. | A stacked convolutional sparse denoising autoencoder model for underwater heterogeneous information data | |
| CN115082336A (en) | SAR image speckle suppression method based on machine learning | |
| Park et al. | Flexible style image super-resolution using conditional objective | |
| CN113763268B (en) | Blind restoration method and system for face image | |
| Jeong et al. | Fast stereo matching using constraints in discrete space |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |