CN107133934A - Image completion method and device - Google Patents
Image completion method and device Download PDFInfo
- Publication number
- CN107133934A CN107133934A CN201710354183.0A CN201710354183A CN107133934A CN 107133934 A CN107133934 A CN 107133934A CN 201710354183 A CN201710354183 A CN 201710354183A CN 107133934 A CN107133934 A CN 107133934A
- Authority
- CN
- China
- Prior art keywords
- layer
- network
- generation
- image
- completion
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 76
- 230000006870 function Effects 0.000 claims description 67
- 230000006872 improvement Effects 0.000 claims description 56
- 230000004069 differentiation Effects 0.000 claims description 50
- 238000005070 sampling Methods 0.000 claims description 25
- 238000012549 training Methods 0.000 claims description 22
- 238000004590 computer program Methods 0.000 claims 1
- 230000008901 benefit Effects 0.000 description 10
- 238000010586 diagram Methods 0.000 description 8
- 230000008569 process Effects 0.000 description 7
- 238000012545 processing Methods 0.000 description 5
- 230000000694 effects Effects 0.000 description 4
- 239000011248 coating agent Substances 0.000 description 3
- 238000000576 coating method Methods 0.000 description 3
- 238000006073 displacement reaction Methods 0.000 description 3
- 238000003475 lamination Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000008439 repair process Effects 0.000 description 3
- 230000006978 adaptation Effects 0.000 description 2
- 238000013527 convolutional neural network Methods 0.000 description 2
- 125000004122 cyclic group Chemical group 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000001914 filtration Methods 0.000 description 2
- 239000011159 matrix material Substances 0.000 description 2
- 230000003321 amplification Effects 0.000 description 1
- 238000007796 conventional method Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 238000003199 nucleic acid amplification method Methods 0.000 description 1
- 230000003595 spectral effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/77—Retouching; Inpainting; Scratch removal
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20021—Dividing image into blocks, subimages or windows
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
Landscapes
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Image Analysis (AREA)
- Image Processing (AREA)
Abstract
The disclosure is directed to image completion method and device.This method includes：According to complex pattern to be repaired, n-layer gaussian pyramid is generated, the n is positive integer；By block matching method, the image of each layer in n-layer gaussian pyramid described in completion obtains each described layer after completion and truly schemed；Truly schemed according to the n-th layer image of the n-layer gaussian pyramid and each described layer, the generation network of generation confrontation network；According to the n-th layer image and the generation network, complex pattern to be repaired described in completion obtains the completion figure after completion.Gaussian pyramid, the true figure of each layer of gaussian pyramid and confrontation network that the technical scheme passes through complex pattern to be repaired, obtain generating network, so as to quickly generate complete display and high-resolution image, therefore, the resolution ratio of image after completion can be improved, the study of generation network can be accelerated again, more real image is obtained.
Description
Technical field
This disclosure relates to image processing field, more particularly to image completion method and device.
Background technology
With the development in epoch, the situation that image completion is applied is more and more, for example, loss in detail after image amplification
Situation, the situation of image section damage, but current scheme carries out completion, completion only by picture search, matching, filtering
Effect is not good.
The content of the invention
The embodiment of the present disclosure provides image completion method and device.The technical scheme is as follows：
According to the first aspect of the embodiment of the present disclosure there is provided a kind of image completion method, including：
According to complex pattern to be repaired, n-layer gaussian pyramid is generated, the n is positive integer；
By block matching method, the image of each layer in n-layer gaussian pyramid described in completion obtains described each after completion
Individual layer is truly schemed；
Truly schemed according to the n-th layer image of the n-layer gaussian pyramid and each described layer, the generation of generation confrontation network
Network；
According to the n-th layer image and the generation network, complex pattern to be repaired described in completion obtains the completion after completion
Figure.
The technical scheme provided by this disclosed embodiment can include the following benefits：Pass through the Gauss of complex pattern to be repaired
Each layer of pyramid, gaussian pyramid is truly schemed and confrontation network, obtains generating network, so as to quickly generate complete display
And high-resolution image, therefore, you can to improve the resolution ratio of image after completion, can accelerate to generate network again
Practise, obtain more real image.
In one embodiment, it is described according to the n-th layer image and the generation network, figure to be repaired described in completion
Picture, obtaining the completion figure after completion includes：
The i-th generation figure is obtained, the i is 0 to the integer between the n；
Described i-th generation figure and described i-th layer true figure are inputted into the generation network, the i-th -1 generation figure is obtained；
Wherein, the 0th generation figure is the completion figure；The n-th generation figure is from the n-th layer image, at random
Up-sampling is obtained and the n-th layer image resolution ratio identical image.
The technical scheme provided by this disclosed embodiment can include the following benefits：By way of circulation, increase
Image pixel, realization generation network completion image.
In one embodiment, the n-th layer image and each described layer according to the n-layer gaussian pyramid is true
Figure, the generation network of generation confrontation network includes：
Truly schemed by the n-th layer image and each described layer, train the confrontation network for meeting object function requirement
Differentiation network；
By the n-th layer image, each described layer truly figure and the differentiation network, train meet object function will
The generation network asked；
Wherein, the object function is used to weigh the loss during completion.
The technical scheme provided by this disclosed embodiment can include the following benefits：Differentiate network by training, from
And generation network is trained, so as to realize image completion.
In one embodiment, it is described truly to be schemed by the n-th layer image and each described layer, train and meet target
The differentiation network of the confrontation network of function requirements includes：
Obtain jth and update figure and the true figure of jth layer, the j is 1 integer arrived between the n；
The true figure of jth layer and the jth are updated figure input and improved and differentiates network, the first judged result is obtained；
When first judged result characterizes the jth renewal figure and the true figure of jth layer is same image, keep
It is described to improve the parameter for differentiating network；
When first judged result characterizes the jth renewal figure and the true figure of jth layer is not same image, lead to
Cross Stochastic gradient method and the object function updates the parameter improved and differentiate network；
The jth renewal figure input is initially generated network, the renewal figure of jth -1 is obtained；
Wherein, when the parameter for improving differentiation network no longer updates, described improve is differentiated that network is sentenced as described
Other network, the n-th renewal figure is that from the n-th layer image, random up-sampling is obtained and the n-th layer image resolution ratio identical
Image.
The technical scheme provided by this disclosed embodiment can include the following benefits：Introduce the training for differentiating network
Journey.
It is in one embodiment, described truly to be schemed by the n-th layer image, each described layer and the differentiation network,
Train and meet the generation network of object function requirement and include：
Obtain pth and improve figure and the true figure of pth layer, the p is described 1 integer arrived between the n；
The true figure of pth layer and the pth are improved into figure and input the differentiation network, the second judged result is obtained；
When second judged result characterizes the pth improvement figure and the true figure of pth layer is same image, keep
Improve the parameter of generation network；The pth is improved into described improve of figure input and generates network, the improvement figure of pth -1 is obtained；
When second judged result characterizes the pth improvement figure and the true figure of pth layer is not same image, root
The parameter for improving generation network is updated according to Stochastic gradient method and the object function；The pth is improved into figure input to update
Improvement generation network afterwards, obtains the improvement figure of pth -1；
Wherein, when the parameter for improving generation network no longer updates, it regard the generation network that improves as the life
Into network；N-th improvement figure is that from the n-th layer image, random up-sampling is obtained and the n-th layer image resolution ratio identical
Image.
The technical scheme provided by this disclosed embodiment can include the following benefits：After differentiating that network is determined,
Introduce the training process of generation network.
It is in one embodiment, described truly to be schemed by the n-th layer image, each described layer and the differentiation network,
Train and meet the generation network of object function requirement and include：
Obtain pth and improve figure and the true figure of pth layer, the p is described 1 integer arrived between the n；
The true figure of pth layer and the pth are improved into figure and input the differentiation network, the second judged result is obtained；
When second judged result characterizes the pth improvement figure and the true figure of pth layer is same image, keep
Improve the parameter of generation network；The pth is improved into described improve of figure input and generates network, the improvement figure of pth -1 is obtained；
When second judged result characterizes the pth improvement figure and the true figure of pth layer is not same image, root
The parameter for improving generation network is updated according to Stochastic gradient method and the object function；The pth is improved into figure input to update
Improvement generation network afterwards, obtains the improvement figure of pth -1；
Wherein, when the parameter for improving generation network no longer updates, it regard the generation network that improves as the life
Into network；N-th improvement figure is that from the n-th layer image, random up-sampling is obtained and the n-th layer image resolution ratio identical
Image.
The technical scheme provided by this disclosed embodiment can include the following benefits：Improved and generated by object function
Network and differentiation network.
In one embodiment, the object function is：
Wherein, the G is generation network, and the D is to differentiate network, the IqIt is q true pictures；E [] is to expect；Institute
State GqIt is q generation images；The q is the n to 0 integer, and a is default value.
The technical scheme provided by this disclosed embodiment can include the following benefits：Introduction can reach training effect most
Good object function.
According to the second aspect of the embodiment of the present disclosure there is provided a kind of image completion device, including：
First generation module, for according to complex pattern to be repaired, generating n-layer gaussian pyramid, the n is positive integer；
First completion module, for by Block- matching device, the image of each layer in n-layer gaussian pyramid described in completion,
Each described layer after completion is obtained truly to scheme；
Second generation module, truly schemes for the n-th layer image according to the n-layer gaussian pyramid and each described layer,
The generation network of generation confrontation network；
Second completion module, for according to the n-th layer image and the generation network, complex pattern to be repaired described in completion,
Obtain the completion figure after completion.
In one embodiment, the second completion module includes：
Acquisition submodule, for obtaining the i-th generation figure, the i is 0 to the integer between the n；
Input submodule, for the described i-th generation figure and described i-th layer true figure to be inputted into the generation network, is obtained
I-th -1 generation figure；
Wherein, the 0th generation figure is the completion figure；The n-th generation figure is from the n-th layer image, at random
Up-sampling is obtained and the n-th layer image resolution ratio identical image.
In one embodiment, second generation module includes：
First training submodule, for truly being schemed by the n-th layer image and each described layer, trains and meets target
The differentiation network of the confrontation network of function requirements；
Second training submodule, for truly being schemed and the differentiation network by the n-th layer image, each described layer,
Train the generation network for meeting object function requirement；
Wherein, the object function is used to weigh the loss during completion.
In one embodiment, the first training submodule includes：
First acquisition unit, for obtaining jth renewal figure and the true figure of jth layer, the j is 1 to arrive whole between the n
Number；
First input block, differentiates network for updating figure input and improving the true figure of jth layer and the jth, obtains
To the first judged result；
First holding unit, for updating figure and the true figure of jth layer when first judged result characterizes the jth
When being same image, the parameter improved and differentiate network is kept；
First updating block, for updating figure and the true figure of jth layer when first judged result characterizes the jth
When being not same image, the parameter for improving differentiation network is updated by Stochastic gradient method and the object function；
Second input block, is initially generated network by the jth renewal figure input for inputting, obtains the renewal figure of jth -1；
Wherein, when the parameter for improving differentiation network no longer updates, described improve is differentiated that network is sentenced as described
Other network, the n-th renewal figure is that from the n-th layer image, random up-sampling is obtained and the n-th layer image resolution ratio identical
Image.
In one embodiment, the second training submodule includes：
Second acquisition unit, for obtaining pth improvement figure and the true figure of pth layer, the p be described 1 to the n it
Between integer；
3rd input block, inputs the differentiation network for the true figure of pth layer and the pth to be improved into figure, obtains
To the second judged result；
Second holding unit, for improving figure and the true figure of pth layer when second judged result characterizes the pth
When being same image, keep improving the parameter of generation network；
4th input block, for the pth to be improved, figure input is described to improve generation network, obtains the improvement figure of pth -1；
Second updating block, for improving figure and the true figure of pth layer when second judged result characterizes the pth
When being not same image, the parameter for improving generation network is updated according to Stochastic gradient method and the object function；
The improvement that 4th input block is additionally operable to input the pth improvement figure after updating generates network, obtains the
P-1 improves figure；
Wherein, when the parameter for improving generation network no longer updates, it regard the generation network that improves as the life
Into network；N-th improvement figure is that from the n-th layer image, random up-sampling is obtained and the n-th layer image resolution ratio identical
Image.
In one embodiment, the object function is：
Wherein, the G is generation network, and the D is to differentiate network, the IqIt is q true pictures；E [] is to expect；Institute
State GqIt is q generation images；The q is the n to 0 integer, and a is default value.
According to the second aspect of the embodiment of the present disclosure there is provided image completion device, including：
Processor；
Memory for storing processor-executable instruction；
Wherein, the processor is configured as：
According to complex pattern to be repaired, n-layer gaussian pyramid is generated, the n is positive integer；
By block matching method, the image of each layer in n-layer gaussian pyramid described in completion obtains described each after completion
Individual layer is truly schemed；
Truly schemed according to the n-th layer image of the n-layer gaussian pyramid and each described layer, the generation of generation confrontation network
Network；
According to the n-th layer image and the generation network, complex pattern to be repaired described in completion obtains the completion after completion
Figure.It should be appreciated that the general description of the above and detailed description hereinafter are only exemplary and explanatory, it can not limit
The disclosure.
Brief description of the drawings
Accompanying drawing herein is merged in specification and constitutes the part of this specification, shows the implementation for meeting the disclosure
Example, and be used to together with specification to explain the principle of the disclosure.
Fig. 1 is the flow chart of the image completion method according to an exemplary embodiment.
Fig. 2 is the flow chart of the image completion method according to an exemplary embodiment.
Fig. 3 is the flow chart of the image completion method according to an exemplary embodiment.
Fig. 4 is the block diagram of the image completion device according to an exemplary embodiment.
Fig. 5 is the block diagram of the image completion device according to an exemplary embodiment.
Fig. 6 is the block diagram of the image completion device according to an exemplary embodiment.
Fig. 7 is the block diagram of the image completion device according to an exemplary embodiment.
Fig. 8 is the block diagram of the image completion device according to an exemplary embodiment.
Fig. 9 is the block diagram of the image completion device according to an exemplary embodiment.
Embodiment
Here exemplary embodiment will be illustrated in detail, its example is illustrated in the accompanying drawings.Following description is related to
During accompanying drawing, unless otherwise indicated, the same numbers in different accompanying drawings represent same or analogous key element.Following exemplary embodiment
Described in embodiment do not represent all embodiments consistent with the disclosure.On the contrary, they be only with it is such as appended
The example of the consistent apparatus and method of some aspects be described in detail in claims, the disclosure.
In correlation technique, in picture completion technology, most common method is Block- matching (Patchmatch) method, block
Basic thought with method is that each two field picture is divided into a series of sub-blocks) calculate in present frame in each sub-block and consecutive frame
The error function of each sub-block, using the corresponding sub-block of the consecutive frame with minimal error as current block prediction block, and two pieces
Relative displacement be defined as displacement vector.
But, patchmatch needs the space of search, it is meant that when needing, the cavity mended is too big, and can not find search
Region when, the effect of completion is undesirable；When textures repair cavity, it is considered to the illumination in the residing region in cavity, Texture eigenvalue
Source block (patch) is different from, therefore, in order that the image after completion does not compare lofty edge, there is more preferable display effect,
Filtering process has been carried out, the image after completion has been directly resulted in the presence of fuzzy.
Embodiment one
Fig. 1 is a kind of flow chart of image completion method according to an exemplary embodiment, as shown in figure 1, image
Complementing method is used in image completion device, and the device can apply in terminal, server, and this method comprises the following steps
101-104：
In a step 101, according to complex pattern to be repaired, n-layer gaussian pyramid is generated.
Here, n is positive integer.
Obtain the corresponding mask of complex pattern to be repaired (mask) image, it would be desirable to repair part and be labeled as 0, need not repair
Part is labeled as 255, the gaussian pyramid of complex pattern to be repaired and correspondence mask images is constructed, untill being not present 0 in mask.
Here, n is positive integer.
In a step 102, by block matching method, the image of each layer, is obtained after completion in completion n-layer gaussian pyramid
Each layer truly scheme.
From the n-th layer start to process of gaussian pyramid, each patch of random initializtion compensation (offset) is somebody's turn to do
The compensation figure (offset map) of layer；By gaussian pyramid successively down, offset map are up-sampled, the place of blank do with
Machine is initialized, and the matching value most matched is found for the place of blank by nearest neighbor search；When the arrival gaussian pyramid bottom
When, the corresponding displacement information matrix between all patch has been obtained, the information of correspondence position has been obtained by transposed matrix, then
The information of position to be repaired is generated by way of textures, is truly schemed so as to obtain each layer.
Here, the method for the true figure of each layer of completion is not limited to BMA, and current matching process can.
In step 103, truly schemed according to the n-th layer image of n-layer gaussian pyramid and each layer, generation confrontation network
Generate network.
Confrontation network (Generative Adversarial Networks, GANS) is a kind of generation network, its behind base
This thought is that many training samples are obtained from training hardship, the probability distribution of learning training case generation, so that implementation method is just
It is to allow two network models to compete with one another for, one of them is called generation network, he constantly catches the probability of the training true picture in storehouse
Distribution, the random noise of input is changed into new sample；Another is called differentiation network, and it can observe true picture simultaneously
And new samples, whether judge new samples is genuine.
At step 104, according to n-th layer image and generation network, completion complex pattern to be repaired obtains the completion after completion
Figure.
In the present embodiment, pass through the gaussian pyramid of complex pattern to be repaired, the true figure of each layer of gaussian pyramid and confrontation net
Network, obtains generating network, so as to quickly generate complete display and high-resolution image, therefore, you can to improve benefit
The resolution ratio of image after complete, can accelerate to generate the study of network, obtain more real image again.
In one embodiment, step 103 can include：
The i-th generation figure is obtained, the i is 0 to the integer between the n；
I-th generation figure and i-th layer of true figure are inputted into the generation network, the i-th -1 generation figure is obtained；
Wherein, the 0th generation figure is the completion figure；The n-th generation figure is from the n-th layer image, at random
Up-sampling is obtained and the n-th layer image resolution ratio identical image.
What deserves to be explained is, truly figure both can be that completion in advance is good or in above-mentioned cyclic process to each layer
Any layer is needed, which layer of completion is obtained.
Here, the n-th generation figure is obtained in the following manner.The present embodiment can judge i-th by object function
Whether layer is true schemes with the i-th generation figure to be same image.
Predetermined number (such as 100) individual pixel is randomly selected from n-th layer image, replaces random noise to make these pixels
For the input vector of convolutional neural networks, then by full articulamentum, the characteristic pattern that vector is changed into enters back into warp lamination,
Characteristic image port number is halved by each warp lamination, that is, characteristic pattern resolution ratio is up-sampled, finally give with
N-th layer image resolution ratio identical n-th generates image.The default value of full articulamentum can be 8192；The pixel of characteristic image can
To be 4*4*512.Convolutional neural networks are pre-set.
In one embodiment, step 102 can include：
Truly schemed by n-th layer image and each layer, train the differentiation net for the confrontation network for meeting object function requirement
Network；Truly scheme and differentiate network by n-th layer image, each layer, train the generation network for meeting object function requirement；
Wherein, object function is used to weigh the loss during completion.
In one embodiment, it is described truly to be schemed by the n-th layer image and each described layer, train and meet target
The differentiation network of the confrontation network of function requirements includes：
Obtain jth and update figure and the true figure of jth layer, the j is 1 integer arrived between the n；By the true figure of jth layer and
Jth, which updates figure input and improved, differentiates network, obtains the first judged result；Figure and jth layer are updated when the first judged result characterizes jth
When true figure is same image, keep improving the parameter for differentiating network；Figure and jth layer are updated when the first judged result characterizes jth
When true figure is not same image, the parameter of network is differentiated by Stochastic gradient method and object function retrofit；Jth is updated
Figure input is initially generated network, obtains the renewal figure of jth -1；
Wherein, when the parameter for improving differentiation network no longer updates, it will improve and differentiate network as network is differentiated, n-th more
New figure is that from n-th layer image, random up-sampling is obtained and n-th layer image resolution ratio identical image.
Here, the n-th renewal figure is identical with the acquisition methods of the n-th generation figure, because the pixel obtained at random is different, therefore,
Obtained image is also different.
Here, above-mentioned is a cyclic process, differentiates that the parameter in network has been stablized if improved, no longer updates, that
, improve and differentiate network just as differentiation network.The improvement of initialization, which differentiates network and is initially generated network, to be all randomly generated
, what its parameter was also randomly generated, therefore, the network that is initially generated in said process is indeclinable.
In the present embodiment, the figure for inputting to be made up of the i-th generation figure and i-th layer of true figure is (due to generation figure and true figure
All it is 3 passages, therefore, the figure of composition is 6 passages), wherein differentiating that network includes y layers of convolutional layer, each convolutional layer heel
Improve linear unit (Rectified Linear Units, ReLU) active coating and Marx collects (MaxPooling) layer, most
Differentiate the i-th generation figure with whether being unified image for i-th layer of true figure eventually.Y can be positive integer, here, and y is 6.
In the present embodiment, jth is generated to the improvement after image input updates and generates network, the generation image bag of jth -1 is obtained
Include：
Jth is generated to image as the input of generation network, by w convolutional layer, a vector, every layer of volume base is parsed into
Layer one ReLU active coating of heel；The subsequent vector by this after parsing is by w+1 layers of warp lamination, and up-sampling turns into 3 passages point
Resolution is the generation image of jth -1 of (2*heighti-1,2*widthi-1), one ReLU active coating of every layer of convolutional layer heel；w
Can be positive integer, here, w is 5.
In one embodiment, it is described truly to scheme and differentiate network by n-th layer image, each layer, train and meet mesh
The generation network of scalar functions requirement includes：
Obtain pth and improve figure and the true figure of pth layer, p is described 1 to the integer between n；The true figure of pth layer and pth are changed
Enter figure input and differentiate network, obtain the second judged result；It is when the second judged result characterizes pth improvement figure and the true figure of pth layer
During same image, keep improving the parameter of generation network；Pth is improved into figure input and improves generation network, the improvement of pth -1 is obtained
Figure；When the second judged result characterizes pth improvement figure and the true figure of pth layer is not same image, according to Stochastic gradient method and mesh
Scalar functions retrofit generates the parameter of network；Pth is improved to the improvement after figure input updates and generates network, pth -1 is obtained and changes
Enter figure；
Wherein, when the parameter for improving generation network no longer updates, generation network will be improved as generation network；N-th changes
It is that from n-th layer image, random up-sampling is obtained and n-th layer image resolution ratio identical image to enter figure.Here, n-th updates figure
Identical with the acquisition methods that n-th improves figure, because the pixel obtained at random is different, therefore, obtained image is also different.
In one embodiment, the object function is：
Wherein, G is generation network, and D is to differentiate network, IqIt is q true pictures；E [] is to expect；GqIt is q generation figures
Picture；Q is n to 0 integer, and a is integer, can be 10, e etc..
In the present embodiment, above-mentioned object function is made up of upper three expectations, first Expectation-based Representation for Concepts q true picture it is flat
Average, second Expectation-based Representation for Concepts q generation spectral discrimination is the expectation of true picture；3rd Expectation-based Representation for Concepts q generates image
Expect with the distance between q true pictures.
Embodiment two
Fig. 2 is a kind of flow chart of image completion method according to an exemplary embodiment, as shown in Fig. 2 image
Complementing method is used in image completion device, and the device can apply in terminal, server, and this method comprises the following steps
201-205：
In step 201, according to complex pattern to be repaired, n-layer gaussian pyramid is generated.
Here, n is positive integer.
In step 202., by block matching method, the image of each layer, is obtained after completion in completion n-layer gaussian pyramid
Each layer truly scheme.
In step 203, truly schemed by n-th layer image and each layer, train the confrontation for meeting object function requirement
The differentiation network of network.
In step 204, by n-th layer image, each layer truly figure and differentiate network, train meet object function will
The generation network asked.
In step 205, according to n-th layer image and generation network, completion complex pattern to be repaired obtains the completion after completion
Figure.
The confrontation network that the present embodiment is provided, can increase pixel, the pixel meets the picture of true picture, therefore, no
Only the resolution ratio of completion image can also be improved with completion image.
Embodiment three
Fig. 3 is a kind of flow chart of image completion method according to an exemplary embodiment, as shown in figure 3, image
Complementing method is used in image completion device, and the device can apply in terminal, server, and this method comprises the following steps
301-320：
In step 301, according to complex pattern to be repaired, n-layer gaussian pyramid is generated.
In step 302, from the n-th layer image of n-layer gaussian pyramid, random up-sampling is obtained and n-th layer image point
The generation figure of resolution identical n-th.
In step 303, jth generation figure is obtained.
In step 304, using block matching method, the jth tomographic image of completion n-layer gaussian pyramid obtains jth layer true
Figure.
In step 305, the true figure of jth layer and jth generation figure input are improved and differentiates network, obtain the first judgement knot
Really.
Within step 306, it is same image to judge whether the first judged result characterizes jth generation figure and the true figure of jth layer.
If so, then performing step 307；If it is not, then performing step 308.
In step 307, keep improving the parameter for differentiating network.Perform step 310.
In step 308, the parameter of network is differentiated by Stochastic gradient method and object function retrofit.
In a step 309, jth generation figure input is initially generated network, obtains the generation image of jth -1, perform step
303。
In the step 310, judge to improve and differentiate whether network no longer updates.If so, then performing step 311；If it is not, then holding
Row step 309.
In step 311, it will improve and differentiate network as differentiation network.
In step 312, obtain pth and improve figure.
In step 313, the true figure of pth layer and pth are improved into figure input and differentiates network, obtain the second judged result.
In a step 314, it is same figure to judge whether the second judged result characterizes pth improvement figure and pth layer true picture
Picture.If so, then performing step 315；If it is not, then performing step 317.
In step 315, keep improving the parameter of generation network.
In step 316, pth is improved into figure input and improves generation network, obtain the improvement figure of pth -1.Perform step 312.
What the parameter of initial improvement generation network was randomly generated.
In step 317, the parameter of network is generated according to Stochastic gradient method and object function retrofit.
In step 318, pth is improved to the improvement after figure input updates and generates network, obtain the improvement figure of pth -1.Perform
Step 312.
In step 319, judge to improve whether generation network no longer updates.If so, then performing step 320；If it is not, then holding
Row step 312.
In step 320, generation network will be improved as generation network, pth is improved into figure input and improves generation network, is obtained
Improve and scheme to pth -1.Perform step 312.
In the present embodiment, by the gaussian pyramid and confrontation network of complex pattern to be repaired, obtain generating network, so as to
Complete display and high-resolution image are quickly generated, therefore, you can to improve the resolution ratio of image after completion, can added again
The study of fast generation network, obtains more real image.
Following is disclosure device embodiment, can be used for performing method of disclosure embodiment.
Example IV
Fig. 4 is a kind of block diagram of image completion device according to an exemplary embodiment, and the device can be by soft
Being implemented in combination with for part, hardware or both is some or all of as electronic equipment.As shown in figure 4, the image completion device
Including：
First generation module 401, for according to complex pattern to be repaired, generating n-layer gaussian pyramid, the n is positive integer；
First completion module 402, for by Block- matching device, the figure of each layer in n-layer gaussian pyramid described in completion
Picture, obtains each described layer after completion and truly schemes；
Second generation module 403, it is true for the n-th layer image according to the n-layer gaussian pyramid and each described layer
Figure, the generation network of generation confrontation network；
Second completion module 404, for according to the n-th layer image and the generation network, figure to be repaired described in completion
Picture, obtains the completion figure after completion.
In one embodiment, as shown in figure 5, the second completion module 404 includes：
Acquisition submodule 4041, for obtaining the i-th generation figure, the i is 0 to the integer between the n；
Input submodule 4042, for the described i-th generation figure and described i-th layer true figure to be inputted into the generation network,
Obtain the i-th -1 generation figure；
Wherein, the 0th generation figure is the completion figure；The n-th generation figure is from the n-th layer image, at random
Up-sampling is obtained and the n-th layer image resolution ratio identical image.
In one embodiment, as shown in fig. 6, second generation module 403 includes：
First training submodule 4031, for truly scheming by the n-th layer image and each described layer, trains satisfaction
The differentiation network of the confrontation network of object function requirement；
Second training submodule 4032, for truly being schemed and the differentiation net by the n-th layer image, each described layer
Network, trains the generation network for meeting object function requirement；
Wherein, the object function is used to weigh the loss during completion.
In one embodiment, as shown in fig. 7, the first training submodule 4031 includes：
First acquisition unit 40311, for obtaining jth renewal figure and the true figure of jth layer, the j is 1 to arrive between the n
Integer；
First input block 40312, net is differentiated for updating figure input and improving the true figure of jth layer and the jth
Network, obtains the first judged result；
First holding unit 40313, for updating figure and jth layer when first judged result characterizes the jth
When true figure is same image, the parameter improved and differentiate network is kept；
First updating block 40314, for updating figure and jth layer when first judged result characterizes the jth
When true figure is not same image, the parameter for improving differentiation network is updated by Stochastic gradient method and the object function；
Second input block 40315, is initially generated network by the jth renewal figure input for inputting, obtains jth -1 more
New figure；
Wherein, when the parameter for improving differentiation network no longer updates, described improve is differentiated that network is sentenced as described
Other network, the n-th renewal figure is that from the n-th layer image, random up-sampling is obtained and the n-th layer image resolution ratio identical
Image.
In one embodiment, as shown in figure 8, the second training submodule 4032 includes：
Second acquisition unit 40321, for obtaining pth improvement figure and the true figure of pth layer, the p is described 1 to institute
State the integer between n；
3rd input block 40322, the differentiation net is inputted for the true figure of pth layer and the pth to be improved into figure
Network, obtains the second judged result；
Second holding unit 40323, for improving figure and pth layer when second judged result characterizes the pth
When true figure is same image, keep improving the parameter of generation network；
4th input block 40324, for the pth to be improved, figure input is described to improve generation network, obtains pth -1 and changes
Enter figure；
Second updating block 40325, for improving figure and pth layer when second judged result characterizes the pth
When true figure is not same image, the parameter for improving generation network is updated according to Stochastic gradient method and the object function；
The improvement that 4th input block 40324 is additionally operable to input the pth improvement figure after updating generates network, obtains
Improve and scheme to pth -1；
Wherein, when the parameter for improving generation network no longer updates, it regard the generation network that improves as the life
Into network；N-th improvement figure is that from the n-th layer image, random up-sampling is obtained and the n-th layer image resolution ratio identical
Image.
In one embodiment, the object function is：
Wherein, the G is generation network, and the D is to differentiate network, the IqIt is q true pictures；E [] is to expect；Institute
State GqIt is q generation images；The q is the n to 0 integer, and a is default value.
According to the fourth aspect of the embodiment of the present disclosure there is provided a kind of image completion device, including：
Processor；
Memory for storing processor-executable instruction；
Wherein, the processor is configured as：
According to complex pattern to be repaired, corresponding n-layer gaussian pyramid is generated, the n is positive integer；
According to the n-layer gaussian pyramid, the generation network of generation confrontation network；
According to the generation network, complex pattern to be repaired described in completion obtains high-resolution completion image.
According to the third aspect of the embodiment of the present disclosure there is provided a kind of image completion device, including：
Processor；
Memory for storing processor-executable instruction；
Wherein, processor is configured as：
According to complex pattern to be repaired, n-layer gaussian pyramid is generated, the n is positive integer；
By block matching method, the image of each layer in n-layer gaussian pyramid described in completion obtains described each after completion
Individual layer is truly schemed；
Truly schemed according to the n-th layer image of the n-layer gaussian pyramid and each described layer, the generation of generation confrontation network
Network；
According to the n-th layer image and the generation network, complex pattern to be repaired described in completion obtains the completion after completion
Figure.Above-mentioned processor is also configured to：
It is described according to the n-th layer image and the generation network, complex pattern to be repaired described in completion obtains the benefit after completion
Full figure includes：
The i-th generation figure is obtained, the i is 0 to the integer between the n；
Described i-th generation figure and described i-th layer true figure are inputted into the generation network, the i-th -1 generation figure is obtained；
Wherein, the 0th generation figure is the completion figure；The n-th generation figure is from the n-th layer image, at random
Up-sampling is obtained and the n-th layer image resolution ratio identical image.
The n-th layer image and each described layer according to the n-layer gaussian pyramid is truly schemed, generation confrontation network
Generation network includes：
Truly schemed by the n-th layer image and each described layer, train the confrontation network for meeting object function requirement
Differentiation network；
By the n-th layer image, each described layer truly figure and the differentiation network, train meet object function will
The generation network asked；
Wherein, the object function is used to weigh the loss during completion.
It is described truly to be schemed by the n-th layer image and each described layer, train the confrontation for meeting object function requirement
The differentiation network of network includes：
Obtain jth and update figure and the true figure of jth layer, the j is 1 integer arrived between the n；
The true figure of jth layer and the jth are updated figure input and improved and differentiates network, the first judged result is obtained；
When first judged result characterizes the jth renewal figure and the true figure of jth layer is same image, keep
It is described to improve the parameter for differentiating network；
When first judged result characterizes the jth renewal figure and the true figure of jth layer is not same image, lead to
Cross Stochastic gradient method and the object function updates the parameter improved and differentiate network；
The jth renewal figure input is initially generated network, the renewal figure of jth -1 is obtained；
Wherein, when the parameter for improving differentiation network no longer updates, described improve is differentiated that network is sentenced as described
Other network, the n-th renewal figure is that from the n-th layer image, random up-sampling is obtained and the n-th layer image resolution ratio identical
Image.
It is described truly to be schemed by the n-th layer image, each described layer and the differentiation network, train and meet target letter
The generation network that number is required includes：
Obtain pth and improve figure and the true figure of pth layer, the p is described 1 integer arrived between the n；
The true figure of pth layer and the pth are improved into figure and input the differentiation network, the second judged result is obtained；
When second judged result characterizes the pth improvement figure and the true figure of pth layer is same image, keep
Improve the parameter of generation network；The pth is improved into described improve of figure input and generates network, the improvement figure of pth -1 is obtained；
When second judged result characterizes the pth improvement figure and the true figure of pth layer is not same image, root
The parameter for improving generation network is updated according to Stochastic gradient method and the object function；The pth is improved into figure input to update
Improvement generation network afterwards, obtains the improvement figure of pth -1；
Wherein, when the parameter for improving generation network no longer updates, it regard the generation network that improves as the life
Into network；N-th improvement figure is that from the n-th layer image, random up-sampling is obtained and the n-th layer image resolution ratio identical
Image.
The object function is：
Wherein, the G is generation network, and the D is to differentiate network, the IqIt is q true pictures；E [] is to expect；Institute
State GqIt is q generation images；The q is the n to 0 integer, and a is default value.
On the device in above-described embodiment, wherein modules perform the concrete mode of operation in relevant this method
Embodiment in be described in detail, explanation will be not set forth in detail herein.
Fig. 9 is a kind of block diagram for image completion device according to an exemplary embodiment.For example, device 1900
It may be provided in a server.Device 1900 includes processing assembly 1922, and it further comprises one or more processors, with
And as the memory resource representated by memory 1932, for store can by the execution of processing assembly 1922 instruction, for example should
Use program.The application program stored in memory 1932 can include it is one or more each correspond to one group of instruction
Module.In addition, processing assembly 1922 is configured as execute instruction, to perform the above method.
Device 1900 can also include the power management that a power supply module 1926 is configured as performs device 1900, one
Wired or wireless network interface 1950 is configured as device 1900 being connected to network, and input and output (I/O) interface
1958.Device 1900 can be operated based on the operating system for being stored in memory 1932, such as Windows ServerTM, Mac
OS XTM, UnixTM, LinuxTM, FreeBSDTM or similar.
A kind of non-transitorycomputer readable storage medium, when the instruction in the storage medium is by the processing of device 1900
When device is performed so that device 1900 is able to carry out above-mentioned image completion method, and methods described includes：
According to complex pattern to be repaired, n-layer gaussian pyramid is generated, the n is positive integer；
By block matching method, the image of each layer in n-layer gaussian pyramid described in completion obtains described each after completion
Individual layer is truly schemed；
Truly schemed according to the n-th layer image of the n-layer gaussian pyramid and each described layer, the generation of generation confrontation network
Network；
According to the n-th layer image and the generation network, complex pattern to be repaired described in completion obtains the completion after completion
Figure.
It is described according to the n-th layer image and the generation network, complex pattern to be repaired described in completion obtains the benefit after completion
Full figure includes：
The i-th generation figure is obtained, the i is 0 to the integer between the n；
Described i-th generation figure and described i-th layer true figure are inputted into the generation network, the i-th -1 generation figure is obtained；
Wherein, the 0th generation figure is the completion figure；The n-th generation figure is from the n-th layer image, at random
Up-sampling is obtained and the n-th layer image resolution ratio identical image.
The n-th layer image and each described layer according to the n-layer gaussian pyramid is truly schemed, generation confrontation network
Generation network includes：
Truly schemed by the n-th layer image and each described layer, train the confrontation network for meeting object function requirement
Differentiation network；
By the n-th layer image, each described layer truly figure and the differentiation network, train meet object function will
The generation network asked；
Wherein, the object function is used to weigh the loss during completion.
It is described truly to be schemed by the n-th layer image and each described layer, train the confrontation for meeting object function requirement
The differentiation network of network includes：
Obtain jth and update figure and the true figure of jth layer, the j is 1 integer arrived between the n；
The true figure of jth layer and the jth are updated figure input and improved and differentiates network, the first judged result is obtained；
When first judged result characterizes the jth renewal figure and the true figure of jth layer is same image, keep
It is described to improve the parameter for differentiating network；
When first judged result characterizes the jth renewal figure and the true figure of jth layer is not same image, lead to
Cross Stochastic gradient method and the object function updates the parameter improved and differentiate network；
The jth renewal figure input is initially generated network, the renewal figure of jth -1 is obtained；
Wherein, when the parameter for improving differentiation network no longer updates, described improve is differentiated that network is sentenced as described
Other network, the n-th renewal figure is that from the n-th layer image, random up-sampling is obtained and the n-th layer image resolution ratio identical
Image.
It is described truly to be schemed by the n-th layer image, each described layer and the differentiation network, train and meet target letter
The generation network that number is required includes：
Obtain pth and improve figure and the true figure of pth layer, the p is described 1 integer arrived between the n；
The true figure of pth layer and the pth are improved into figure and input the differentiation network, the second judged result is obtained；
When second judged result characterizes the pth improvement figure and the true figure of pth layer is same image, keep
Improve the parameter of generation network；The pth is improved into described improve of figure input and generates network, the improvement figure of pth -1 is obtained；
When second judged result characterizes the pth improvement figure and the true figure of pth layer is not same image, root
The parameter for improving generation network is updated according to Stochastic gradient method and the object function；The pth is improved into figure input to update
Improvement generation network afterwards, obtains the improvement figure of pth -1；
Wherein, when the parameter for improving generation network no longer updates, it regard the generation network that improves as the life
Into network；N-th improvement figure is that from the n-th layer image, random up-sampling is obtained and the n-th layer image resolution ratio identical
Image.
The object function is：
Wherein, the G is generation network, and the D is to differentiate network, the IqIt is q true pictures；E [] is to expect；Institute
State GqIt is q generation images；The q is the n to 0 integer, and a is default value.
Those skilled in the art will readily occur to its of the disclosure after considering specification and putting into practice disclosure disclosed herein
Its embodiment.The application is intended to any modification, purposes or the adaptations of the disclosure, these modifications, purposes or
Person's adaptations follow the general principle of the disclosure and including the undocumented common knowledge in the art of the disclosure
Or conventional techniques.Description and embodiments are considered only as exemplary, and the true scope of the disclosure and spirit are by following
Claim is pointed out.
It should be appreciated that the precision architecture that the disclosure is not limited to be described above and is shown in the drawings, and
And various modifications and changes can be being carried out without departing from the scope.The scope of the present disclosure is only limited by appended claim.
Claims (14)
1. a kind of image completion method, it is characterised in that including：
According to complex pattern to be repaired, n-layer gaussian pyramid is generated, the n is positive integer；
By block matching method, the image of each layer in n-layer gaussian pyramid described in completion obtains each described layer after completion
True figure；
Truly schemed according to the n-th layer image of the n-layer gaussian pyramid and each described layer, the generation net of generation confrontation network
Network；
According to the n-th layer image and the generation network, complex pattern to be repaired described in completion obtains the completion figure after completion.
2. according to the method described in claim 1, it is characterised in that described according to the n-th layer image and the generation network,
Complex pattern to be repaired described in completion, obtaining the completion figure after completion includes：
The i-th generation figure is obtained, the i is 0 to the integer between the n；
Described i-th generation figure and described i-th layer true figure are inputted into the generation network, the i-th -1 generation figure is obtained；
Wherein, the 0th generation figure is the completion figure；The n-th generation figure be from the n-th layer image, it is random on adopt
Sample is obtained and the n-th layer image resolution ratio identical image.
3. according to the method described in claim 1, it is characterised in that the n-th layer image according to the n-layer gaussian pyramid
Truly scheme with each described layer, the generation network of generation confrontation network includes：
Truly schemed by the n-th layer image and each described layer, train sentencing for the confrontation network that meets object function requirement
Other network；
Truly schemed by the n-th layer image, each described layer and the differentiation network, train and meet object function requirement
The generation network；
Wherein, the object function is used to weigh the loss during completion.
4. method according to claim 3, it is characterised in that described true by the n-th layer image and each described layer
Real figure, training the differentiation network for the confrontation network for meeting object function requirement includes：
Obtain jth and update figure and the true figure of jth layer, the j is 1 integer arrived between the n；
The true figure of jth layer and the jth are updated figure input and improved and differentiates network, the first judged result is obtained；
When first judged result characterizes the jth renewal figure and the true figure of jth layer is same image, keep described
Improve the parameter for differentiating network；
When first judged result characterizes the jth renewal figure and the true figure of jth layer is not same image, by with
Machine gradient method and the object function update the parameter improved and differentiate network；
The jth renewal figure input is initially generated network, the renewal figure of jth -1 is obtained；
Wherein, when the parameter for improving differentiation network no longer updates, described improve is differentiated that network is used as the differentiation net
Network, the n-th renewal figure is that from the n-th layer image, random up-sampling is obtained and the n-th layer image resolution ratio identical figure
Picture.
5. method according to claim 3, it is characterised in that described true by the n-th layer image, each described layer
Figure and the differentiation network, train and meet the generation network of object function requirement and include：
Obtain pth and improve figure and the true figure of pth layer, the p is described 1 integer arrived between the n；
The true figure of pth layer and the pth are improved into figure and input the differentiation network, the second judged result is obtained；
When second judged result characterizes the pth improvement figure and the true figure of pth layer is same image, keep improving
Generate the parameter of network；The pth is improved into described improve of figure input and generates network, the improvement figure of pth -1 is obtained；
When second judged result characterizes the pth improvement figure and the true figure of pth layer is not same image, according to
Machine gradient method and the object function update the parameter for improving generation network；The pth is improved after figure input renewal
Generation network is improved, the improvement figure of pth -1 is obtained；
Wherein, when the parameter for improving generation network no longer updates, it regard the generation network that improves as the generation net
Network；N-th improvement figure is that from the n-th layer image, random up-sampling is obtained and the n-th layer image resolution ratio identical figure
Picture.
6. method according to claim 3, it is characterised in that the object function is：
<mrow>
<mi>G</mi>
<mo>=</mo>
<mi>arg</mi>
<munder>
<mrow>
<mi>m</mi>
<mi>i</mi>
<mi>n</mi>
</mrow>
<mi>G</mi>
</munder>
<munder>
<mrow>
<mi>m</mi>
<mi>a</mi>
<mi>x</mi>
</mrow>
<mi>D</mi>
</munder>
<mrow>
<mo>(</mo>
<mi>E</mi>
<mo>&lsqb;</mo>
<msub>
<mi>log</mi>
<mi>a</mi>
</msub>
<mi>D</mi>
<mo>(</mo>
<msub>
<mi>I</mi>
<mi>q</mi>
</msub>
<mo>)</mo>
<mo>&rsqb;</mo>
<mo>+</mo>
<mi>E</mi>
<mo>&lsqb;</mo>
<mn>1</mn>
<mo>-</mo>
<mi>D</mi>
<mo>(</mo>
<msub>
<mi>G</mi>
<mi>q</mi>
</msub>
<mo>)</mo>
<mo>&rsqb;</mo>
<mo>+</mo>
<mi>E</mi>
<mo>&lsqb;</mo>
<mo>|</mo>
<mo>|</mo>
<msub>
<mi>I</mi>
<mi>q</mi>
</msub>
<mo>-</mo>
<msub>
<mi>G</mi>
<mi>q</mi>
</msub>
<mo>|</mo>
<mo>|</mo>
<mo>&rsqb;</mo>
<mo>)</mo>
</mrow>
<mo>;</mo>
</mrow>
Wherein, the G is generation network, and the D is to differentiate network, the IqIt is q true pictures；E [] is to expect；The Gq
It is q generation images；The q is the n to 0 integer, and a is default value.
7. a kind of image completion device, it is characterised in that including：
First generation module, for according to complex pattern to be repaired, generating n-layer gaussian pyramid, the n is positive integer；
First completion module, for by Block- matching device, the image of each layer, to be obtained in n-layer gaussian pyramid described in completion
Each described layer after completion is truly schemed；
Second generation module, truly schemes for the n-th layer image according to the n-layer gaussian pyramid and each described layer, generation
Resist the generation network of network；
Second completion module, for according to the n-th layer image and the generation network, complex pattern to be repaired described in completion to be obtained
Completion figure after completion.
8. device according to claim 7, it is characterised in that the second completion module includes：
Acquisition submodule, for obtaining the i-th generation figure, the i is 0 to the integer between the n；
Input submodule, for the described i-th generation figure and described i-th layer true figure to be inputted into the generation network, obtains i-th -1
Generation figure；
Wherein, the 0th generation figure is the completion figure；The n-th generation figure be from the n-th layer image, it is random on adopt
Sample is obtained and the n-th layer image resolution ratio identical image.
9. device according to claim 7, it is characterised in that second generation module includes：
First training submodule, for truly being schemed by the n-th layer image and each described layer, trains and meets object function
It is required that confrontation network differentiation network；
Second training submodule, for truly being schemed and the differentiation network by the n-th layer image, each described layer, training
Go out to meet the generation network of object function requirement；
Wherein, the object function is used to weigh the loss during completion.
10. device according to claim 9, it is characterised in that the first training submodule includes：
First acquisition unit, for obtaining jth renewal figure and the true figure of jth layer, the j is 1 integer arrived between the n；
First input block, network is differentiated for updating figure input and improving the true figure of jth layer and the jth, obtains the
One judged result；
First holding unit, for being same when first judged result characterizes the jth renewal figure and the true figure of jth layer
During one image, the parameter improved and differentiate network is kept；
First updating block, for not being when first judged result characterizes the jth renewal figure and the true figure of jth layer
During same image, the parameter for improving differentiation network is updated by Stochastic gradient method and the object function；
Second input block, is initially generated network by the jth renewal figure input for inputting, obtains the renewal figure of jth -1；
Wherein, when the parameter for improving differentiation network no longer updates, described improve is differentiated that network is used as the differentiation net
Network, the n-th renewal figure is that from the n-th layer image, random up-sampling is obtained and the n-th layer image resolution ratio identical figure
Picture.
11. device according to claim 9, it is characterised in that the second training submodule includes：
Second acquisition unit, for obtaining pth improvement figure and the true figure of pth layer, the p is described 1 to arrive between the n
Integer；
3rd input block, the differentiation network is inputted for the true figure of pth layer and the pth to be improved into figure, obtains the
Two judged results；
Second holding unit, for being same when second judged result characterizes the pth improvement figure and the true figure of pth layer
During one image, keep improving the parameter of generation network；
4th input block, for the pth to be improved, figure input is described to improve generation network, obtains the improvement figure of pth -1；
Second updating block, for not being when second judged result characterizes the pth improvement figure and the true figure of pth layer
During same image, the parameter for improving generation network is updated according to Stochastic gradient method and the object function；
The improvement that 4th input block is additionally operable to input the pth improvement figure after updating generates network, obtains pth -1 and changes
Enter figure；
Wherein, when the parameter for improving generation network no longer updates, it regard the generation network that improves as the generation net
Network；N-th improvement figure is that from the n-th layer image, random up-sampling is obtained and the n-th layer image resolution ratio identical figure
Picture.
12. device according to claim 9, it is characterised in that the object function is：
<mrow>
<mi>G</mi>
<mo>=</mo>
<mi>arg</mi>
<munder>
<mrow>
<mi>m</mi>
<mi>i</mi>
<mi>n</mi>
</mrow>
<mi>G</mi>
</munder>
<munder>
<mrow>
<mi>m</mi>
<mi>a</mi>
<mi>x</mi>
</mrow>
<mi>D</mi>
</munder>
<mrow>
<mo>(</mo>
<mi>E</mi>
<mo>&lsqb;</mo>
<msub>
<mi>log</mi>
<mi>a</mi>
</msub>
<mi>D</mi>
<mo>(</mo>
<msub>
<mi>I</mi>
<mi>q</mi>
</msub>
<mo>)</mo>
<mo>&rsqb;</mo>
<mo>+</mo>
<mi>E</mi>
<mo>&lsqb;</mo>
<mn>1</mn>
<mo>-</mo>
<mi>D</mi>
<mo>(</mo>
<msub>
<mi>G</mi>
<mi>q</mi>
</msub>
<mo>)</mo>
<mo>&rsqb;</mo>
<mo>+</mo>
<mi>E</mi>
<mo>&lsqb;</mo>
<mo>|</mo>
<mo>|</mo>
<msub>
<mi>I</mi>
<mi>q</mi>
</msub>
<mo>-</mo>
<msub>
<mi>G</mi>
<mi>q</mi>
</msub>
<mo>|</mo>
<mo>|</mo>
<mo>&rsqb;</mo>
<mo>)</mo>
</mrow>
<mo>;</mo>
</mrow>
Wherein, the G is generation network, and the D is to differentiate network, the IqIt is q true pictures；E [] is to expect；The Gq
It is q generation images；The q is the n to 0 integer, and a is default value.
13. a kind of image completion device, it is characterised in that including：
Processor；
Memory for storing processor-executable instruction；
Wherein, the processor is configured as：
According to complex pattern to be repaired, n-layer gaussian pyramid is generated, the n is positive integer；
By block matching method, the image of each layer in n-layer gaussian pyramid described in completion obtains each described layer after completion
True figure；
Truly schemed according to the n-th layer image of the n-layer gaussian pyramid and each described layer, the generation net of generation confrontation network
Network；
According to the n-th layer image and the generation network, complex pattern to be repaired described in completion obtains the completion figure after completion.
14. a kind of computer-readable recording medium, is stored thereon with computer program, it is characterised in that the program is by processor
The step of any one of claim 1-6 methods described is realized during execution.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710354183.0A CN107133934B (en) | 2017-05-18 | 2017-05-18 | Image completion method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710354183.0A CN107133934B (en) | 2017-05-18 | 2017-05-18 | Image completion method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107133934A true CN107133934A (en) | 2017-09-05 |
| CN107133934B CN107133934B (en) | 2020-03-17 |
Family
ID=59732014
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710354183.0A Active CN107133934B (en) | 2017-05-18 | 2017-05-18 | Image completion method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107133934B (en) |
Cited By (27)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107609560A (en) * | 2017-09-27 | 2018-01-19 | 北京小米移动软件有限公司 | Character recognition method and device |
| CN107679533A (en) * | 2017-09-27 | 2018-02-09 | 北京小米移动软件有限公司 | Character recognition method and device |
| CN107767384A (en) * | 2017-11-03 | 2018-03-06 | 电子科技大学 | A kind of image, semantic dividing method based on dual training |
| CN107798669A (en) * | 2017-12-08 | 2018-03-13 | 北京小米移动软件有限公司 | Image defogging method, device and computer-readable recording medium |
| CN107945140A (en) * | 2017-12-20 | 2018-04-20 | 中国科学院深圳先进技术研究院 | A kind of image repair method, device and equipment |
| CN107945133A (en) * | 2017-11-30 | 2018-04-20 | 北京小米移动软件有限公司 | Image processing method and device |
| CN107958472A (en) * | 2017-10-30 | 2018-04-24 | 深圳先进技术研究院 | PET imaging methods, device, equipment and storage medium based on sparse projection data |
| CN107958471A (en) * | 2017-10-30 | 2018-04-24 | 深圳先进技术研究院 | CT imaging methods, device, CT equipment and storage medium based on lack sampling data |
| CN107977511A (en) * | 2017-11-30 | 2018-05-01 | 浙江传媒学院 | A kind of industrial design material high-fidelity real-time emulation algorithm based on deep learning |
| CN108122212A (en) * | 2017-12-21 | 2018-06-05 | 北京小米移动软件有限公司 | Image repair method and device |
| CN108171663A (en) * | 2017-12-22 | 2018-06-15 | 哈尔滨工业大学 | The image completion system for the convolutional neural networks that feature based figure arest neighbors is replaced |
| CN108226892A (en) * | 2018-03-27 | 2018-06-29 | 天津大学 | A kind of radar signal restoration methods under complicated noise based on deep learning |
| CN108269245A (en) * | 2018-01-26 | 2018-07-10 | 深圳市唯特视科技有限公司 | A kind of eyes image restorative procedure based on novel generation confrontation network |
| CN108460830A (en) * | 2018-05-09 | 2018-08-28 | 厦门美图之家科技有限公司 | Image repair method, device and image processing equipment |
| CN108470326A (en) * | 2018-03-27 | 2018-08-31 | 北京小米移动软件有限公司 | Image completion method and device |
| CN108520503A (en) * | 2018-04-13 | 2018-09-11 | 湘潭大学 | A method of based on self-encoding encoder and generating confrontation network restoration face Incomplete image |
| CN108564550A (en) * | 2018-04-25 | 2018-09-21 | Oppo广东移动通信有限公司 | Image processing method, device and terminal device |
| CN108564527A (en) * | 2018-04-04 | 2018-09-21 | 百度在线网络技术（北京）有限公司 | The method and device of the completion of panorama sketch content and reparation based on neural network |
| CN108573479A (en) * | 2018-04-16 | 2018-09-25 | 西安电子科技大学 | The facial image deblurring and restoration methods of confrontation type network are generated based on antithesis |
| CN108805418A (en) * | 2018-05-22 | 2018-11-13 | 福州大学 | A kind of traffic data fill method fighting network based on production |
| CN108898527A (en) * | 2018-06-21 | 2018-11-27 | 福州大学 | A kind of traffic data fill method based on the generation model for having loss measurement |
| CN109003297A (en) * | 2018-07-18 | 2018-12-14 | 亮风台（上海）信息科技有限公司 | A kind of monocular depth estimation method, device, terminal and storage medium |
| CN109118438A (en) * | 2018-06-29 | 2019-01-01 | 上海航天控制技术研究所 | A kind of Gaussian Blur image recovery method based on generation confrontation network |
| CN109165664A (en) * | 2018-07-04 | 2019-01-08 | 华南理工大学 | A kind of attribute missing data collection completion and prediction technique based on generation confrontation network |
| CN109360159A (en) * | 2018-09-07 | 2019-02-19 | 华南理工大学 | A kind of image completion method based on generation confrontation network model |
| CN109685724A (en) * | 2018-11-13 | 2019-04-26 | 天津大学 | A kind of symmetrical perception facial image complementing method based on deep learning |
| CN109712092A (en) * | 2018-12-18 | 2019-05-03 | 上海中信信息发展股份有限公司 | Archives scan image repair method, device and electronic equipment |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102509271A (en) * | 2011-11-21 | 2012-06-20 | 洪涛 | Image restoration method based on multi-dimensional decomposition, iteration enhancement and correction |
| CN104282000A (en) * | 2014-09-15 | 2015-01-14 | 天津大学 | Image repairing method based on rotation and scale change |
| US20160300327A1 (en) * | 2015-04-08 | 2016-10-13 | Google Inc. | Image Editing and Repair |
| CN106683048A (en) * | 2016-11-30 | 2017-05-17 | 浙江宇视科技有限公司 | Image super-resolution method and image super-resolution equipment |
-
2017
- 2017-05-18 CN CN201710354183.0A patent/CN107133934B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102509271A (en) * | 2011-11-21 | 2012-06-20 | 洪涛 | Image restoration method based on multi-dimensional decomposition, iteration enhancement and correction |
| CN104282000A (en) * | 2014-09-15 | 2015-01-14 | 天津大学 | Image repairing method based on rotation and scale change |
| US20160300327A1 (en) * | 2015-04-08 | 2016-10-13 | Google Inc. | Image Editing and Repair |
| CN106683048A (en) * | 2016-11-30 | 2017-05-17 | 浙江宇视科技有限公司 | Image super-resolution method and image super-resolution equipment |
Non-Patent Citations (2)
| Title |
|---|
| EMILY DENTON 等: "Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks", 《29TH ANNUAL CONFERENCE ON NEURAL INFORMATION PROCESSING SYSTEMS》 * |
| IAN J. GOODFELLOW 等: "Generative Adversarial Nets", 《ADVANCES IN NEURAL INFORMATION PROCESSING SYSTEMS》 * |
Cited By (40)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107679533A (en) * | 2017-09-27 | 2018-02-09 | 北京小米移动软件有限公司 | Character recognition method and device |
| CN107609560A (en) * | 2017-09-27 | 2018-01-19 | 北京小米移动软件有限公司 | Character recognition method and device |
| CN107958471B (en) * | 2017-10-30 | 2020-12-18 | 深圳先进技术研究院 | CT imaging method and device based on undersampled data, CT equipment and storage medium |
| CN107958472B (en) * | 2017-10-30 | 2020-12-25 | 深圳先进技术研究院 | PET imaging method, device and equipment based on sparse projection data and storage medium |
| CN107958472A (en) * | 2017-10-30 | 2018-04-24 | 深圳先进技术研究院 | PET imaging methods, device, equipment and storage medium based on sparse projection data |
| CN107958471A (en) * | 2017-10-30 | 2018-04-24 | 深圳先进技术研究院 | CT imaging methods, device, CT equipment and storage medium based on lack sampling data |
| CN107767384A (en) * | 2017-11-03 | 2018-03-06 | 电子科技大学 | A kind of image, semantic dividing method based on dual training |
| CN107767384B (en) * | 2017-11-03 | 2021-12-03 | 电子科技大学 | Image semantic segmentation method based on countermeasure training |
| CN107945133B (en) * | 2017-11-30 | 2022-08-05 | 北京小米移动软件有限公司 | Image processing method and device |
| CN107945133A (en) * | 2017-11-30 | 2018-04-20 | 北京小米移动软件有限公司 | Image processing method and device |
| CN107977511A (en) * | 2017-11-30 | 2018-05-01 | 浙江传媒学院 | A kind of industrial design material high-fidelity real-time emulation algorithm based on deep learning |
| CN107798669A (en) * | 2017-12-08 | 2018-03-13 | 北京小米移动软件有限公司 | Image defogging method, device and computer-readable recording medium |
| CN107798669B (en) * | 2017-12-08 | 2021-12-21 | 北京小米移动软件有限公司 | Image defogging method and device and computer readable storage medium |
| CN107945140A (en) * | 2017-12-20 | 2018-04-20 | 中国科学院深圳先进技术研究院 | A kind of image repair method, device and equipment |
| CN108122212A (en) * | 2017-12-21 | 2018-06-05 | 北京小米移动软件有限公司 | Image repair method and device |
| CN108171663A (en) * | 2017-12-22 | 2018-06-15 | 哈尔滨工业大学 | The image completion system for the convolutional neural networks that feature based figure arest neighbors is replaced |
| CN108171663B (en) * | 2017-12-22 | 2021-05-25 | 哈尔滨工业大学 | Image filling system of convolutional neural network based on feature map nearest neighbor replacement |
| CN108269245A (en) * | 2018-01-26 | 2018-07-10 | 深圳市唯特视科技有限公司 | A kind of eyes image restorative procedure based on novel generation confrontation network |
| CN108226892A (en) * | 2018-03-27 | 2018-06-29 | 天津大学 | A kind of radar signal restoration methods under complicated noise based on deep learning |
| CN108470326B (en) * | 2018-03-27 | 2022-01-11 | 北京小米移动软件有限公司 | Image completion method and device |
| CN108470326A (en) * | 2018-03-27 | 2018-08-31 | 北京小米移动软件有限公司 | Image completion method and device |
| CN108564527A (en) * | 2018-04-04 | 2018-09-21 | 百度在线网络技术（北京）有限公司 | The method and device of the completion of panorama sketch content and reparation based on neural network |
| CN108520503A (en) * | 2018-04-13 | 2018-09-11 | 湘潭大学 | A method of based on self-encoding encoder and generating confrontation network restoration face Incomplete image |
| CN108520503B (en) * | 2018-04-13 | 2020-12-22 | 湘潭大学 | Face defect image restoration method based on self-encoder and generation countermeasure network |
| CN108573479A (en) * | 2018-04-16 | 2018-09-25 | 西安电子科技大学 | The facial image deblurring and restoration methods of confrontation type network are generated based on antithesis |
| CN108564550B (en) * | 2018-04-25 | 2020-10-02 | Oppo广东移动通信有限公司 | Image processing method and device and terminal equipment |
| CN108564550A (en) * | 2018-04-25 | 2018-09-21 | Oppo广东移动通信有限公司 | Image processing method, device and terminal device |
| CN108460830A (en) * | 2018-05-09 | 2018-08-28 | 厦门美图之家科技有限公司 | Image repair method, device and image processing equipment |
| CN108805418A (en) * | 2018-05-22 | 2018-11-13 | 福州大学 | A kind of traffic data fill method fighting network based on production |
| CN108898527A (en) * | 2018-06-21 | 2018-11-27 | 福州大学 | A kind of traffic data fill method based on the generation model for having loss measurement |
| CN108898527B (en) * | 2018-06-21 | 2021-10-29 | 福州大学 | Traffic data filling method of generative model based on destructive measurement |
| CN109118438A (en) * | 2018-06-29 | 2019-01-01 | 上海航天控制技术研究所 | A kind of Gaussian Blur image recovery method based on generation confrontation network |
| CN109165664A (en) * | 2018-07-04 | 2019-01-08 | 华南理工大学 | A kind of attribute missing data collection completion and prediction technique based on generation confrontation network |
| CN109165664B (en) * | 2018-07-04 | 2020-09-22 | 华南理工大学 | Attribute-missing data set completion and prediction method based on generation of countermeasure network |
| CN109003297A (en) * | 2018-07-18 | 2018-12-14 | 亮风台（上海）信息科技有限公司 | A kind of monocular depth estimation method, device, terminal and storage medium |
| CN109003297B (en) * | 2018-07-18 | 2020-11-24 | 亮风台（上海）信息科技有限公司 | Monocular depth estimation method, device, terminal and storage medium |
| CN109360159A (en) * | 2018-09-07 | 2019-02-19 | 华南理工大学 | A kind of image completion method based on generation confrontation network model |
| CN109685724A (en) * | 2018-11-13 | 2019-04-26 | 天津大学 | A kind of symmetrical perception facial image complementing method based on deep learning |
| CN109712092B (en) * | 2018-12-18 | 2021-01-05 | 上海信联信息发展股份有限公司 | File scanning image restoration method and device and electronic equipment |
| CN109712092A (en) * | 2018-12-18 | 2019-05-03 | 上海中信信息发展股份有限公司 | Archives scan image repair method, device and electronic equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107133934B (en) | 2020-03-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107133934A (en) | Image completion method and device | |
| US11966839B2 (en) | Auto-regressive neural network systems with a soft attention mechanism using support data patches | |
| CN110210486B (en) | Sketch annotation information-based generation countermeasure transfer learning method | |
| McGeoch et al. | Experimental evaluation of an adiabiatic quantum system for combinatorial optimization | |
| CN111833237B (en) | Image registration method based on convolutional neural network and local homography transformation | |
| CN108416805B (en) | Intrinsic image decomposition method and device based on deep learning | |
| CN104166859B (en) | Based on SSAE and FSALS SVM Classification of Polarimetric SAR Image | |
| Ning et al. | Accurate and lightweight image super-resolution with model-guided deep unfolding network | |
| CN109643383A (en) | Domain separates neural network | |
| CN107330437A (en) | Feature extracting method based on the real-time detection model of convolutional neural networks target | |
| CN111242852A (en) | Boundary aware object removal and content filling | |
| CN110766038A (en) | Unsupervised landform classification model training and landform image construction method | |
| CN109902667A (en) | Human face in-vivo detection method based on light stream guide features block and convolution GRU | |
| CN110023964A (en) | Training and/or the intermediate output that spectrum picture is generated using neural network model | |
| CN109934239A (en) | Image characteristic extracting method | |
| CN111179272B (en) | Rapid semantic segmentation method for road scene | |
| CN113378812A (en) | Digital dial plate identification method based on Mask R-CNN and CRNN | |
| Song et al. | Siamese-discriminant deep reinforcement learning for solving jigsaw puzzles with large eroded gaps | |
| CN107943750A (en) | A kind of decomposition convolution method based on WGAN models | |
| CN113554653A (en) | Semantic segmentation method for long-tail distribution of point cloud data based on mutual information calibration | |
| CN109978897A (en) | A kind of multiple dimensioned heterologous remote sensing image registration method and device for generating confrontation network | |
| CN109102018A (en) | The multi-tag classification method of Gynecological secretion image | |
| CN105354833B (en) | A kind of method and apparatus of shadow Detection | |
| CN111985488A (en) | Target detection segmentation method and system based on offline Gaussian model | |
| CN107679474A (en) | Face matching process and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |