CN111292262A

CN111292262A - Image processing method, image processing apparatus, electronic device, and storage medium

Info

Publication number: CN111292262A
Application number: CN202010060550.8A
Authority: CN
Inventors: 储文青; 邰颖; 汪铖杰; 李季檩; 葛彦昊
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd
Priority date: 2020-01-19
Filing date: 2020-01-19
Publication date: 2020-06-16
Anticipated expiration: 2040-01-19
Also published as: CN111292262B

Abstract

The embodiment of the invention discloses an image processing method, an image processing device, electronic equipment and a storage medium, wherein the image processing method comprises the following steps: the method comprises the steps of obtaining a material image sample and an image sample to be beautified, beautifying the image sample to be beautified by using a generator and the material image sample in a preset network model to obtain an image sample after beautification, generating corresponding difference information of the image sample after beautification and the image sample to be beautified under different scales through a discriminator in the preset network model, converging the preset network model according to the corresponding difference information under different scales to obtain a generated confrontation network model, and beautifying the image to be beautified based on the generated confrontation network model to obtain an beautified image.

Description

Image processing method, image processing apparatus, electronic device, and storage medium

Technical Field

The present invention relates to the field of computer technologies, and in particular, to an image processing method and apparatus, an electronic device, and a storage medium.

Background

With the rapid development of computer technology, terminal applications, such as a camera and image editing software, for processing pictures can be installed in terminal devices, such as smart phones, handheld computers, tablet computers, and the like, and users can add special effects, decorate and beautify, beautify and make up, and/or change figures and the like to original pictures (such as people, landscapes, buildings, and the like) or videos based on the terminal applications. People usually enjoy beauty or fun, and people can choose to beautify or modify the face photos appropriately when disclosing the photos in social network sites or live broadcast network sites.

Taking image beautifying as an example, the processing process of the image modification terminal application on the face image data usually performs image processing on the face image according to an image beautifying algorithm and a material image, however, in the existing image processing scheme, the processing effect is often relatively rigid and the beautifying effect is relatively poor.

Disclosure of Invention

The embodiment of the invention provides an image processing method, an image processing device, electronic equipment and a storage medium, which can improve the image beautifying effect.

The embodiment of the invention provides an image processing method, which comprises the following steps:

acquiring a material image sample and an image sample to be beautified;

beautifying the image sample to be beautified by using a generator in a preset network model and a material image sample to obtain an beautified image sample;

generating corresponding difference information of the beautified image sample and the image sample to be beautified under different scales through a discriminator in a preset network model;

converging the preset network model according to the corresponding difference information under different scales to obtain a generated confrontation network model;

and beautifying the image to be beautified based on the generated confrontation network model to obtain the beautified image.

Correspondingly, an embodiment of the present invention further provides an image processing apparatus, including:

the acquisition module is used for acquiring a material image sample and an image sample to be beautified;

the first beautifying module is used for beautifying the image sample to be beautified by utilizing a generator in a preset network model and a material image sample to obtain an beautified image sample;

the generation module is used for generating corresponding difference information of the beautified image sample and the image sample to be beautified under different scales through a discriminator in a preset network model;

the convergence module is used for converging the preset network model according to the corresponding difference information under different scales to obtain a generated confrontation network model;

and the second beautifying module is used for beautifying the image to be beautified based on the generated confrontation network model to obtain a beautified image.

Optionally, in some embodiments of the present invention, the generating module includes:

the scale transformation unit is used for carrying out image scale transformation on the beautified image samples to obtain a plurality of first image samples and carrying out image scale transformation on the image samples to be beautified to obtain a plurality of second image samples;

the adding unit is used for adding the first image sample and the second image sample with the same scale into the same set to obtain a plurality of sample pairs with the same scale;

and the generating unit is used for generating difference information of each same-scale sample pair through a discriminator in a preset network model.

Optionally, in some embodiments of the present invention, the generating unit includes:

the extraction subunit is used for extracting the scale of each same-scale sample pair;

the determining subunit is used for determining a same-scale sample pair with the scale larger than a preset threshold value as a first sample pair, and determining a same-scale sample pair with the scale smaller than or equal to the preset threshold value as a second sample pair;

a construction subunit for constructing a plurality of first regions on a first image sample of the first sample pair;

the generation subunit is used for generating difference information between each first area and the corresponding area of the second image sample through a discriminator in a preset network model to obtain first difference information, and generating difference information of each second sample pair through the discriminator in the preset network model to obtain second difference information;

the convergence module is specifically configured to: and converging the preset network model according to the first difference information and the second difference information to obtain the generated countermeasure network.

Optionally, in some embodiments of the present invention, the convergence module includes:

the building unit is used for building a loss function corresponding to the preset network model according to the first difference information and the second difference information to obtain a target loss function;

and the convergence unit is used for converging a preset network model based on the target loss function to obtain a generated countermeasure network.

Optionally, in some embodiments of the present invention, the convergence unit is specifically configured to:

extracting an image error value from the first difference information to obtain a first image error value, and;

extracting an image error value from the second difference information to obtain a second image error value;

and constructing a loss function corresponding to a preset network model based on the first image error value, the second image error value and a preset gradient optimization algorithm to obtain a target loss function.

Optionally, in some embodiments of the invention, the first beautification module includes:

the extraction unit is used for respectively extracting the characteristics of the material image sample and the image sample to be beautified by utilizing the convolution layer in the generator in the preset network model to obtain a first characteristic vector corresponding to the material image sample and a second characteristic vector corresponding to the image sample to be beautified;

and the beautifying unit is used for beautifying the image sample to be beautified based on the first feature vector and the second feature vector to obtain an beautified image sample.

Optionally, in some embodiments of the present invention, the beautifying unit is specifically configured to:

splicing the first feature vector and the second feature vector to obtain a spliced feature vector;

and generating a beautified image sample based on the spliced feature vectors.

Optionally, in some embodiments of the present invention, the second beautification module is specifically configured to:

receiving an image beautifying request, wherein the image beautifying request carries a material image and an image to be beautified;

and beautifying the image to be beautified based on the generated confrontation network model and the material image to obtain an beautified image.

Optionally, in some embodiments of the present invention, the apparatus further includes a processing module, where the processing module is specifically configured to:

determining an area to be beautified in an image sample to be beautified according to a preset strategy;

intercepting image blocks corresponding to the area to be beautified from the image sample to be beautified to obtain a processed image sample to be beautified;

the first beautification module is specifically configured to: and beautifying the processed image sample to be beautified by using a generator and the material image sample in a preset network model to obtain an beautified image sample.

After a material image sample and an image sample to be beautified are obtained, beautifying processing is carried out on the image sample to be beautified by using a generator in a preset network model and the material image sample to obtain an beautified image sample, then, corresponding difference information of the beautified image sample and the image sample to be beautified under different scales is generated through a discriminator in the preset network model, then, the preset network model is converged according to the corresponding difference information under different scales to obtain a generated confrontation network model, and finally, beautifying processing is carried out on the image to be beautified based on the generated confrontation network model to obtain an beautified image. Therefore, the scheme can improve the beautification effect of the image.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.

FIG. 1a is a schematic view of a scene of an image processing method according to an embodiment of the present invention;

FIG. 1b is a schematic flow chart of an image processing method according to an embodiment of the present invention;

FIG. 1c is a schematic diagram of face beautification in the image processing method according to the embodiment of the present invention;

fig. 1d is a schematic structural diagram of a preset network model in the image processing method according to the embodiment of the present invention;

FIG. 1e is a schematic diagram of an image processing method and a first area according to an embodiment of the present invention

FIG. 1f is a schematic diagram of a local feature error in an image processing method according to an embodiment of the present invention;

FIG. 2 is a schematic flow chart of an image processing method according to an embodiment of the present invention;

FIG. 3a is a schematic structural diagram of an image processing apparatus according to an embodiment of the present invention;

FIG. 3b is a schematic diagram of another structure of an image processing apparatus according to an embodiment of the present invention

Fig. 4 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

The embodiment of the invention provides an image processing method, an image processing device, electronic equipment and a storage medium.

First, a concept of image beautification is introduced, which is a behavior of processing image information using a computer to satisfy a human visual psychology or an application demand, such as increasing contrast, removing blur and noise, correcting geometric distortion, and face changing of a human figure.

The image processing apparatus may be specifically integrated in a terminal or a server, the terminal may include a mobile phone, a tablet Computer, a Personal Computer (PC), or a monitoring device, and the server may include an independently operating server or a distributed server, or may include a server cluster including a plurality of servers.

For example, referring to fig. 1a, the image processing apparatus is integrated on a terminal, the terminal may include a camera, and in a model training phase, the terminal may first obtain a material image sample and an image sample to be beautified, then the terminal may perform beautification processing on the image sample to be beautified by using a generator in a preset network model and the material image sample to obtain an beautified image sample, then the terminal may generate difference information corresponding to the beautified image sample and the image sample to be beautified at different scales through a discriminator in the preset network model, and finally the terminal converges the preset network model according to the corresponding difference information at different scales to obtain a generated confrontation network model; in the using stage, when the terminal receives an image beautifying request triggered by a user, the terminal can acquire a corresponding image to be beautified and a material image according to the image beautifying request, for example, the user needs to perform expression replacement on self-portrait, and after acquiring the image to be beautified and the material image, the terminal performs beautifying processing on the image to be beautified based on a generated confrontation network model to obtain an beautified image.

Compared with the existing image processing method, the method generates the corresponding difference information of the beautified image sample and the image sample to be beautified under different scales through the discriminator in the preset network model, then converges the preset network model according to the corresponding difference information under different scales to obtain the generated confrontation network model, namely, in the training stage, the confrontation between the discriminator and the generator is realized by considering the relationship between the beautified image sample and the image sample to be beautified, so that the parameters of the generated confrontation network model are optimized, and the generator can improve the image beautification effect when in use.

The following are detailed below. It should be noted that the description sequence of the following embodiments is not intended to limit the priority sequence of the embodiments.

An image processing method comprising: the method comprises the steps of obtaining a material image sample and an image sample to be beautified, beautifying the image sample to be beautified by utilizing a generator in a preset network model and the material image sample to obtain an image sample after beautification, generating difference information corresponding to the image sample after beautification and the image sample to be beautified under different scales through a discriminator in the preset network model, converging the preset network model according to the corresponding difference information under different scales to obtain a generated confrontation network model, and beautifying the image to be beautified based on the generated confrontation network model to obtain an beautified image.

Referring to fig. 1b, fig. 1b is a schematic flow chart of an image processing method according to an embodiment of the invention. The specific flow of the image processing method may be as follows:

101. and acquiring a material image sample and an image sample to be beautified.

For example, the image sample to be beautified may be a sample including a human body image, where the human body image may include an image of a head, an image of a trunk, and images of four limbs, it should be noted that the image of the head may include a face image, and the material image may be an image including an expression, an image including various human body poses, and/or an image including various wallpapers, and the like, which is specifically selected according to an actual situation and is not described herein again. The method for acquiring the material image sample and the image sample to be beautified can be various, for example, the material image sample and the image sample to be beautified can be acquired from a local database, data can be pulled through accessing a network interface, and the data can be acquired by shooting through a camera in real time, and is specifically determined according to actual conditions.

102. And beautifying the material image sample and the image sample to be beautified by using a generator in the preset network model to obtain an beautified image sample.

The preset beautification model may be a Generic Adaptive Networks (GAN) model. Generating a countermeasure network is a deep learning model that includes at least two modules in a framework: the generator (Generative Model) and the discriminator (Discriminative Model) produce quite good output through mutual game learning of the generator and the discriminator. In the original GAN theory, it is not required that G and D are both neural networks, but only that functions that can be generated and discriminated correspondingly are fitted. Deep neural networks are generally used as G and D in practice. An excellent GAN application requires a good training method, otherwise the output may be unsatisfactory due to the freedom of neural network models.

Wherein, the discriminator needs to input variables, and the variables are predicted by a certain model. The generator randomly generates observed data given some kind of implicit information. For example, the arbiter: given a graph, it is determined whether the animal in the graph is a cat or a dog. A generator: a new cat (not in the data set) is generated for a series of pictures of cats, where the conditional feature network can be used to extract specific conditional features, such as specific beautification features. The setting mode of the specific condition characteristic can be various, for example, the setting mode can be flexibly set according to the requirement of practical application, and the network setting of the pre-trained condition characteristic can also be stored in the network equipment. In addition, the specific condition features may be built into the network device, or may be stored in a memory and transmitted to the network device, and so on. For example, the conditional feature network may have a two-class network. As the name implies, a two-class network is one in which data input into the network is divided into two classes, e.g., 0 or 1, yes or no, etc. For example, the binary network may have the ability to recognize un-beautified images and beautified images through early training.

Specifically, feature extraction may be performed on the material image sample and the image sample to be beautified, and then, image beautification is performed based on the extracted features, so as to obtain an image sample after beautification, that is, optionally, in some embodiments, the step "beautify the image sample to be beautified by using the generator in the preset network model and the material image sample, so as to obtain an image sample after beautification" may specifically include:

(11) respectively extracting the characteristics of the material image sample and the image sample to be beautified by utilizing a convolution layer in a generator in a preset network model to obtain a first characteristic vector corresponding to the material image sample and a second characteristic vector corresponding to the image sample to be beautified;

(12) and beautifying the image sample to be beautified based on the first feature vector and the second feature vector to obtain an beautified image sample.

Specifically, the first feature vector and the second feature vector may be spliced in a feature dimension to obtain a spliced feature vector, and then the beautified image sample is generated based on the spliced feature vector, that is, optionally, in some embodiments, the step "beautify the image sample to be beautified based on the first feature vector and the second feature vector to obtain the beautified image sample" may specifically include:

(21) splicing the first feature vector and the second feature vector to obtain a spliced feature vector;

(22) and generating a beautified image sample based on the spliced feature vectors.

Wherein, the generator and the discriminator can comprise a full connection layer, a multi-layer convolution layer and the like.

And (3) rolling layers: the method is mainly used for feature extraction of an input image (such as an image sample to be beautified or a material image sample), wherein the size of a convolution kernel and the number of the convolution kernels can be determined according to practical application, for example, the sizes of the convolution kernels from a first layer of convolution layer to a fourth layer of convolution layer can be (7, 7), (5, 5), (3, 3), (3, 3); optionally, in order to reduce the complexity of the calculation and improve the calculation efficiency, in this embodiment, the sizes of convolution kernels of the four convolution layers may all be set to (3, 3), the activation functions all adopt "relu (Linear rectification function, Rectified Linear Unit)", the padding (padding, which refers to a space between an attribute definition element border and an element content) modes are all set to "same", and the "same" padding mode may be simply understood as padding an edge with 0, and the number of 0 padding on the left side (upper side) is the same as or less than the number of 0 padding on the right side (lower side). Optionally, the convolutional layers may be directly connected to each other, so as to accelerate the network convergence speed, and in order to further reduce the amount of computation, downsampling (downsampling) may be performed on all layers or any 1 to 2 layers of the second to fourth convolutional layers, where the downsampling operation is substantially the same as the operation of convolution, and the downsampling convolution kernel is only a maximum value (max) or an average value (average) of corresponding positions.

It should be noted that, for convenience of description, in the embodiment of the present invention, both the layer where the activation function is located and the down-sampling layer (also referred to as a pooling layer) are included in the convolution layer, and it should be understood that the structure may also be considered to include the convolution layer, the layer where the activation function is located, the down-sampling layer (i.e., a pooling layer), and a full-connection layer, and of course, the structure may also include an input layer for inputting data and an output layer for outputting data, which are not described herein again.

Full connection layer: the learned features may be mapped to a sample label space, which mainly functions as a "classifier" in the whole convolutional neural network, and each node of the fully-connected layer is connected to all nodes output by the previous layer (e.g., the down-sampling layer in the convolutional layer), where one node of the fully-connected layer is referred to as one neuron in the fully-connected layer, and the number of neurons in the fully-connected layer may be determined according to the requirements of the practical application, for example, in the text detection model, the number of neurons in the fully-connected layer may be set to 512 each, or may be set to 128 each, and so on. Similar to the convolutional layer, optionally, in the fully-connected layer, a non-linear factor may be added by adding an activation function, for example, an activation function sigmoid (sigmoid function) may be added.

It should be noted that, in order to make the beautification effect of the model more realistic and to better process the details of the image, before beautifying the image sample to be beautified by using the generator and the material image sample in the preset network model, the image sample to be beautified may be preprocessed, for example, before determining the area to be beautified in the image sample to be beautified, then retaining the image block corresponding to the area to be beautified, and finally beautifying the retained image block by using the generator and the material image sample in the preset network model, that is, optionally, in some embodiments, before performing beautification processing on the image sample to be beautified by using the generator and the material image sample in the preset network model, the steps may specifically include:

(31) determining an area to be beautified in an image sample to be beautified according to a preset strategy;

(32) and intercepting an image block corresponding to the area to be beautified from the image sample to be beautified to obtain the processed image sample to be beautified.

For example, referring to fig. 1c, the image sample to be beautified is an image sample including a human face, and the material image sample is an expression a, first, the area to be beautified is determined to be the area where the human face is located in the image sample to be beautified according to a preset policy, then, an image block corresponding to the area to be beautified is intercepted from the image sample to be beautified, so as to obtain the processed image sample to be beautified, and after the processed image sample to be beautified is obtained, the processed image sample to be beautified can be beautified by using a generator in a preset network model and the material image sample, so as to obtain the beautified image sample, as shown in fig. 1 c.

103. And generating corresponding difference information of the beautified image sample and the image sample to be beautified under different scales by a discriminator in a preset network model.

The preset network model may include a generator and a discriminator, and specific network parameters of the generator and the discriminator may be set according to requirements of actual applications.

The preset network model comprises a generator and a discriminator, as shown in fig. 1d, after the material image sample and the image sample to be beautified pass through the generator, generating a beautified image sample, generating corresponding difference information between the beautified image sample and the image sample to be beautified under different scales by a discriminator, the difference information can be used to characterize whether the beautified image sample and the image sample to be beautified belong to the real image, for example, the discriminator can generate the difference information of the beautified image sample and the image sample to be beautified under the scale of 40 × 40 and the difference information under the scale of 20 × 20, that is, optionally, in some embodiments, the step "generating difference information corresponding to the beautified image sample and the image sample to be beautified at different scales through a discriminator in the preset network model" may specifically include:

(41) carrying out image scale transformation on the beautified image samples to obtain a plurality of first image samples, and carrying out image scale transformation on the image samples to be beautified to obtain a plurality of second image samples;

(42) adding the first image sample and the second image sample with the same scale into the same set to obtain a plurality of sample pairs with the same scale;

(43) and generating difference information of each same-scale sample pair through a discriminator in a preset network model.

Specifically, the beautified image samples and the image samples to be beautified can be respectively subjected to scale transformation to obtain a plurality of beautified image samples (i.e. first image samples) subjected to scale transformation, and a plurality of scaled image samples to be beautified (i.e., second image samples), for example, 4 first image samples and 4 second image samples are obtained, then, adding the first image sample and the second image sample of the same scale to the same set to obtain a plurality of sample pairs of the same scale, for example, adding the first image sample of 40 × 40 and the second image sample of 40 × 40 to the same set to obtain a sample pair of the same scale, adding the first image sample of 20 × 20 and the second image sample of 20 × 20 to the same set to obtain a sample pair of the same scale, and finally, generating difference information of each sample pair of the same scale by a discriminator in a preset network model.

Further, a scale corresponding to each pair of samples with the same scale may be extracted, the pair of samples with the same scale having a scale larger than a threshold is determined as a first pair of samples, the pair of samples with the same scale having a scale smaller than or equal to a preset threshold is determined as a second pair of samples, then a plurality of first regions are constructed in the first image sample in the first pair of samples, then, a discriminator in the preset network model is used to generate difference information between each first region and the corresponding region of the second image sample to obtain first difference information, and a discriminator in the preset network model is used to generate difference information of each second pair of samples to obtain second difference information, that is, optionally, in some embodiments, the step "generating difference information of each pair of samples with the same scale by the discriminator in the preset network model":

(51) extracting the scale of each same-scale sample pair;

(52) determining the same-scale sample pair with the scale larger than a preset threshold value as a first sample pair, and determining the same-scale sample pair with the scale smaller than or equal to the preset threshold value as a second sample pair;

(53) constructing a plurality of first regions on a first image sample in a first sample pair;

(54) and generating difference information between each first area and the corresponding area of the second image sample through a discriminator in a preset network model to obtain first difference information, and generating difference information of each second sample pair through the discriminator in the preset network model to obtain second difference information.

For example, the preset threshold is 10 × 10, the scales of the plurality of pairs of the same-scale samples are 40 × 40, 20 × 20, 10 × 10, and 5 × 5, that is, the pair of the same-scale samples of 40 × 40 and the pair of the same-scale samples of 20 × 20 may be determined as a first pair of samples, the pair of the same-scale samples of 10 × 10 and the pair of the same-scale samples of 5 × 5 may be determined as a second pair of samples, and a plurality of first regions are constructed on a first image sample in the first pair of samples, where the number of the first regions may be 3, 4, or 5, and specifically selected according to actual situations, and the positions of the 4 first regions on the first image sample are: upper left, upper right, lower left, and lower right, as shown in fig. 1e, it should be noted that the first area and the corresponding area of the second image sample refer to: after obtaining the first difference information and the second difference information, step 104 may be executed to determine a region of the second image sample that matches the first region in the first image sample, for example, the position of the first region a in the first image sample is upper left, and the position of the region b of the second image sample corresponding to the first region a in the second image sample is also upper left.

104. And converging the preset network model according to the corresponding difference information under different scales to obtain the generated confrontation network model.

For example, after the first difference information and the second difference information are obtained, the preset network model may be converged according to the first difference information and the second difference information, that is, optionally, in some embodiments, the step "converging the preset network model according to the corresponding difference information at different scales to obtain the generated confrontation network model" may specifically include: and converging the preset network model according to the first difference information and the second difference information to obtain the generated countermeasure network.

Specifically, a loss function corresponding to the preset network model may be constructed according to the first difference information and the second difference information to obtain a target loss function, and then the preset network model is converged based on the target loss function to obtain the generated countermeasure network, that is, optionally, in some embodiments, the step "converging the preset network model according to the first difference information and the second difference information to obtain the generated countermeasure network" may specifically include:

(61) constructing a loss function corresponding to a preset network model according to the first difference information and the second difference information to obtain a target loss function;

(62) and converging the preset network model based on the target loss function to obtain the generated countermeasure network.

The method includes the steps of generating an image error value between a beautified image sample and an image sample to be beautified through a discriminator, and then constructing a loss function corresponding to a preset network model based on the image error value and a preset gradient optimization algorithm to obtain a target loss function, that is, optionally, in some embodiments, the step "converging the preset network model based on the target loss function to obtain a generated countermeasure network" may specifically include:

(71) extracting an image error value from the first difference information to obtain a first image error value, and extracting an image error value from the second difference information to obtain a second image error value;

(72) and constructing a loss function corresponding to the preset network model based on the first image error value, the second image error value and a preset gradient optimization algorithm to obtain a target loss function.

Wherein, the first image error value can be expressed by an confrontation error, a pixel error and a characteristic error between the first image sample and the second image sample in the first sample pair, and similarly, the second image error value is also the same, in the embodiment of the present invention, the generator performs beautification processing on the image sample to be beautified through the material image sample to obtain the beautified image sample, therefore, the confrontation error refers to the confrontation loss between the image sample to be beautified processed by the generator and the image sample to be beautified not processed by the generator, similarly, the pixel error refers to the pixel loss between the image sample to be beautified processed by the generator and the image sample to be beautified not processed by the generator, the characteristic error refers to the characteristic loss between the image sample to be beautified processed by the generator and the image sample to be beautified not processed by the generator, for example, referring to fig. 1f, the local feature error refers to the feature loss between the area a of the image sample to be beautified and the area B of the image sample after beautification, so that there may be a plurality of local feature errors between the image sample to be beautified and the image sample after beautification, which is specifically selected according to the actual situation

The probability that the image sample to be beautified and the beautified image sample are true can be detected by the discriminator, for example, the probability that the image sample to be beautified is detected by the discriminator is 0.8, and the probability that the image sample to be beautified is detected by the discriminator is 0.3, so the countermeasure error between the image sample to be beautified and the beautified image sample is 0.5, and the calculation mode of the pixel error characteristic error and the calculation mode of the characteristic error can refer to the calculation mode of the countermeasure error, and are not described herein again.

105. And beautifying the image to be beautified based on the generated confrontation network model to obtain the beautified image.

For example, specifically, when an image beautification instruction triggered by a user is received, an image to be beautified and a material image corresponding to the image beautification instruction may be obtained, taking face change as an example, the material image may be a face image including a star, and the image to be beautified may be a face image of the user, and then, beautify the image to be beautified and the material image based on the generated confrontation network model to obtain an beautified image, that is, optionally, in some embodiments, the step "beautify the image to be beautified based on the generated confrontation network model to obtain an beautified image" may specifically include:

(81) receiving an image beautifying request, wherein the image beautifying request carries a material image and an image to be beautified;

(82) and beautifying the material image and the image to be beautified based on the generated confrontation network model to obtain a target image.

In order to improve the quality of image beautification, the material image and the image to be beautified can be preprocessed before the material image and the image to be beautified are beautified, for example, the background of the image to be beautified can be removed, if the image to be beautified is a face image, the image of a non-face area can be removed to obtain a preprocessed image, and then the preprocessed image is beautified based on the generated confrontation network model to obtain an beautified image.

After a material image sample and an image sample to be beautified are obtained, beautification processing is carried out on the image sample to be beautified by using a generator in a preset network model and the material image sample to obtain an beautified image sample, then, corresponding difference information of the beautified image sample and the image sample to be beautified under different scales is generated through a discriminator in the preset network model, then, the preset network model is converged according to the corresponding difference information under different scales to obtain a generated confrontation network model, and finally, the beautification processing is carried out on the image to be beautified based on the generated confrontation network model to obtain an beautified image. Compared with the existing image processing method, the method generates the corresponding difference information of the beautified image sample and the image sample to be beautified under different scales through the discriminator in the preset network model, then converges the preset network model according to the corresponding difference information under different scales to obtain the generated confrontation network model, namely, in the training stage, the confrontation between the discriminator and the generator is realized by considering the relationship between the beautified image sample and the image sample to be beautified, so that the parameters of the generated confrontation network model are optimized, and the generator can improve the image beautification effect when in use.

The method according to the examples is further described in detail below by way of example.

In the present embodiment, the image processing apparatus will be described by taking an example in which it is specifically integrated in a terminal.

Referring to fig. 2, a specific flow of an image beautifying method may be as follows:

201. and the terminal acquires a material image sample and an image sample to be beautified.

For example, the image sample to be beautified may be a sample including a human body image, where the human body image may include an image of a head, an image of a trunk, and images of four limbs, it should be noted that the image of the head may include a face image, and the material image may be an image including an expression, an image including various human body poses, and/or an image including various wallpapers, and the like, which is specifically selected according to an actual situation and is not described herein again. The terminal can obtain the material image sample and the image sample to be beautified, for example, the terminal can obtain the material image sample and the image sample to be beautified from a local database, the terminal can also pull data through accessing a network interface, and the terminal can also shoot the material image sample and the image sample to be beautified in real time through a camera, which is determined according to actual conditions.

202. And the terminal beautifies the material image sample and the material image sample to-be-beautified image sample by using a generator in a preset network model to obtain an beautified image sample.

The preset beautification model may be a Generic Adaptive Networks (GAN) model. Generating a countermeasure network is a deep learning model that includes at least two modules in a framework: the generator (Generative Model) and the discriminator (Discriminative Model) produce quite good output through mutual game learning of the generator and the discriminator.

Specifically, the terminal may respectively perform feature extraction on the material image sample and the image sample to be beautified, and perform image beautification based on the extracted features to obtain the beautified image sample, for example, the terminal may respectively perform feature extraction on the material image sample and the image sample to be beautified by using a convolution layer in a generator in a preset network model to obtain a first feature vector corresponding to the material image sample and a second feature vector corresponding to the image sample to be beautified, then, the terminal splices the first feature vector and the second feature vector to obtain a spliced feature vector, and finally, the terminal generates the beautified image sample based on the spliced feature vector.

203. And the terminal generates corresponding difference information of the beautified image sample and the image sample to be beautified under different scales through a discriminator in a preset network model.

The preset network model can comprise a generator and a discriminator, specific network parameters of the generator and the discriminator can be set according to requirements of practical application, the terminal can extract scales corresponding to all same-scale sample pairs, then the terminal determines the same-scale sample pairs with the scales larger than a threshold value as first sample pairs, determines the same-scale sample pairs with the scales smaller than or equal to a preset threshold value as second sample pairs, then, a plurality of first areas are constructed in first image samples in the first sample pairs, then, difference information between the first areas and the corresponding second image sample areas is generated through the discriminator in the preset network model, first difference information is obtained, and difference information of the second sample pairs is generated through the discriminator in the preset network model, and second difference information is obtained.

204. And the terminal converges the preset network model according to the corresponding difference information under different scales to obtain the generated confrontation network model.

The terminal can construct a loss function corresponding to the preset network model according to the first difference information and the second difference information to obtain a target loss function, and then the terminal converges the preset network model based on the target loss function to obtain the generated countermeasure network.

205. And the terminal beautifies the image to be beautified based on the generated confrontation network model to obtain an beautified image.

For example, specifically, when the terminal receives an image beautification instruction triggered by a user, the terminal may obtain an image to be beautified and a material image corresponding to the image beautification instruction, and beautify the image to be beautified and the material image based on the generated confrontation network model to obtain an beautified image.

After a material image sample and an image sample to be beautified are obtained, a generator in a preset network model and the material image sample are utilized to beautify the image sample to be beautified to obtain an beautified image sample, then the terminal generates corresponding difference information of the beautified image sample and the image sample to be beautified under different scales through a discriminator in the preset network model, then the terminal converges the preset network model according to the corresponding difference information under different scales to obtain a generated confrontation network model, and finally the terminal beautifies the image to be beautified based on the generated confrontation network model to obtain an beautified image. Compared with the existing image processing method, the terminal generates the corresponding difference information of the beautified image sample and the image sample to be beautified under different scales through the discriminator in the preset network model, then the terminal converges the preset network model according to the corresponding difference information under different scales to obtain the generated confrontation network model, namely, in the training stage, the confrontation between the discriminator and the generator is realized by considering the relationship between the beautified image sample and the image sample to be beautified, so that the parameters of the generated confrontation network model are optimized, and the generator can improve the image beautification effect when in use.

In order to better implement the image processing method according to the embodiment of the present invention, an embodiment of the present invention further provides an image processing apparatus (processing apparatus for short) based on the foregoing image processing method. The terms are the same as those in the image processing method, and details of implementation can be referred to the description in the method embodiment.

Referring to fig. 3a, fig. 3a is a schematic structural diagram of an image processing apparatus according to an embodiment of the present invention, wherein the identification apparatus may include an obtaining module 301, a first beautification module 302, a generating module 303, a converging module 304, and a second beautification module 305, which may specifically be as follows:

the obtaining module 301 is configured to obtain a material image sample and an image sample to be beautified.

For example, the image sample to be beautified may be a sample including a human body image, where the human body image may include an image of a head, an image of a trunk, and images of four limbs, it should be noted that the image of the head may include a face image, and the material image may be an image including an expression, an image including various human body poses, and/or an image including various wallpapers, and the like, which is specifically selected according to an actual situation and is not described herein again. The material image samples and the image samples to be beautified can be obtained in various ways, for example, the obtaining module 301 can obtain the material image samples and the image samples to be beautified from a local database, the obtaining module 301 can also pull data by accessing a network interface, and the obtaining module 301 can also obtain the material image samples and the image samples to be beautified by shooting in real time through a camera, which is specifically determined according to actual conditions.

The first beautification module 302 is configured to perform beautification processing on an image sample to be beautified by using a generator and a material image sample in a preset network model, so as to obtain an beautified image sample.

The first beautifying module 302 may respectively perform feature extraction on the material image sample and the image sample to be beautified, and perform image beautification based on the extracted features to obtain an image sample after beautification, that is, optionally, in some embodiments, the first beautifying module 302 may specifically include:

Wherein, beautify the unit and can splice first eigenvector and second eigenvector in the feature dimension, obtain the eigenvector after the concatenation, then, generate beautified back image sample based on the eigenvector after the concatenation again, that is, in some embodiments, beautify the unit and specifically can be used for: and splicing the first feature vector and the second feature vector to obtain a spliced feature vector, and generating a beautified image sample based on the spliced feature vector.

Optionally, in some embodiments, referring to fig. 3b, the processing apparatus may further include a processing module 306, and the processing module 306 may specifically be configured to: determining an area to be beautified in an image sample to be beautified according to a preset strategy, and intercepting an image block corresponding to the area to be beautified in the image sample to be beautified to obtain a processed image sample to be beautified;

the first beautification module 302 is specifically configured to: and beautifying the processed image sample to be beautified by using a generator in a preset network model and the material image sample to obtain an beautified image sample.

The generating module 303 is configured to generate difference information corresponding to the beautified image sample and the image sample to be beautified at different scales through a discriminator in a preset network model.

Optionally, in some embodiments, the generating module 303 may specifically include:

Optionally, in some embodiments, the generating unit may specifically include:

a construction subunit for constructing a plurality of first regions on a first image sample in a first sample pair;

and the generating subunit is used for generating difference information between each first area and the corresponding area of the second image sample through a discriminator in the preset network model to obtain first difference information, and generating difference information of each second sample pair through the discriminator in the preset network model to obtain second difference information.

And the convergence module 304 is configured to converge the preset network model according to the corresponding difference information at different scales, so as to obtain a generated confrontation network model.

Optionally, in some embodiments, the convergence module 304 may be specifically configured to: and converging the preset network model according to the first difference information and the second difference information to obtain the generated countermeasure network.

Optionally, in some embodiments, the convergence module 304 may specifically include:

and the convergence unit is used for converging the preset network model based on the target loss function to obtain the generated countermeasure network.

Optionally, in some embodiments, the convergence unit may be specifically configured to: and constructing a loss function corresponding to the preset network model based on the first image error value, the second image error value and a preset gradient optimization algorithm to obtain a target loss function.

And a second beautification module 305, configured to perform beautification processing on the image to be beautified based on the generated confrontation network model, so as to obtain an beautified image.

For example, specifically, when receiving an image beautification instruction triggered by a user, the second beautification module 305 may obtain an image and a material image to be beautified corresponding to the image beautification instruction, and beautify the image and the material image to be beautified based on the generated confrontation network model to obtain an beautified image

Optionally, in some embodiments, the second beautification module 305 may specifically be configured to: receiving an image beautification request, wherein the image beautification request carries a material image and an image to be beautified, and beautifying the image to be beautified based on the generated countermeasure network model and the material image to obtain an beautified image

After the obtaining module 301 of the embodiment of the present invention obtains the material image sample and the image sample to be beautified, the first beautifying module 302 uses the generator in the preset network model and the material image sample to beautify the image sample to be beautified, so as to obtain the beautified image sample, then the generating module 303 generates the difference information corresponding to the beautified image sample and the image sample to be beautified at different scales through the discriminator in the preset network model, then the converging module 304 converges the preset network model according to the corresponding difference information at different scales, so as to obtain the generated confrontation network model, and finally, the second beautifying module 305 beautifies the image to be beautified based on the generated confrontation network model, so as to obtain the beautified image. Compared with the existing image processing method, the method generates the corresponding difference information of the beautified image sample and the image sample to be beautified under different scales through the discriminator in the preset network model, then converges the preset network model according to the corresponding difference information under different scales to obtain the generated confrontation network model, namely, in the training stage, the confrontation between the discriminator and the generator is realized by considering the relationship between the beautified image sample and the image sample to be beautified, so that the parameters of the generated confrontation network model are optimized, and the generator can improve the image beautification effect when in use.

In addition, an embodiment of the present invention further provides an electronic device, as shown in fig. 4, which shows a schematic structural diagram of the electronic device according to the embodiment of the present invention, specifically:

the electronic device may include components such as a processor 401 of one or more processing cores, memory 402 of one or more computer-readable storage media, a power supply 403, and an input unit 404. Those skilled in the art will appreciate that the electronic device configuration shown in fig. 4 does not constitute a limitation of the electronic device and may include more or fewer components than those shown, or some components may be combined, or a different arrangement of components. Wherein:

the processor 401 is a control center of the electronic device, connects various parts of the whole electronic device by various interfaces and lines, performs various functions of the electronic device and processes data by running or executing software programs and/or modules stored in the memory 402 and calling data stored in the memory 402, thereby performing overall monitoring of the electronic device. Optionally, processor 401 may include one or more processing cores; preferably, the processor 401 may integrate an application processor, which mainly handles operating systems, user interfaces, application programs, etc., and a modem processor, which mainly handles wireless communications. It will be appreciated that the modem processor described above may not be integrated into the processor 401.

The memory 402 may be used to store software programs and modules, and the processor 401 executes various functional applications and data processing by operating the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application program required by at least one function (such as a sound playing function, an image playing function, etc.), and the like; the storage data area may store data created according to use of the electronic device, and the like. Further, the memory 402 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid state storage device. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 access to the memory 402.

The electronic device further comprises a power supply 403 for supplying power to the various components, and preferably, the power supply 403 is logically connected to the processor 401 through a power management system, so that functions of managing charging, discharging, and power consumption are realized through the power management system. The power supply 403 may also include any component of one or more dc or ac power sources, recharging systems, power failure detection circuitry, power converters or inverters, power status indicators, and the like.

The electronic device may further include an input unit 404, and the input unit 404 may be used to receive input numeric or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

Although not shown, the electronic device may further include a display unit and the like, which are not described in detail herein. Specifically, in this embodiment, the processor 401 in the electronic device loads the executable file corresponding to the process of one or more application programs into the memory 402 according to the following instructions, and the processor 401 runs the application program stored in the memory 402, thereby implementing various functions as follows:

the method comprises the steps of obtaining a material image sample and an image sample to be beautified, beautifying the image sample to be beautified by utilizing a generator in a preset network model and the material image sample to obtain an image sample after beautification, generating difference information corresponding to the image sample after beautification and the image sample to be beautified under different scales through a discriminator in the preset network model, converging the preset network model according to the corresponding difference information under different scales to obtain a generated confrontation network model, and beautifying the image to be beautified based on the generated confrontation network model to obtain an beautified image.

The above operations can be implemented in the foregoing embodiments, and are not described in detail herein.

It will be understood by those skilled in the art that all or part of the steps of the methods of the above embodiments may be performed by instructions or by associated hardware controlled by the instructions, which may be stored in a computer readable storage medium and loaded and executed by a processor.

To this end, the present invention provides a storage medium, in which a plurality of instructions are stored, and the instructions can be loaded by a processor to execute the steps in any one of the image processing methods provided by the embodiments of the present invention. For example, the instructions may perform the steps of:

Wherein the storage medium may include: read Only Memory (ROM), Random Access Memory (RAM), magnetic or optical disks, and the like.

Since the instructions stored in the storage medium can execute the steps in any image processing method provided in the embodiment of the present invention, the beneficial effects that can be achieved by any image processing method provided in the embodiment of the present invention can be achieved, which are detailed in the foregoing embodiments and will not be described herein again.

The foregoing describes in detail an image processing method, a terminal, an apparatus, an electronic device, and a storage medium according to embodiments of the present invention, and a specific example is applied in the present disclosure to explain the principles and embodiments of the present invention, and the description of the foregoing embodiments is only used to help understand the method and the core idea of the present invention; meanwhile, for those skilled in the art, according to the idea of the present invention, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present invention.

Claims

1. An image processing method, comprising:

acquiring a material image sample and an image sample to be beautified;

2. The method according to claim 1, wherein the generating difference information corresponding to the beautified image sample and the image sample to be beautified at different scales by a discriminator in a preset network model comprises:

carrying out image scale transformation on the beautified image samples to obtain a plurality of first image samples;

carrying out image scale transformation on the image sample to be beautified to obtain a plurality of second image samples;

adding the first image sample and the second image sample with the same scale into the same set to obtain a plurality of sample pairs with the same scale;

and generating difference information of each same-scale sample pair through a discriminator in a preset network model.

3. The method according to claim 2, wherein the generating difference information of each pair of same-scale samples by a discriminator in a preset network model comprises:

extracting the scale of each same-scale sample pair;

determining a same-scale sample pair with the scale larger than a preset threshold value as a first sample pair, and;

determining a same-scale sample pair with the scale smaller than or equal to a preset threshold value as a second sample pair;

constructing a plurality of first regions on a first image sample in the first sample pair;

generating difference information between each first area and the corresponding area of the second image sample through a discriminator in a preset network model to obtain first difference information;

generating difference information of each second sample pair through a discriminator in a preset network model to obtain second difference information;

the converging of the preset network model according to the corresponding difference information under different scales to obtain the generated countermeasure network comprises the following steps: and converging the preset network model according to the first difference information and the second difference information to obtain the generated countermeasure network.

4. The method of claim 3, wherein converging the predetermined network model according to the first difference information and the second difference information to obtain the generated countermeasure network comprises:

constructing a loss function corresponding to a preset network model according to the first difference information and the second difference information to obtain a target loss function;

and converging a preset network model based on the target loss function to obtain a generated countermeasure network.

5. The method according to claim 4, wherein the constructing a loss function corresponding to the preset network model according to the first difference information and the second difference information to obtain a target loss function comprises:

6. The method according to any one of claims 1 to 5, wherein the beautifying processing of the image sample to be beautified by using a generator in a preset network model and a material image sample to obtain an beautified image sample comprises:

respectively extracting the characteristics of the material image sample and the image sample to be beautified by utilizing a convolution layer in a generator in a preset network model to obtain a first characteristic vector corresponding to the material image sample and a second characteristic vector corresponding to the image sample to be beautified;

and beautifying the image sample to be beautified based on the first feature vector and the second feature vector to obtain an beautified image sample.

7. The method according to claim 6, wherein the performing beautification processing on the image sample to be beautified based on the first feature vector and the second feature vector to obtain an image sample after beautification comprises:

and generating a beautified image sample based on the spliced feature vectors.

8. The method according to any one of claims 1 to 5, wherein the beautification processing of the image to be beautified based on the generation of the confrontation network model to obtain the beautified image comprises:

9. The method according to any one of claims 1 to 5, wherein the beautifying processing of the image sample to be beautified by using a generator in a preset network model and a material image sample, before obtaining an beautified image sample, further comprises:

the beautifying processing of the image sample to be beautified by using a generator and a material image sample in a preset network model to obtain an beautified image sample comprises the following steps: and beautifying the processed image sample to be beautified by using a generator and the material image sample in a preset network model to obtain an beautified image sample.

10. An image processing apparatus characterized by comprising:

the convergence module is used for converging the preset network model according to the corresponding difference information under the same scale to obtain a generated confrontation network model;

11. The apparatus of claim 10, wherein the generating module comprises:

12. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the steps of the image processing method according to any of claims 1-9 are implemented when the program is executed by the processor.

13. A computer-readable storage medium, on which a computer program is stored, wherein the computer program, when being executed by a processor, carries out the steps of the image processing method according to any one of claims 1 to 9.