CN113780326A - Image processing method and device, storage medium and electronic equipment - Google Patents
Image processing method and device, storage medium and electronic equipment Download PDFInfo
- Publication number
- CN113780326A CN113780326A CN202110231306.8A CN202110231306A CN113780326A CN 113780326 A CN113780326 A CN 113780326A CN 202110231306 A CN202110231306 A CN 202110231306A CN 113780326 A CN113780326 A CN 113780326A
- Authority
- CN
- China
- Prior art keywords
- feature
- image
- migration
- attribute
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 21
- 230000005012 migration Effects 0.000 claims abstract description 200
- 238000013508 migration Methods 0.000 claims abstract description 200
- 238000000034 method Methods 0.000 claims abstract description 25
- 238000012545 processing Methods 0.000 claims abstract description 17
- 230000004927 fusion Effects 0.000 claims abstract description 8
- 238000000605 extraction Methods 0.000 claims description 26
- 238000006243 chemical reaction Methods 0.000 claims description 14
- 238000004590 computer program Methods 0.000 claims description 9
- 230000002194 synthesizing effect Effects 0.000 claims description 9
- 230000009467 reduction Effects 0.000 claims description 4
- 230000011218 segmentation Effects 0.000 claims description 3
- 238000013526 transfer learning Methods 0.000 abstract description 21
- 238000010586 diagram Methods 0.000 description 14
- 238000012549 training Methods 0.000 description 12
- 238000012546 transfer Methods 0.000 description 12
- 230000006870 function Effects 0.000 description 8
- 230000008569 process Effects 0.000 description 8
- 230000003287 optical effect Effects 0.000 description 6
- 230000015572 biosynthetic process Effects 0.000 description 4
- 238000003786 synthesis reaction Methods 0.000 description 4
- 230000003044 adaptive effect Effects 0.000 description 2
- 238000013135 deep learning Methods 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- 238000003062 neural network model Methods 0.000 description 2
- 239000013307 optical fiber Substances 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- PXFBZOLANLWPMH-UHFFFAOYSA-N 16-Epiaffinine Natural products C1C(C2=CC=CC=C2N2)=C2C(=O)CC2C(=CC)CN(C)C1C2CO PXFBZOLANLWPMH-UHFFFAOYSA-N 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000012512 characterization method Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 239000002131 composite material Substances 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000003058 natural language processing Methods 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 230000008707 rearrangement Effects 0.000 description 1
- 238000009877 rendering Methods 0.000 description 1
- 238000005096 rolling process Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/40—Analysis of texture
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Computation (AREA)
- Bioinformatics & Computational Biology (AREA)
- Evolutionary Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Image Analysis (AREA)
Abstract
The embodiment of the invention discloses an image processing method, an image processing device, a storage medium and electronic equipment, wherein the method comprises the following steps: acquiring an image to be processed, and extracting current attribute features, shape features and texture features of the image to be processed and target attribute features of the image to be processed; inputting the current attribute feature, the shape feature and the target attribute feature into a pre-trained shape migration model to generate a shape feature corresponding to the target attribute; inputting the current attribute feature, the texture feature and the target attribute feature into a pre-trained texture migration model to generate a texture feature corresponding to the target attribute; and performing feature fusion based on the texture features and the shape features corresponding to the target attributes to obtain a target image corresponding to the target attributes. And the transfer learning on two dimensions of the shape feature and the texture feature improves the transfer learning precision of the image to be processed and the quality of the target image.
Description
Technical Field
The present invention relates to an automatic driving technology, and in particular, to an image processing method, an image processing apparatus, a storage medium, and an electronic device.
Background
In recent years, deep learning has achieved excellent results in many fields, from image classification, speech recognition to natural language processing, and the like. At present, deep learning is widely applied to the field of image processing.
However, in the process of implementing the present invention, the inventors found that at least the following technical problems exist in the prior art: in the image migration learning, the migration of the same type adopts a general mode, the individual difference of the learned images is small, and the image migration precision is low.
Disclosure of Invention
The embodiment of the invention provides an image processing method, an image processing device, a storage medium and electronic equipment, and aims to provide image precision.
In a first aspect, an embodiment of the present invention provides an image processing method, including:
acquiring an image to be processed, and extracting current attribute features, shape features and texture features of the image to be processed and target attribute features of the image to be processed;
inputting the current attribute feature, the shape feature and the target attribute feature into a pre-trained shape migration model to generate a shape feature corresponding to the target attribute;
inputting the current attribute feature, the texture feature and the target attribute feature into a pre-trained texture migration model to generate a texture feature corresponding to the target attribute;
and performing feature fusion based on the texture features and the shape features corresponding to the target attributes to obtain a target image corresponding to the target attributes.
In a second aspect, an embodiment of the present invention further provides an image processing method, where:
the characteristic extraction module is used for acquiring an image to be processed and extracting the current attribute characteristic, the shape characteristic and the texture characteristic of the image to be processed and the target attribute characteristic of the image to be processed;
the shape feature migration module is used for inputting the current attribute feature, the shape feature and the target attribute feature into a pre-trained shape migration model to generate a shape feature corresponding to the target attribute;
the texture feature migration module is used for inputting the current attribute feature, the texture feature and the target attribute feature into a pre-trained texture migration model to generate a texture feature corresponding to the target attribute;
and the target image generation module is used for carrying out feature fusion on the basis of the texture features and the shape features corresponding to the target attributes to obtain a target image corresponding to the target attributes.
In a third aspect, an embodiment of the present invention further provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor executes the computer program to implement the image processing method according to any embodiment of the present invention.
In a fourth aspect, the present invention further provides a computer-readable storage medium, on which a computer program is stored, where the computer program is executed by a processor to implement the image processing method according to any embodiment of the present invention.
According to the technical scheme provided by the embodiment of the invention, the shape transfer model and the texture transfer model are set, the image to be processed is decoupled on two dimensions of the shape and the texture, and the shape feature and the texture feature are respectively transferred and learned on the basis of the set shape transfer model and the set texture transfer model, so that the transfer learning with independent distribution of the shape feature and the texture feature of the image to be processed is realized, the transfer learning precision of each dimension feature is improved, meanwhile, the transfer learning on the two dimensions of the shape feature and the texture feature is improved, the comprehensiveness of the transfer learning of the image to be processed is improved, and the transfer learning precision of the image to be processed and the quality of a target image are further improved.
Drawings
Fig. 1 is a schematic flowchart of an image processing method according to an embodiment of the present invention;
FIG. 2 is a schematic structural diagram of a shape migration model or a texture migration model in an embodiment of the present invention;
FIG. 3 is a schematic structural diagram of an attribute migration module according to an embodiment of the present invention;
fig. 4 is a flowchart illustrating an image processing method according to a second embodiment of the present invention;
FIG. 5 is a schematic structural diagram of a texture migration model provided by an embodiment of the present invention;
FIG. 6 is a schematic structural diagram of an attribute migration module according to an embodiment of the present invention;
fig. 7 is a schematic structural diagram of an image processing apparatus according to an embodiment of the present invention;
fig. 8 is a schematic structural diagram of an electronic device according to a fourth embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be further noted that, for the convenience of description, only some of the structures related to the present invention are shown in the drawings, not all of the structures.
Example one
Fig. 1 is a schematic flow chart of an image processing method according to an embodiment of the present invention, where the present embodiment is applicable to a case of performing migration learning on an image, the method may be executed by an image processing apparatus according to an embodiment of the present invention, the deployment and control apparatus may be implemented by software and/or hardware, and the deployment and control apparatus may be configured on an electronic device such as a mobile phone, a tablet computer, and a computer, and specifically includes the following steps:
s110, obtaining an image to be processed, and extracting the current attribute feature, the shape feature and the texture feature of the image to be processed and the target attribute feature of the image to be processed.
And S120, inputting the current attribute feature, the shape feature and the target attribute feature into a pre-trained shape migration model, and generating a shape feature corresponding to the target attribute.
S130, inputting the current attribute feature, the texture feature and the target attribute feature into a pre-trained texture migration model, and generating the texture feature corresponding to the target attribute.
And S140, performing feature fusion based on the texture features and the shape features corresponding to the target attributes to obtain target images corresponding to the target attributes.
In this embodiment, the attribute migration processing is performed on the image to be processed through the shape migration model and the texture migration model, where the attribute may be an attribute of the image to be processed, such as an image style, and may also be an attribute of image content in the image to be processed, such as that the image content in the image to be processed is a person, and the attribute includes, but is not limited to, age, gender, and the like.
In this embodiment, a shape migration model and a texture migration model are set, the image to be processed is subjected to migration learning in the two aspects of the shape feature and the texture feature to obtain a target image after the migration learning, the image to be processed is processed in the two aspects of the shape feature and the texture feature to provide comprehensiveness of the migration learning, and the problems of poor migration learning quality and low image accuracy caused by single feature migration are solved.
Optionally, the image to be processed is a three-dimensional image. And extracting the features of the image to be processed, and extracting the current attribute features, shape features and texture features in the image to be processed. In some embodiments, the shape feature uv map and the texture feature uv texture map of the image to be processed may be obtained by a preset feature extraction model, such as a shape feature extraction model and a texture feature extraction model, which may be, for example, a machine learning model, specifically, a neural network model, or the like, and the specific structures of the shape feature extraction model and the texture feature extraction model are not limited, so that the extraction of the shape feature and the texture feature may be achieved. In some embodiments, obtaining the shape feature and the texture feature of the image to be processed comprises: analyzing an image to be processed through a preset normalized coordinate coding prediction network to obtain normalized coordinate coding map information, and searching UV space coordinate information corresponding to the current normalized coordinate coding map information from a preset corresponding relation between reference normalized coordinate coding map information and reference UV space coordinate information to obtain shape characteristic UV map; and sequentially mapping the current normalized coordinate coding map information to UV space coordinate information to form UV space texture mapping information, namely texture feature UV texture map.
The current attribute features of the images to be processed are exclusive features of the images to be processed, and are used for improving the difference of the images to be processed in the transfer learning process. The current attribute feature of the image to be processed may be obtained based on a preset attribute feature extraction model, the attribute feature extraction model may be set according to an attribute requirement, for example, the attribute is an age, the attribute feature extraction model may be an age estimation model, the age estimation model may identify an age feature of a person in the image to be processed, and the age feature may be a one-hot vector, that is, a vector with a length of n. Illustratively, the attribute is gender, and the attribute feature extraction model may be a gender identification model that can identify gender features of a person in the image to be processed. An exemplary attribute is an image style, and the attribute feature extraction model may be a style recognition model that can recognize the image style of the image to be processed.
The target attribute feature of the image to be processed is a migration learning target of the image to be processed, and exemplarily, if the attribute is age, the target attribute feature is used for representing the target age of the migration learning, if the attribute is gender, the target attribute feature is used for representing the target gender of the migration learning, and if the attribute is image style, the target attribute feature is used for representing the target image style of the migration learning. Taking the attribute as the age as an example, the target attribute feature may be a one-hot vector with a length of n, where n represents the number of age groups, and if the person in the image to be processed belongs to the ith age group, the ith dimension of the vector is 1, and the remaining dimensions are 0, which represents a general age feature. For other attributes, the representation form of the target attribute feature may be determined according to the characterization requirement of each attribute feature, which is not limited herein.
In this embodiment, a shape migration model and a texture migration model are respectively set, and are used to respectively perform migration learning on the shape features and the texture features of the image to be processed. The shape migration model and the texture migration model may be neural network models, and the network structures of the shape migration model and the texture migration model may be the same or different and are obtained by training respectively. In some embodiments, the shape migration model or texture migration model may be, but is not limited to, a GAN (generic adaptive Networks generated countermeasure network) network or a VAE (variant automatic encoder).
And for the transfer learning of the shape characteristics, taking the current attribute characteristics of the image to be processed, the shape characteristics and the target attribute characteristics as input information of a shape transfer model, and performing transfer learning on the input information by the shape transfer model to obtain the shape characteristics corresponding to the target attributes. Similarly, for the transfer learning of the texture features, the current attribute features, the texture features and the target attribute features of the image to be processed are used as the input information of the texture transfer model, and the texture transfer model performs the transfer learning on the input information to obtain the texture features corresponding to the target attributes.
And synthesizing the learned texture features and shape features to obtain a target image, or optionally, synthesizing and rendering the texture features and shape features in a render synthesis manner to obtain the target image.
In the embodiment, the shape transfer model and the texture transfer model are set, the to-be-processed image is decoupled in two dimensions of the shape and the texture, and the shape feature and the texture feature are respectively subjected to transfer learning based on the set shape transfer model and the set texture transfer model, so that the transfer learning with independent distribution of the shape feature and the texture feature of the to-be-processed image is realized, the transfer learning precision of each dimension feature is improved, meanwhile, the transfer learning in the two dimensions of the shape feature and the texture feature is improved, the comprehensiveness of the transfer learning of the to-be-processed image is improved, and the transfer learning precision of the to-be-processed image and the quality of a target image are further improved.
On the basis of the above-described embodiment, the shape migration model and the texture migration model have the same network structure. Exemplarily, referring to fig. 2, fig. 2 is a schematic structural diagram of a shape migration model or a texture migration model in an embodiment of the present invention. For the shape migration model or the texture migration model, the model comprises a feature extraction module and at least one attribute migration module, and the attribute migration module generates a current output feature based on the output feature of the feature extraction module or the output feature of the previous attribute migration module, the current attribute feature and the target attribute feature and inputs the current output feature to the next attribute migration module. It should be noted that fig. 2 is only an exemplary diagram, and the number of the attribute migration modules in fig. 2 may be 3, or may be another number, and may be set according to a user requirement. Optionally, the feature extraction module and the attribute migration module may include at least one convolutional layer connected in sequence, or include a preset number of convolutional layers and other network blocks. The input feature may be a shape feature or a texture feature, and in some embodiments, the output feature of the last attribute migration module may be determined as a shape feature or a texture feature corresponding to the target attribute, or a rolling block or a full connection layer may be connected to the structure shown in fig. 2, which is not limited to this.
On the basis of the foregoing embodiment, the structures of the attribute migration modules may be the same, for example, refer to fig. 3, and fig. 3 is a schematic structural diagram of an attribute migration module according to an embodiment of the present invention. Any of the attribute migration modules includes: a first convolutional layer, a first style migration network, a second style migration network and a second convolutional layer; the first convolution layer is used for performing convolution operation on input features and inputting the convolution features to the first style migration network and the second style migration network respectively; the first style migration network performs migration operation on the current attribute feature and the convolution feature to generate a first migration feature, and the second style migration network performs migration operation on the target attribute feature and the convolution feature to generate a second migration feature; and the second convolutional layer performs dimensionality reduction on the spliced convolutional characteristic, the first migration characteristic and the second migration characteristic to obtain a current output characteristic, so that the dimensionality of the current output characteristic is consistent with the dimensionality of the input characteristic.
In this embodiment, the first and second style migration networks may be AdaIN (Adaptive Instance Normalization) networks. The AdaIN is used for respectively processing the current attribute feature and the target attribute feature to obtain a first migration feature and a second migration feature, further, the second convolution layer is used for carrying out convolution processing after carrying out splicing dimension reduction processing on the convolution feature, the first migration feature and the second migration feature, and in the migration learning process, the exclusive attribute feature and the target attribute feature of the image to be processed are fused, so that the problem that the image obtained by migration learning is uniform in length is solved, the exclusive feature of the image to be processed is highlighted, and the quality of the target image is improved.
On the basis of the above embodiment, a first parameter conversion module and a second parameter conversion module are respectively provided for the shape migration model and the texture migration model, the first parameter conversion module is configured to convert the current attribute feature into a first style migration parameter and output the first style migration parameter to the first style migration network, and the second parameter conversion module is configured to convert the target attribute feature into a second style migration parameter and input the second style migration parameter to the second style migration network. In some embodiments, the first parameter conversion module and the second parameter conversion module may be a full connection layer, and taking the first-style migration network and the second-style migration network as AdaIN networks as an example, the first-style migration parameter and the second-style migration parameter may be affine transformation parameters β and γ required by the AdaIN networks. In other embodiments, the first style migration parameter and the second style migration parameter may be set according to network types of the first style migration network and the second style migration network.
On the basis of the above embodiment, the training method of the shape migration model is to obtain a sample image and a target image corresponding to a target attribute of the sample image, respectively extract shape features of the sample image and the target image based on a shape feature extraction model, input the shape features, current attribute features, and target attribute features of the sample image into the shape migration model to be trained to obtain training shape features, determine a loss function based on the training shape features and the shape features of the target image, and adjust network parameters of the shape migration model to be trained based on the loss function. And (4) iteratively executing the training process until a training condition is met, such as the number of iterations is met or the precision requirement is met, and determining that the training of the shape migration model is finished.
Similarly, the training method of the texture migration model comprises the steps of obtaining a sample image and a target image corresponding to the target attribute of the sample image, extracting the texture features of the sample image and the target image respectively based on a texture feature extraction model, inputting the current attribute features and the target attribute features of the texture features of the sample image into the texture migration model to be trained to obtain training texture features, determining a loss function based on the texture features of the texture migration model and the target image, and adjusting network parameters of the texture migration model to be trained based on the loss function. And (4) iteratively executing the training process until a training condition is met, such as the iteration number is met or the precision requirement is met, and determining that the training of the texture migration model is finished.
The texture migration model and the shape migration model provided by this embodiment respectively perform migration learning on the texture features and the shape features, and input the current attribute features and the target attribute features of the to-be-processed image during the migration learning, so that the proprietary attribute features and the target attribute features of the to-be-processed image are fused, the problem that the images obtained by the migration learning are uniform is avoided, the proprietary features of the to-be-processed image are highlighted, and the target image quality is improved.
Example two
Fig. 4 is a schematic flowchart of an image processing method according to a second embodiment of the present invention, which is refined on the basis of the second embodiment, and optionally, the image to be processed is a face image, and the attribute includes an age attribute. The method specifically comprises the following steps:
s210, obtaining an initial image, and performing face segmentation on the initial image to obtain a face image and a background image in the initial image.
S220, extracting the current attribute feature, the shape feature and the texture feature of the face image and the target attribute feature of the face image.
And S230, inputting the current attribute feature, the shape feature and the target attribute feature into a pre-trained shape migration model, and generating a shape feature corresponding to the target attribute.
S240, inputting the current attribute feature, the texture feature and the target attribute feature into a pre-trained texture migration model, and generating the texture feature corresponding to the target attribute.
S250, inputting the current attribute feature, the texture feature and the target attribute feature into a pre-trained texture migration model, and generating the texture feature corresponding to the target attribute.
And S260, synthesizing the target image and the background image.
In this embodiment, the age attribute of the face image is subjected to the transfer learning operation, after the initial image is obtained, the face region in the initial image is identified, the identified face region is segmented to obtain the face image, the transfer learning operation of the age attribute is performed on the face image, and interference and invalidation processing of the background image in the transfer learning operation process are avoided.
And correspondingly, the current attribute feature is the current age feature, and the target attribute feature is the target age feature, wherein the current age feature is obtained by processing the face image to be processed based on the age estimation model. The target age characteristic is set according to a target age of the migration learning, illustratively, the target age is 70-79, accordingly, in a one-hot vector of length n, data of age group 70-79 is set to 1, and others are set to 0. The target age may be any desired age, and may be, for example, the age of a person smaller than the face image or the age of a person larger than the face image. It should be noted that the age group in the target age characteristic may be set according to a requirement, and is not limited to this.
The method comprises the steps of carrying out shape migration learning on a face image based on a preset shape migration model to obtain shape features corresponding to a target age, carrying out texture migration learning on the face image based on a preset texture migration model to obtain texture features corresponding to the target age, and carrying out feature fusion on the texture features corresponding to the target age and the shape features to obtain the target face image of the target age.
And synthesizing the target face image and the background image to obtain a synthesized image comprising the background image and the target face image. In some embodiments, the background may be synthesized with the target face image based on a refine model. It should be noted that, since the shape feature of the face image changes during the migration learning process, for example, the shape feature may change to be large or small, and accordingly, there is a synthesis difference between the target face image and the background image, which results in a poor quality of the synthesized image.
Optionally, the synthesizing the target image and the background image includes: adjusting the background image based on the target image; and merging the target image and the adjusted background image. Specifically, the shape of the background image is adjusted according to the shape characteristics of the target image (i.e., the target face image), so that the adjusted shape of the background image is adapted to the target face image, and the target image and the adjusted background image are combined to obtain a high-quality composite image.
According to the technical scheme, the current age characteristics, the shape characteristics and the texture characteristics of the face image are extracted, the characteristics of the shape characteristics and the texture characteristics are migrated and learned on two dimensions of the shape characteristics and the texture characteristics through the preset shape migration model and the preset texture migration model, the accuracy of migrating and learning of the face image on the two dimensions of the shape characteristics and the texture characteristics is improved, meanwhile, the humanized characteristics of the face image are considered in the migration and learning process through the migration and learning of the current age characteristics and the target age characteristics, the difference of the target face image is improved, the uniform migration and learning is avoided, and the quality of the target face image is improved.
On the basis of the foregoing embodiments, a preferred example of an image processing method is further provided, and exemplarily, referring to fig. 5 and fig. 6, fig. 5 is a schematic structural diagram of a texture migration model provided in an embodiment of the present invention, and fig. 6 is a schematic structural diagram of an attribute migration module provided in an embodiment of the present invention.
The input image in fig. 5 and 6 is a face image to be processed, the input image is processed based on an age estimation model to obtain a current age feature, a shape feature uv map of the input image is extracted based on a preset shape feature extraction model, a texture feature uv texture map of the input image is extracted based on a preset texture feature extraction model, and the texture migration model in fig. 5 has input information of the current age feature, the texture feature uv texture map and a target age feature, the target age feature is a one-hot vector with a length of n, and output information of the texture migration model is a texture feature uv texture map of an output image. Similarly, the structure of the shape migration model may be the same as that of the texture migration model, the input information of the shape migration model is the shape feature uv map of the face image, the current age feature and the target age feature, and the output information thereof is the shape feature uv map of the output image. And synthesizing the converted face image based on the shape feature uv map of the output image and the texture feature uv texture map of the output image by a render method. And synthesizing information such as a background and the like through a refine model, and optimizing the face image generated in the third step to obtain a more vivid face image.
The texture migration model includes three attribute migration modules, and the structure of the attribute migration module is shown in fig. 6. Each attribute migration module in fig. 6 includes two convolutional layers (i.e., a first convolutional layer and a second convolutional layer), and two AdaIN networks (i.e., a first style migration network and a second style migration network). The input feature of the attribute migration module may be an output feature of the feature extraction module, or an output feature of the last attribute migration module.
EXAMPLE III
Fig. 7 is a schematic structural diagram of an image processing apparatus according to an embodiment of the present invention, where the apparatus includes:
an image obtaining module 310, configured to obtain an image to be processed.
A feature extraction module 320, configured to extract a current attribute feature, a shape feature, a texture feature, and a target attribute feature of the image to be processed;
a shape feature migration module 330, configured to input the current attribute feature, the shape feature, and the target attribute feature into a pre-trained shape migration model, and generate a shape feature corresponding to the target attribute;
the texture feature migration module 340 is configured to input the current attribute feature, the texture feature, and the target attribute feature into a pre-trained texture migration model, and generate a texture feature corresponding to the target attribute;
and a target image generation module 350, configured to perform feature fusion based on the texture feature and the shape feature corresponding to the target attribute, so as to obtain a target image corresponding to the target attribute.
On the basis of the above embodiment, the shape migration model and the texture migration model respectively include a feature extraction module and at least one attribute migration module, and the attribute migration module generates a current output feature based on an output feature of the feature extraction module or an output feature of a previous attribute migration module, the current attribute feature, and the target attribute feature, and inputs the current output feature to a next attribute migration module.
On the basis of the above embodiment, the attribute migration module includes: a first convolutional layer, a first style migration network, a second style migration network and a second convolutional layer;
the first convolution layer is used for performing convolution operation on input features and inputting the convolution features to the first style migration network and the second style migration network respectively;
the first style migration network performs migration operation on the current attribute feature and the convolution feature to generate a first migration feature, and the second style migration network performs migration operation on the target attribute feature and the convolution feature to generate a second migration feature;
and the second convolutional layer performs dimensionality reduction on the spliced convolution characteristic, the first migration characteristic and the second migration characteristic to obtain a current output characteristic.
On the basis of the above embodiment, the shape migration model and the texture migration model respectively include a first parameter conversion module and a second parameter conversion module, the first parameter conversion module is configured to convert the current attribute feature into a first style migration parameter and output the first style migration parameter to the first style migration network, and the second parameter conversion module is configured to convert the target attribute feature into a second style migration parameter and input the second style migration parameter to the second style migration network.
On the basis of the above embodiment, the image to be treated is a face image, and the attribute includes an age attribute.
On the basis of the above embodiment, the image acquisition module 310 is configured to:
acquiring an initial image, and performing face segmentation on the initial image to obtain a face image and a background image in the initial image;
on the basis of the above embodiment, the apparatus further includes:
and the image synthesis module is used for synthesizing the target image and the background image.
On the basis of the above embodiment, the image synthesis module is configured to:
adjusting the background image based on the target image;
and merging the target image and the adjusted background image.
The image processing device provided by the embodiment of the invention can execute the image processing method provided by any embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method.
Example four
Fig. 8 is a schematic structural diagram of an electronic device according to a fourth embodiment of the present invention. FIG. 8 illustrates a block diagram of an electronic device 12 suitable for use in implementing embodiments of the present invention. The electronic device 12 shown in fig. 8 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiment of the present invention. The device 12 is typically an electronic device that undertakes image classification functions.
As shown in FIG. 8, electronic device 12 is embodied in the form of a general purpose computing device. The components of electronic device 12 may include, but are not limited to: one or more processors 16, a memory device 28, and a bus 18 that connects the various system components (including the memory device 28 and the processors 16).
A program 36 having a set (at least one) of program modules 26 may be stored, for example, in storage 28, such program modules 26 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which examples or some combination thereof may include an implementation of a gateway environment. Program modules 26 generally perform the functions and/or methodologies of the described embodiments of the invention.
The processor 16 executes various functional applications and data processing, for example, implementing the image processing method provided by the above-described embodiment of the present invention, by executing programs stored in the storage device 28.
EXAMPLE five
Fifth embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the image processing method provided in the fifth embodiment of the present invention.
Of course, the computer program stored on the computer-readable storage medium provided by the embodiments of the present invention is not limited to the method operations described above, and may also execute the image processing method provided by any embodiment of the present invention.
Computer storage media for embodiments of the invention may employ any combination of one or more computer-readable media. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
A computer readable signal medium may include a propagated data signal with computer readable source code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Source code embodied on a computer-readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Computer source code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + +, or the like, as well as conventional procedural programming languages, such as the "C" programming language or similar programming languages. The source code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of gateway, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims (10)
1. An image processing method, comprising:
acquiring an image to be processed, and extracting current attribute features, shape features and texture features of the image to be processed and target attribute features of the image to be processed;
inputting the current attribute feature, the shape feature and the target attribute feature into a pre-trained shape migration model to generate a shape feature corresponding to the target attribute;
inputting the current attribute feature, the texture feature and the target attribute feature into a pre-trained texture migration model to generate a texture feature corresponding to the target attribute;
and performing feature fusion based on the texture features and the shape features corresponding to the target attributes to obtain a target image corresponding to the target attributes.
2. The method according to claim 1, wherein the shape migration model and the texture migration model respectively comprise a feature extraction module and at least one attribute migration module, and the attribute migration module generates a current output feature based on an output feature of the feature extraction module or an output feature of a previous attribute migration module, the current attribute feature and the target attribute feature, and inputs the current output feature to a next attribute migration module.
3. The method of claim 2, wherein the attribute migration module comprises: a first convolutional layer, a first style migration network, a second style migration network and a second convolutional layer;
the first convolution layer is used for performing convolution operation on input features and inputting the convolution features to the first style migration network and the second style migration network respectively;
the first style migration network performs migration operation on the current attribute feature and the convolution feature to generate a first migration feature, and the second style migration network performs migration operation on the target attribute feature and the convolution feature to generate a second migration feature;
and the second convolutional layer performs dimensionality reduction on the spliced convolution characteristic, the first migration characteristic and the second migration characteristic to obtain a current output characteristic.
4. The method according to claim 3, wherein a first parameter conversion module and a second parameter conversion module are respectively disposed in the shape migration model and the texture migration model, the first parameter conversion module is configured to convert the current attribute feature into a first style migration parameter and output the first style migration parameter to the first style migration network, and the second parameter conversion module is configured to convert the target attribute feature into a second style migration parameter and input the second style migration parameter to the second style migration network.
5. The method according to any one of claims 1 to 4, wherein the image to be processed is a face image, and the attribute includes an age attribute.
6. The method of claim 5, wherein the acquiring the image to be processed comprises:
acquiring an initial image, and performing face segmentation on the initial image to obtain a face image and a background image in the initial image;
after obtaining the target image corresponding to the target attribute, the method further includes:
and synthesizing the target image and the background image.
7. The method of claim 6, wherein the combining the target image and the background image comprises:
adjusting the background image based on the target image;
and merging the target image and the adjusted background image.
8. An image processing apparatus characterized by comprising:
the characteristic extraction module is used for acquiring an image to be processed and extracting the current attribute characteristic, the shape characteristic and the texture characteristic of the image to be processed and the target attribute characteristic of the image to be processed;
the shape feature migration module is used for inputting the current attribute feature, the shape feature and the target attribute feature into a pre-trained shape migration model to generate a shape feature corresponding to the target attribute;
the texture feature migration module is used for inputting the current attribute feature, the texture feature and the target attribute feature into a pre-trained texture migration model to generate a texture feature corresponding to the target attribute;
and the target image generation module is used for carrying out feature fusion on the basis of the texture features and the shape features corresponding to the target attributes to obtain a target image corresponding to the target attributes.
9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the image processing method according to any of claims 1-7 when executing the program.
10. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the image processing method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110231306.8A CN113780326A (en) | 2021-03-02 | 2021-03-02 | Image processing method and device, storage medium and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110231306.8A CN113780326A (en) | 2021-03-02 | 2021-03-02 | Image processing method and device, storage medium and electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113780326A true CN113780326A (en) | 2021-12-10 |
Family
ID=78835648
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110231306.8A Pending CN113780326A (en) | 2021-03-02 | 2021-03-02 | Image processing method and device, storage medium and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113780326A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115147526A (en) * | 2022-06-30 | 2022-10-04 | 北京百度网讯科技有限公司 | Method and device for training clothing generation model and method and device for generating clothing image |
| WO2023125374A1 (en) * | 2021-12-29 | 2023-07-06 | 北京字跳网络技术有限公司 | Image processing method and apparatus, electronic device, and storage medium |
| WO2023168667A1 (en) * | 2022-03-10 | 2023-09-14 | 深圳市大疆创新科技有限公司 | Image processing method and apparatus, neural network training method, and storage medium |
-
2021
- 2021-03-02 CN CN202110231306.8A patent/CN113780326A/en active Pending
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2023125374A1 (en) * | 2021-12-29 | 2023-07-06 | 北京字跳网络技术有限公司 | Image processing method and apparatus, electronic device, and storage medium |
| WO2023168667A1 (en) * | 2022-03-10 | 2023-09-14 | 深圳市大疆创新科技有限公司 | Image processing method and apparatus, neural network training method, and storage medium |
| CN115147526A (en) * | 2022-06-30 | 2022-10-04 | 北京百度网讯科技有限公司 | Method and device for training clothing generation model and method and device for generating clothing image |
| CN115147526B (en) * | 2022-06-30 | 2023-09-26 | 北京百度网讯科技有限公司 | Training of clothing generation model and method and device for generating clothing image |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10380996B2 (en) | Method and apparatus for correcting speech recognition result, device and computer-readable storage medium | |
| CN108509915B (en) | Method and device for generating face recognition model | |
| CN112699991A (en) | Method, electronic device, and computer-readable medium for accelerating information processing for neural network training | |
| CN113780326A (en) | Image processing method and device, storage medium and electronic equipment | |
| CN110009059B (en) | Method and apparatus for generating a model | |
| JP2022177232A (en) | Method for processing image, method for recognizing text, and device for recognizing text | |
| CN110516598B (en) | Method and apparatus for generating image | |
| CN111950570B (en) | Target image extraction method, neural network training method and device | |
| CN115861462B (en) | Training method and device for image generation model, electronic equipment and storage medium | |
| CN115937033A (en) | Image generation method and device and electronic equipment | |
| CN114529574A (en) | Image matting method and device based on image segmentation, computer equipment and medium | |
| CN116564355A (en) | Multi-mode emotion recognition method, system, equipment and medium based on self-attention mechanism fusion | |
| CN115828889A (en) | Text analysis method, emotion classification model, device, medium, terminal and product | |
| CN111815748B (en) | Animation processing method and device, storage medium and electronic equipment | |
| CN113902789A (en) | Image feature processing method, depth image generating method, depth image processing apparatus, depth image generating medium, and device | |
| CN117315758A (en) | Facial expression detection method and device, electronic equipment and storage medium | |
| CN111144374A (en) | Facial expression recognition method and device, storage medium and electronic equipment | |
| CN116402914A (en) | Method, device and product for determining stylized image generation model | |
| CN116363561A (en) | Time sequence action positioning method, device, equipment and storage medium | |
| CN117011156A (en) | Image processing method, device, equipment and storage medium | |
| CN113065585B (en) | Training method and device of image synthesis model and electronic equipment | |
| CN115457365A (en) | Model interpretation method and device, electronic equipment and storage medium | |
| CN115049546A (en) | Sample data processing method and device, electronic equipment and storage medium | |
| CN114926322A (en) | Image generation method and device, electronic equipment and storage medium | |
| CN114399708A (en) | Video motion migration deep learning system and method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |