CN114973382A - Human face replacement method, device, equipment and storage medium based on artificial intelligence - Google Patents

Human face replacement method, device, equipment and storage medium based on artificial intelligence Download PDF

Info

Publication number
CN114973382A
CN114973382A CN202210679353.3A CN202210679353A CN114973382A CN 114973382 A CN114973382 A CN 114973382A CN 202210679353 A CN202210679353 A CN 202210679353A CN 114973382 A CN114973382 A CN 114973382A
Authority
CN
China
Prior art keywords
face
image
replacement
unit
target
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202210679353.3A
Other languages
Chinese (zh)
Inventor
郑喜民
周成昊
舒畅
陈又新
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Ping An Technology Shenzhen Co Ltd
Original Assignee
Ping An Technology Shenzhen Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Ping An Technology Shenzhen Co Ltd filed Critical Ping An Technology Shenzhen Co Ltd
Priority to CN202210679353.3A priority Critical patent/CN114973382A/en
Publication of CN114973382A publication Critical patent/CN114973382A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16Human faces, e.g. facial parts, sketches or expressions
    • G06V40/168Feature extraction; Face representation
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00Computing arrangements based on biological models
    • G06N3/02Neural networks
    • G06N3/04Architecture, e.g. interconnection topology
    • G06N3/045Combinations of networks
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00Computing arrangements based on biological models
    • G06N3/02Neural networks
    • G06N3/08Learning methods
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00Arrangements for image or video recognition or understanding
    • G06V10/70Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/774Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00Arrangements for image or video recognition or understanding
    • G06V10/70Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/80Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00Arrangements for image or video recognition or understanding
    • G06V10/70Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16Human faces, e.g. facial parts, sketches or expressions
    • G06V40/161Detection; Localisation; Normalisation
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16Human faces, e.g. facial parts, sketches or expressions
    • G06V40/172Classification, e.g. identification

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Health & Medical Sciences (AREA)
  • General Physics & Mathematics (AREA)
  • General Health & Medical Sciences (AREA)
  • Evolutionary Computation (AREA)
  • Multimedia (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Software Systems (AREA)
  • Computing Systems (AREA)
  • Artificial Intelligence (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Human Computer Interaction (AREA)
  • Medical Informatics (AREA)
  • Databases & Information Systems (AREA)
  • Biomedical Technology (AREA)
  • Data Mining & Analysis (AREA)
  • Mathematical Physics (AREA)
  • Biophysics (AREA)
  • General Engineering & Computer Science (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Molecular Biology (AREA)
  • Computational Linguistics (AREA)
  • Image Processing (AREA)

Abstract

The application relates to the technical field of artificial intelligence, and discloses a human face replacement method, a human face replacement device, human face replacement equipment and a storage medium based on artificial intelligence, wherein the method comprises the following steps: acquiring a face image to be replaced and a target face identifier; carrying out face replacement according to the face image to be replaced, the target face identification and a preset face replacement model to obtain a high-resolution face image; wherein, the face replacement model sequentially comprises: the device comprises an encoding unit, a human face distinguishing unit, each decoding unit and a high-resolution processing unit. Therefore, the finally output face image has higher resolution, the face replacement model can replace any face, and the face judgment unit is connected after the encoding unit, so that the face judgment unit can supervise the encoding unit to extract the common characteristics of the face, the generalization capability of the face image is improved, the training time is short, and the loss function is easy to converge.

Description

Human face replacement method, device, equipment and storage medium based on artificial intelligence
Technical Field
The present application relates to the field of artificial intelligence technologies, and in particular, to a method, an apparatus, a device, and a storage medium for face replacement based on artificial intelligence.
Background
Face replacement technology, as the name implies, is to replace one face with another face in an image or video. The main method for face replacement at present is a GAN (generation countermeasure network) based face replacement method.
The general flow of the FSGAN (arbitrary image face interchange) face replacement method is as follows: inputting a picture A and a picture B, firstly calling a third-party model to obtain an Euler angle and key points of a face B in the picture B, then generating a face A 'with a posture angle consistent with that of the face B and the characteristics of the face A according to the Euler angle and the face A in the picture A by using a generation model Gr, then obtaining a face area Mask B of the face B by using a segmentation model S, then completing incomplete area or erasing redundant parts of the face A' by using a generation model Gc according to the face A 'and the Mask B to obtain the face A', and finally rendering the illumination skin color and the like of the face in the face A 'by using the generation model Gb according to the face A', the Mask B and the face B to make the result more natural and finally outputting a face changing result. It is clear that the whole method requires training four models. The defects are that the training time is too long and the loss function is not easy to converge due to 3 GANs, and the face edge blurring is easily generated because face changing is performed based on face generation.
Disclosure of Invention
The application mainly aims to provide a human face replacement method, a human face replacement device, human face replacement equipment and a storage medium based on artificial intelligence, and aims to solve the technical problems that in the human face replacement method based on GAN in the prior art, 3 GANs cause that the training time is too long, the loss function is not easy to converge, and the generation of human face edge blurring is easy to occur due to face changing which is based on human face generation.
In order to achieve the above object, the present application provides a face replacement method based on artificial intelligence, the method comprising:
acquiring a face image to be replaced and a target face identifier;
carrying out face replacement according to the face image to be replaced, the target face identification and a preset face replacement model to obtain a high-resolution face image;
wherein, the face replacement model sequentially comprises: the device comprises an encoding unit, a human face distinguishing unit, each decoding unit and a high-resolution processing unit.
Further, the step of performing face replacement according to the face image to be replaced, the target face identifier and a preset face replacement model to obtain a high-resolution face image includes:
inputting the face image to be replaced into the coding unit for feature extraction to obtain initial features;
inputting the initial features into the face discrimination unit to perform face discrimination to obtain face discrimination probability;
inputting the initial features and the face discrimination probability into a fusion unit of the face replacement model for fusion processing to obtain fusion features;
inputting the fusion features into the decoding unit corresponding to the target face identification for decoding to obtain a replaced face image;
and inputting the replaced face image into the high-resolution processing unit for high-resolution reconstruction to obtain the high-resolution face image.
Further, the step of inputting the initial feature and the face discrimination probability into a fusion unit of the face replacement model for fusion processing to obtain a fusion feature includes:
and inputting the initial features and the face discrimination probability into the fusion unit for point multiplication calculation to obtain the fusion features.
Further, before the step of obtaining the face image to be replaced and the target face identifier, the method further includes:
acquiring an image to be processed;
extracting a face image and a mask from the image to be processed to obtain a face image set and a mask set;
taking any one of the face images in the face image set as the face image to be replaced;
according to the face image to be replaced, a mask is obtained from the mask set and is used as a target mask;
after the step of performing face replacement according to the face image to be replaced, the target face identifier and a preset face replacement model to obtain a high-resolution face image, the method further comprises the following steps:
and replacing the face image to be replaced in the image to be processed according to the target mask and the high-resolution face image to obtain a target image.
Further, the step of taking any one of the facial images in the facial image set as the facial image to be replaced includes:
taking any one of the facial images in the facial image set as an image to be analyzed;
carrying out front face registration on the image to be analyzed to obtain the face image to be replaced and a face alignment matrix;
the step of replacing the face image to be replaced in the image to be processed according to the target mask and the high-resolution face image to obtain a target image comprises the following steps:
converting the high-resolution face image according to the face alignment matrix to obtain a converted face image;
and carrying out replacement processing on the face image to be replaced in the image to be processed according to the target mask and the converted face image to obtain the target image.
Further, before the step of performing face replacement according to the face image to be replaced, the target face identifier and a preset face replacement model to obtain a high-resolution face image, the method further includes:
acquiring each training sample corresponding to the target face identification;
training a preset initial model by adopting each training sample, and taking the initial model after training as the face replacement model, wherein the initial model sequentially comprises: the device comprises an encoding initial unit, a human face distinguishing initial unit, a decoding initial unit and a high-resolution initial processing unit.
Further, each of the training samples comprises: the face image label is a face image corresponding to the target face identification;
the step of training a preset initial model by adopting each training sample and taking the initial model after training as the face replacement model comprises the following steps:
taking any one of the training samples as a target training sample;
inputting the face image samples in the target training samples into the initial model for face replacement to obtain a face replacement result;
performing loss value according to the face replacement result and the face image label in the target training sample;
updating network parameters of the encoding initial unit of the initial model, the decoding initial unit corresponding to the target face identification and the high-resolution initial processing unit according to the loss value;
repeatedly executing the step of taking any one of the training samples as a target training sample until a preset training end condition is reached;
and taking the initial model reaching the training end condition as the face replacement model.
The application also provides a face replacement device based on artificial intelligence, the device includes:
the data acquisition module is used for acquiring a face image to be replaced and a target face identifier;
the face replacement module is used for carrying out face replacement according to the face image to be replaced, the target face identification and a preset face replacement model to obtain a high-resolution face image; wherein, the face replacement model sequentially comprises: the device comprises an encoding unit, a human face distinguishing unit, each decoding unit and a high-resolution processing unit.
The present application further proposes a computer device comprising a memory and a processor, the memory storing a computer program, the processor implementing the steps of any of the above methods when executing the computer program.
The present application also proposes a computer-readable storage medium having stored thereon a computer program which, when being executed by a processor, carries out the steps of the method of any of the above-mentioned.
The method, the device, the equipment and the storage medium for replacing the human face based on the artificial intelligence are characterized in that the method comprises the steps of obtaining a human face image to be replaced and a target human face identifier; carrying out face replacement according to the face image to be replaced, the target face identification and a preset face replacement model to obtain a high-resolution face image; wherein, the face replacement model sequentially comprises: the device comprises an encoding unit, a human face distinguishing unit, each decoding unit and a high-resolution processing unit. The high-resolution processing unit is added behind the decoding unit, so that the finally output face image has higher resolution, and the technical problem that face change is performed based on face generation, so that face edge blurring is easily generated is solved; because each decoding unit shares one coding unit, the coding unit can learn the characteristics of the faces of a great number of different people, and the face replacement model can replace any face; the face distinguishing unit is connected behind the coding unit, so that the face distinguishing unit can supervise the coding unit to extract the common features of the face, and the generalization capability of the application is improved; by adopting the coding unit, the face distinguishing unit, each decoding unit and the high-resolution processing unit, a plurality of GANs do not need to be trained, and compared with the existing face replacing method based on GANs, the method has the advantages of short training time and easy convergence of loss functions.
Drawings
Fig. 1 is a schematic flow chart of a face replacement method based on artificial intelligence according to an embodiment of the present application;
fig. 2 is a schematic block diagram of a structure of a human face replacement device based on artificial intelligence according to an embodiment of the present application;
fig. 3 is a block diagram illustrating a structure of a computer device according to an embodiment of the present application.
The implementation, functional features and advantages of the objectives of the present application will be further explained with reference to the accompanying drawings.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.
Referring to fig. 1, an embodiment of the present application provides a face replacement method based on artificial intelligence, where the method includes:
s1: acquiring a face image to be replaced and a target face identifier;
s2: carrying out face replacement according to the face image to be replaced, the target face identification and a preset face replacement model to obtain a high-resolution face image;
wherein, the face replacement model sequentially comprises: the device comprises an encoding unit, a human face distinguishing unit, each decoding unit and a high-resolution processing unit.
In the embodiment, the high-resolution processing unit is added behind the decoding unit, so that the finally output face image has higher resolution, and the technical problem that face change is performed based on face generation, so that face edge blurring is easily generated is solved; because each decoding unit shares one coding unit, the coding unit can learn the characteristics of the faces of a great number of different people, and the face replacement model can replace any face; the face distinguishing unit is connected behind the coding unit, so that the face distinguishing unit can supervise the coding unit to extract the common features of the face, and the generalization capability of the application is improved; by adopting the coding unit, the face distinguishing unit, each decoding unit and the high-resolution processing unit, a plurality of GANs do not need to be trained, and compared with the existing face replacing method based on GANs, the method has the advantages of short training time and easy convergence of loss functions.
For S1, the face image to be replaced and the target face identifier input by the user may be acquired, the face image to be replaced and the target face identifier may also be acquired from the database, and the face image to be replaced and the target face identifier may also be acquired from a third-party application.
The face image to be replaced is the face image needing face replacement. The face image is an image including a face.
The target face identification is the face identification of the face to be replaced. The face identification may be data that uniquely identifies a face, such as a face name, a face ID, and the like.
That is to say, the present application intends to replace the face in the face image to be replaced with the face corresponding to the target face identifier.
For step S2, the face image to be replaced is input into the face replacement model for face replacement, and when face replacement is performed, the decoding units corresponding to the target face identifiers are used for each decoding unit in the face replacement model; and taking the face image output by the face replacement model as the high-resolution face image.
It can be understood that if the face replacement model needs to be replaced with a new face, a decoding unit needs to be trained for the new face.
The coding unit is used for extracting features. The coding unit is derived based on encoder training.
The face judging unit is used for judging the probability of whether the face is the face or not. The face discrimination unit is a unit obtained based on neural network training.
And the decoding unit is used for restoring the face image. The decoding unit is derived based on decoder training.
And the high-resolution processing unit is used for reconstructing the image at high resolution. The high-resolution processing unit is a unit obtained based on SRCNN (high-resolution reconstruction convolutional neural network) training.
The network structure of the SRCNN is very simple, and only three convolutional layers are used. Firstly, a bicubic interpolation is used for magnifying the low-resolution image into a target size, then nonlinear mapping is fitted through a three-layer convolution network, and finally a high-resolution image result is output. The structure of the triple layer convolution represents three steps: extracting image blocks, performing feature representation, performing feature nonlinear mapping and finally reconstructing. The sizes of convolution kernels used by the three convolutional layers are divided into 9x9, 1x1 and 5x5, and the number of output features of the first two is 64 and 32 respectively.
It can be understood that the output of the coding unit is used as the input of the face discrimination unit, the input of the decoding unit is determined according to the output of the face discrimination unit and the output of the coding unit, only one decoding unit is needed for each face replacement, and the output of the decoding unit is used as the input of the high-resolution processing unit.
Optionally, the step of determining the input of the decoding unit according to the output of the face discrimination unit and the output of the encoding unit includes: judging whether the output of the face distinguishing unit is larger than a preset probability threshold value or not, if so, indicating that the face image to be replaced contains a face, and taking the output of the coding unit as a decoding unit; if not, the face image to be replaced does not contain the face, and face replacement is not needed at the moment, so that face image restoration and image reconstruction are not needed.
And the face in the high-resolution face image is the face corresponding to the target face identification.
In an embodiment, the step of performing face replacement according to the face image to be replaced, the target face identifier, and a preset face replacement model to obtain a high-resolution face image includes:
s21: inputting the face image to be replaced into the coding unit for feature extraction to obtain initial features;
s22: inputting the initial features into the face discrimination unit to perform face discrimination to obtain face discrimination probability;
s23: inputting the initial features and the face discrimination probability into a fusion unit of the face replacement model for fusion processing to obtain fusion features;
s24: inputting the fusion characteristics into the decoding unit corresponding to the target face identification for decoding to obtain a replaced face image;
s25: and inputting the replaced face image into the high-resolution processing unit for high-resolution reconstruction to obtain the high-resolution face image.
In the embodiment, the high-resolution processing unit is added behind the decoding unit, so that the finally output face image has higher resolution, and the technical problem that face change is performed based on face generation, so that face edge blurring is easily generated is solved; because each decoding unit shares one coding unit, the coding unit can learn the characteristics of the faces of a great number of different people, and the face replacement model can replace any face; the face distinguishing unit is connected behind the coding unit, so that the face distinguishing unit can supervise the coding unit to extract the common features of the face, and the generalization capability of the application is improved; by adopting the coding unit, the face distinguishing unit, each decoding unit and the high-resolution processing unit, a plurality of GANs do not need to be trained, and compared with the existing face replacing method based on GANs, the method has the advantages that the training time is short, and the loss function is easy to converge; and the initial features and the face discrimination probability are input into a fusion unit of the face replacement model for fusion processing and then input into a decoding unit, which is favorable for improving the accuracy of the final output face image.
And S21, inputting the face image to be replaced into the coding unit for feature extraction, and taking the obtained features as initial features. Thereby extracting the common characteristics of the human faces.
For S22, the initial features are input to the face discrimination unit to perform face discrimination, and the probability obtained by discrimination is taken as a face discrimination probability. Through the face discrimination unit, all images lost into the coding unit are required to have sufficiently similar codes, so that the coding unit extracts the information of a real face, and the purpose of extracting the common characteristics of the face by the supervision coding unit is realized.
And S23, inputting the initial features and the face discrimination probability into a fusion unit of the face replacement model for fusion processing, and taking data obtained by fusion processing as fusion features.
And S24, inputting the fusion features into the decoding unit corresponding to the target face identification for decoding, and taking the decoded image as a replaced face image.
And the decoding unit corresponding to the target face identification is used for decoding and restoring the individual characteristics of the face corresponding to the target face identification. Therefore, the face in the face image to be replaced is replaced by the face corresponding to the target face identification.
And S25, inputting the replaced face image into the high-resolution processing unit for high-resolution reconstruction, and taking the reconstructed image as the high-resolution face image. Thereby enabling the high-resolution face image to have higher resolution.
In an embodiment, the step of inputting the initial feature and the face discrimination probability into the fusion unit of the face replacement model to perform fusion processing to obtain a fusion feature includes:
s231: and inputting the initial features and the face discrimination probability into the fusion unit for point multiplication calculation to obtain the fusion features.
In the embodiment, the initial feature and the face discrimination probability are input into the fusion unit to perform point multiplication calculation to serve as the fusion feature, which is beneficial to improving the accuracy of the final output face image.
And corresponding to S231, inputting the initial feature and the face discrimination probability into the fusion unit for point multiplication calculation, and taking the feature obtained by the point multiplication calculation as the fusion feature.
In an embodiment, before the step of obtaining the face image to be replaced and the target face identifier, the method further includes:
s11: acquiring an image to be processed;
s12: extracting a face image and a mask from the image to be processed to obtain a face image set and a mask set;
s13: taking any one of the face images in the face image set as the face image to be replaced;
s14: according to the face image to be replaced, a mask is obtained from the mask set and is used as a target mask;
after the step of performing face replacement according to the face image to be replaced, the target face identifier and a preset face replacement model to obtain a high-resolution face image, the method further comprises the following steps:
s31: and replacing the face image to be replaced in the image to be processed according to the target mask and the high-resolution face image to obtain a target image.
In this embodiment, any one of the face images in the face image set is taken as the face image to be replaced, so that a basis is provided for replacing one face at a time; through a target mask, the high-resolution face image can be accurately replaced with the face image to be replaced in the image to be processed, so that automatic face replacement of each face in the image to be processed is realized.
For S11, the image to be processed input by the user may be obtained, the image to be processed may be obtained from a database, the image to be processed may be obtained from a third-party application, and a frame of image may be obtained from a video as the image to be processed.
The image to be processed is a digital image.
For step S12, each face image and the mask corresponding to each face image are extracted from the image to be processed, each extracted face image is used as a face image set, and each extracted mask is used as a mask set.
For S13, any one of the facial images in the facial image set is taken as the facial image to be replaced, so as to provide a basis for performing facial replacement of one facial image at a time.
For step S14, a mask corresponding to the face image to be replaced is obtained from the mask set, and the obtained mask is used as a target mask.
The size of the target mask is the same as the size of the image to be processed. The pixel value of each pixel in the target mask is 0 or 1, if the pixel value is 1, the pixel is the pixel of the face corresponding to the face image to be replaced, and if the pixel value is 1, the pixel is not the pixel of the face corresponding to the face image to be replaced.
For step S31, finding an image region corresponding to the face image to be replaced in the image to be processed according to the target mask, performing replacement processing on the found image region by using the high-resolution face image, and taking the high-resolution face image subjected to the replacement processing as the target image.
In an embodiment, the step of using any one of the facial images in the facial image set as the facial image to be replaced includes:
s131: taking any one of the facial images in the facial image set as an image to be analyzed;
s132: carrying out front face registration on the image to be analyzed to obtain the image of the human face to be replaced and a human face alignment matrix;
the step of performing replacement processing on the face image to be replaced in the image to be processed according to the target mask and the high-resolution face image to obtain a target image comprises the following steps:
s311: converting the high-resolution face image according to the face alignment matrix to obtain a converted face image;
s312: and carrying out replacement processing on the face image to be replaced in the image to be processed according to the target mask and the converted face image to obtain the target image.
In this embodiment, any one of the facial images in the facial image set is subjected to front face registration and then is used as the facial image to be replaced, so that a basis is provided for replacing the face of one facial image at a time; the face angle of the high-resolution face image is converted into a face angle corresponding to an image to be analyzed through the face alignment matrix, and the converted image can be accurately replaced with the face image to be replaced in the image to be processed, so that the face replacement of each face in the image to be processed is automatically carried out; the face replacement is carried out by adopting the front face image, so that the accuracy of the face replacement is improved.
For step S131, any one of the face images in the face image set is used as an image to be analyzed, so as to provide a basis for performing face replacement of one face image at a time.
And S132, performing front face registration on the image to be analyzed, taking the image to be analyzed after registration as the face image to be replaced, and taking a conversion matrix between the image to be analyzed and the face image to be replaced as a face alignment matrix. That is, the face image to be replaced is a front face image.
And S311, converting the high-resolution face image according to the face alignment matrix, wherein the face angle of the converted image is the same as the face angle of the image to be analyzed, and the converted image is used as the converted face image.
For step S312, an image region corresponding to the face image to be replaced in the image to be processed is found according to the target mask, the found image region is replaced by using the converted face image, and the converted face image after the replacement is used as the target image.
In an embodiment, before the step of performing face replacement according to the face image to be replaced, the target face identifier, and a preset face replacement model to obtain a high-resolution face image, the method further includes:
s021: acquiring each training sample corresponding to the target face identification;
s022: training a preset initial model by adopting each training sample, and taking the initial model after training as the face replacement model, wherein the initial model sequentially comprises: the device comprises an encoding initial unit, a human face distinguishing initial unit, a decoding initial unit and a high-resolution initial processing unit.
In the embodiment, each training sample corresponding to the target face identification is adopted to train an encoding initial unit, a face discrimination initial unit, a decoding initial unit and a high-resolution initial processing unit to obtain the face replacement model, and the high-resolution initial processing unit is added after the decoding initial unit, so that a finally output face image has higher resolution, and the technical problem that face edge blurring is easily generated because face replacement is performed based on face generation is solved; the face distinguishing unit is connected after the coding initial unit, so that the face distinguishing initial unit can supervise the coding initial unit to extract the common features of the face, and the generalization capability of the application is improved; by adopting the encoding initial unit, the face discrimination initial element, the decoding initial unit and the high-resolution processing unit, a plurality of GANs are not required to be trained, and compared with the existing face replacement method based on GANs, the method has the advantages of short training time and easy convergence of loss functions.
For S021, each training sample corresponding to the target face identifier input by the user may be obtained, each training sample corresponding to the target face identifier may also be obtained from a database, each training sample corresponding to the target face identifier may also be obtained from a third-party application, and a frame of image may also be obtained from a video as each training sample corresponding to the target face identifier.
The face image labels of the training samples corresponding to the target face identification are face images corresponding to the target face identification.
And S022, training a preset initial model by adopting each training sample corresponding to the target face identification, and taking the initial model after training as the face replacement model, so as to obtain the face replacement model reaching the expected performance.
The encoding initial unit adopts an encoder.
The face identification initial unit is a unit obtained based on a neural network.
The decoding initial unit adopts a decoder.
The high-resolution initial processing unit is a unit obtained based on SRCNN (high-resolution reconstruction convolutional neural network).
It can be understood that, when each training sample corresponding to the target face identifier is used to train a preset initial model, the decoding initial unit used is used as the decoding unit corresponding to the target face identifier in the face replacement model after the training is finished.
In one embodiment, each of the training samples described above includes: the face image label is a face image corresponding to the target face identification;
the step of training a preset initial model by adopting each training sample and taking the initial model after training as the face replacement model comprises the following steps:
s0221: taking any one of the training samples as a target training sample;
s0222: inputting the face image samples in the target training samples into the initial model for face replacement to obtain a face replacement result;
s0223: performing loss value according to the face replacement result and the face image label in the target training sample;
s0224: updating network parameters of the encoding initial unit of the initial model, the decoding initial unit corresponding to the target face identification and the high-resolution initial processing unit according to the loss value;
s0225: repeatedly executing the step of taking any one of the training samples as a target training sample until a preset training end condition is reached;
s0226: and taking the initial model reaching the training end condition as the face replacement model.
In this embodiment, the network parameters of the encoding initial unit of the initial model, the decoding initial unit corresponding to the target face identifier, and the high-resolution initial processing unit are updated according to the loss value, that is, the network parameters of the face identification initial unit do not need to be updated, so that the trained network parameters are reduced; moreover, the high-resolution initial processing unit needs to be updated, so that the resolution of the output image can be improved.
And S0222, inputting the face image samples in the target training samples into the initial model for face replacement, and taking images obtained by replacement as face replacement results.
And S0223, inputting the face replacement result and the face image label in the target training sample into a preset loss function to calculate a loss value.
Optionally, the preset loss function is a cross entropy loss function.
For S0224, network parameters of the coding initialization unit of the initialization model, the decoding initialization unit corresponding to the target face identifier, and the high-resolution initialization processing unit are updated according to the loss value, so that the network parameters of the face identification initialization unit do not need to be updated.
For S0225, repeatedly performing the step of taking any one of the training samples as a target training sample, that is, repeatedly performing steps S0221 to S0225 until a preset training end condition is reached; when the preset training end condition is reached, the repeated execution of steps S0221 to S0225 is stopped, and step S0226 is started.
Optionally, the preset training end condition is that the loss value of the initial model converges to a preset value.
For S0226, the initial model reaching the training end condition is a model whose performance meets an expected requirement, and therefore, the initial model reaching the training end condition is directly used as the face replacement model.
Referring to fig. 2, the present application further provides a face replacement device based on artificial intelligence, the device including:
a data obtaining module 100, configured to obtain a face image to be replaced and a target face identifier;
the face replacement module 200 is configured to perform face replacement according to the face image to be replaced, the target face identifier and a preset face replacement model to obtain a high-resolution face image; wherein, the face replacement model sequentially comprises: the device comprises an encoding unit, a human face distinguishing unit, each decoding unit and a high-resolution processing unit.
In the embodiment, the high-resolution processing unit is added behind the decoding unit, so that the finally output face image has higher resolution, and the technical problem that face change is performed based on face generation, so that face edge blurring is easily generated is solved; because each decoding unit shares one coding unit, the coding unit can learn the characteristics of the faces of a great number of different people, and the face replacement model can replace any face; the face distinguishing unit is connected behind the coding unit, so that the face distinguishing unit can supervise the coding unit to extract the common features of the face, and the generalization capability of the application is improved; by adopting the coding unit, the human face distinguishing unit, each decoding unit and the high-resolution processing unit, a plurality of GANs do not need to be trained, and compared with the existing human face replacing method based on GAN, the method has the advantages of short training time and easy convergence of loss functions.
In one embodiment, the face replacement module 200 includes:
an initial feature determination submodule, configured to input the face image to be replaced into the encoding unit for feature extraction, so as to obtain an initial feature;
a face discrimination probability determination submodule, configured to input the initial feature to the face discrimination unit to perform face discrimination to obtain a face discrimination probability;
a fusion feature determination submodule, configured to input the initial feature and the face discrimination probability into a fusion unit of the face replacement model for fusion processing, so as to obtain a fusion feature;
the replaced face image sub-module is used for inputting the fusion characteristics into the decoding unit corresponding to the target face identification for decoding to obtain a replaced face image;
and the high-resolution face image determining submodule is used for inputting the replaced face image into the high-resolution processing unit for high-resolution reconstruction to obtain the high-resolution face image.
In one embodiment, the fused feature determination sub-module includes: and inputting the initial features and the face discrimination probability into the fusion unit to perform point multiplication calculation to obtain the fusion features.
In one embodiment, the above apparatus further comprises:
the system comprises a to-be-replaced face image determining module, a to-be-replaced face image determining module and a to-be-replaced face image determining module, wherein the to-be-replaced face image determining module is used for acquiring a to-be-processed image, extracting a face image and a mask from the to-be-processed image to obtain a face image set and a mask set, taking any one face image in the face image set as the to-be-replaced face image, and acquiring the mask from the mask set according to the to-be-replaced face image to serve as a target mask;
and the target image determining module is used for replacing the face image to be replaced in the image to be processed according to the target mask and the high-resolution face image to obtain a target image.
In one embodiment, the module for determining a face image to be replaced includes:
the front face registration submodule is used for taking any one of the face images in the face image set as an image to be analyzed, and carrying out front face registration on the image to be analyzed to obtain the face image to be replaced and a face alignment matrix;
the target image determination module includes:
the conversion submodule is used for converting the high-resolution face image according to the face alignment matrix to obtain a converted face image;
and the replacement processing submodule is used for performing replacement processing on the face image to be replaced in the image to be processed according to the target mask and the converted face image to obtain the target image.
In one embodiment, the above apparatus further comprises:
the model training module is used for acquiring training samples corresponding to the target face identification, training a preset initial model by adopting the training samples, and taking the initial model after training as the face replacement model, wherein the initial model sequentially comprises: the device comprises an encoding initial unit, a human face distinguishing initial unit, a decoding initial unit and a high-resolution initial processing unit.
In one embodiment, each of the training samples described above includes: the face image label is a face image corresponding to the target face identification;
the model training module comprises:
a training submodule, configured to use any one of the training samples as a target training sample, input the face image sample in the target training sample into the initial model to perform face replacement, obtain a face replacement result, perform a loss value according to the face replacement result and a face image label in the target training sample, and update network parameters of the encoding initial unit, the decoding initial unit corresponding to the target face identifier, and the high-resolution initial processing unit of the initial model according to the loss value;
the cyclic control sub-module is used for repeatedly executing the step of taking any one of the training samples as a target training sample until a preset training end condition is reached;
and the face replacement model determining submodule is used for taking the initial model reaching the training end condition as the face replacement model.
Referring to fig. 3, an embodiment of the present application further provides a computer device, where the computer device may be a server, and an internal structure of the computer device may be as shown in fig. 3. The computer device includes a processor, a memory, a network interface, and a database connected by a system bus. Wherein the computer designed processor is used to provide computational and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used for storing data such as a human face replacement method based on artificial intelligence. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement an artificial intelligence based face replacement method. The artificial intelligence based face replacement method comprises the following steps: acquiring a face image to be replaced and a target face identifier; carrying out face replacement according to the face image to be replaced, the target face identification and a preset face replacement model to obtain a high-resolution face image; wherein, the face replacement model sequentially comprises: the device comprises an encoding unit, a human face distinguishing unit, each decoding unit and a high-resolution processing unit.
In the embodiment, the high-resolution processing unit is added behind the decoding unit, so that the finally output face image has higher resolution, and the technical problem that face change is performed based on face generation, so that face edge blurring is easily generated is solved; because each decoding unit shares one coding unit, the coding units can learn the characteristics of the faces of a great number of different people, and the face replacement model can replace any face; the face distinguishing unit is connected behind the coding unit, so that the face distinguishing unit can supervise the coding unit to extract the common features of the face, and the generalization capability of the application is improved; by adopting the coding unit, the human face distinguishing unit, each decoding unit and the high-resolution processing unit, a plurality of GANs do not need to be trained, and compared with the existing human face replacing method based on GAN, the method has the advantages of short training time and easy convergence of loss functions.
An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the method for face replacement based on artificial intelligence is implemented, including the steps of: acquiring a face image to be replaced and a target face identifier; carrying out face replacement according to the face image to be replaced, the target face identification and a preset face replacement model to obtain a high-resolution face image; wherein, the face replacement model sequentially comprises: the device comprises an encoding unit, a human face distinguishing unit, each decoding unit and a high-resolution processing unit.
According to the human face replacement method based on artificial intelligence, the high-resolution processing unit is added behind the decoding unit, so that the finally output human face image has higher resolution, and the technical problem that the generated human face edge is easy to blur because face changing is performed based on human face generation is solved; because each decoding unit shares one coding unit, the coding unit can learn the characteristics of the faces of a great number of different people, and the face replacement model can replace any face; the face distinguishing unit is connected behind the coding unit, so that the face distinguishing unit can supervise the coding unit to extract the common features of the face, and the generalization capability of the application is improved; by adopting the coding unit, the face distinguishing unit, each decoding unit and the high-resolution processing unit, a plurality of GANs do not need to be trained, and compared with the existing face replacing method based on GANs, the method has the advantages of short training time and easy convergence of loss functions.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium provided herein and used in the examples may include non-volatile and/or volatile memory. Non-volatile memory can include read-only memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), double-rate SDRAM (SSRSDRAM), Enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), Rambus Direct RAM (RDRAM), direct bus dynamic RAM (DRDRAM), and bus dynamic RAM (RDRAM).
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, apparatus, article, or method. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in the process, apparatus, article or method that comprises the element.
The above description is only a preferred embodiment of the present application, and not intended to limit the scope of the present application, and all the equivalent structures or equivalent processes that can be directly or indirectly applied to other related technical fields by using the contents of the specification and the drawings of the present application are also included in the scope of the present application.

Claims (10)

1. A face replacement method based on artificial intelligence is characterized by comprising the following steps:
acquiring a face image to be replaced and a target face identifier;
carrying out face replacement according to the face image to be replaced, the target face identification and a preset face replacement model to obtain a high-resolution face image;
wherein, the face replacement model sequentially comprises: the device comprises an encoding unit, a human face distinguishing unit, each decoding unit and a high-resolution processing unit.
2. The artificial intelligence based face replacement method according to claim 1, wherein the step of performing face replacement according to the face image to be replaced, the target face identifier and a preset face replacement model to obtain a high-resolution face image comprises:
inputting the face image to be replaced into the coding unit for feature extraction to obtain initial features;
inputting the initial features into the face discrimination unit to perform face discrimination to obtain face discrimination probability;
inputting the initial features and the face discrimination probability into a fusion unit of the face replacement model for fusion processing to obtain fusion features;
inputting the fusion characteristics into the decoding unit corresponding to the target face identification for decoding to obtain a replaced face image;
and inputting the replaced face image into the high-resolution processing unit for high-resolution reconstruction to obtain the high-resolution face image.
3. The artificial intelligence based face replacement method according to claim 2, wherein the step of inputting the initial feature and the face discrimination probability into the fusion unit of the face replacement model for fusion processing to obtain a fusion feature comprises:
and inputting the initial features and the face discrimination probability into the fusion unit for point multiplication calculation to obtain the fusion features.
4. The artificial intelligence based face replacement method according to claim 1, wherein the step of obtaining the face image to be replaced and the target face identifier is preceded by the steps of:
acquiring an image to be processed;
extracting a face image and a mask from the image to be processed to obtain a face image set and a mask set;
taking any one of the face images in the face image set as the face image to be replaced;
according to the face image to be replaced, a mask is obtained from the mask set and is used as a target mask;
after the step of performing face replacement according to the face image to be replaced, the target face identifier and a preset face replacement model to obtain a high-resolution face image, the method further comprises the following steps:
and replacing the face image to be replaced in the image to be processed according to the target mask and the high-resolution face image to obtain a target image.
5. The artificial intelligence based face replacement method according to claim 4, wherein the step of using any one of the face images in the face image set as the face image to be replaced comprises:
taking any one of the facial images in the facial image set as an image to be analyzed;
carrying out front face registration on the image to be analyzed to obtain the face image to be replaced and a face alignment matrix;
the step of performing replacement processing on the face image to be replaced in the image to be processed according to the target mask and the high-resolution face image to obtain a target image comprises the following steps:
converting the high-resolution face image according to the face alignment matrix to obtain a converted face image;
and carrying out replacement processing on the face image to be replaced in the image to be processed according to the target mask and the converted face image to obtain the target image.
6. The artificial intelligence based face replacement method according to claim 1, wherein before the step of performing face replacement according to the face image to be replaced, the target face identifier and a preset face replacement model to obtain a high-resolution face image, the method further comprises:
acquiring each training sample corresponding to the target face identification;
training a preset initial model by adopting each training sample, and taking the initial model after training as the face replacement model, wherein the initial model sequentially comprises: the device comprises an encoding initial unit, a human face distinguishing initial unit, a decoding initial unit and a high-resolution initial processing unit.
7. The artificial intelligence based face replacement method of claim 6, wherein each of the training samples comprises: the face image label is a face image corresponding to the target face identification;
the step of training a preset initial model by adopting each training sample and taking the initial model after training as the face replacement model comprises the following steps:
taking any one of the training samples as a target training sample;
inputting the face image samples in the target training samples into the initial model for face replacement to obtain a face replacement result;
performing loss value according to the face replacement result and the face image label in the target training sample;
updating network parameters of the encoding initial unit of the initial model, the decoding initial unit corresponding to the target face identification and the high-resolution initial processing unit according to the loss value;
repeatedly executing the step of taking any one of the training samples as a target training sample until a preset training end condition is reached;
and taking the initial model reaching the training end condition as the face replacement model.
8. An artificial intelligence based face replacement apparatus, the apparatus comprising:
the data acquisition module is used for acquiring a face image to be replaced and a target face identifier;
the face replacement module is used for carrying out face replacement according to the face image to be replaced, the target face identification and a preset face replacement model to obtain a high-resolution face image; wherein, the face replacement model sequentially comprises: the device comprises an encoding unit, a human face distinguishing unit, each decoding unit and a high-resolution processing unit.
9. A computer device comprising a memory and a processor, the memory storing a computer program, wherein the processor implements the steps of the method of any one of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the method according to any one of claims 1 to 7.
CN202210679353.3A 2022-06-15 2022-06-15 Human face replacement method, device, equipment and storage medium based on artificial intelligence Pending CN114973382A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210679353.3A CN114973382A (en) 2022-06-15 2022-06-15 Human face replacement method, device, equipment and storage medium based on artificial intelligence

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210679353.3A CN114973382A (en) 2022-06-15 2022-06-15 Human face replacement method, device, equipment and storage medium based on artificial intelligence

Publications (1)

Publication Number Publication Date
CN114973382A true CN114973382A (en) 2022-08-30

Family

ID=82963064

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210679353.3A Pending CN114973382A (en) 2022-06-15 2022-06-15 Human face replacement method, device, equipment and storage medium based on artificial intelligence

Country Status (1)

Country Link
CN (1) CN114973382A (en)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20190171908A1 (en) * 2017-12-01 2019-06-06 The University Of Chicago Image Transformation with a Hybrid Autoencoder and Generative Adversarial Network Machine Learning Architecture
CN110503601A (en) * 2019-08-28 2019-11-26 上海交通大学 Face based on confrontation network generates picture replacement method and system
CN112446364A (en) * 2021-01-29 2021-03-05 中国科学院自动化研究所 High-definition face replacement video generation method and system
CN113570497A (en) * 2021-02-04 2021-10-29 腾讯科技(深圳)有限公司 Image processing method, image processing device, computer equipment and storage medium
WO2021258920A1 (en) * 2020-06-24 2021-12-30 百果园技术(新加坡)有限公司 Generative adversarial network training method, image face swapping method and apparatus, and video face swapping method and apparatus

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20190171908A1 (en) * 2017-12-01 2019-06-06 The University Of Chicago Image Transformation with a Hybrid Autoencoder and Generative Adversarial Network Machine Learning Architecture
CN110503601A (en) * 2019-08-28 2019-11-26 上海交通大学 Face based on confrontation network generates picture replacement method and system
WO2021258920A1 (en) * 2020-06-24 2021-12-30 百果园技术(新加坡)有限公司 Generative adversarial network training method, image face swapping method and apparatus, and video face swapping method and apparatus
CN112446364A (en) * 2021-01-29 2021-03-05 中国科学院自动化研究所 High-definition face replacement video generation method and system
CN113570497A (en) * 2021-02-04 2021-10-29 腾讯科技(深圳)有限公司 Image processing method, image processing device, computer equipment and storage medium

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
易旭;白天;: "基于Star-Gan的人脸互换算法", 信息技术与网络安全, vol. 39, no. 5, 10 May 2020 (2020-05-10), pages 12 - 16 *

Similar Documents

Publication Publication Date Title
CN114387430B (en) Image description generation method, device, equipment and medium based on artificial intelligence
CN110516541B (en) Text positioning method and device, computer readable storage medium and computer equipment
JP2021089730A (en) Method of detecting at least one element of interest visible in input image by means of convolutional neural network
CN112581370A (en) Training and reconstruction method of super-resolution reconstruction model of face image
CN110163864B (en) Image segmentation method and device, computer equipment and storage medium
CN110969154A (en) Text recognition method and device, computer equipment and storage medium
CN113435330A (en) Micro-expression identification method, device, equipment and storage medium based on video
CN111833360A (en) Image processing method, device, equipment and computer readable storage medium
CN114429636A (en) Image scanning identification method and device and electronic equipment
CN113361521A (en) Scene image detection method and device
CN115861614A (en) Method and device for automatically generating semantic segmentation graph based on down jacket image
CN113538530B (en) Ear medical image segmentation method and device, electronic equipment and storage medium
CN110807463A (en) Image segmentation method and device, computer equipment and storage medium
CN111950496B (en) Mask person identity recognition method
CN114973382A (en) Human face replacement method, device, equipment and storage medium based on artificial intelligence
CN114821811B (en) Method and device for generating person composite image, computer device and storage medium
CN116342385A (en) Training method and device for text image super-resolution network and storage medium
CN111899263B (en) Image segmentation method, device, computer equipment and storage medium
CN115100050A (en) Method, device and equipment for removing ring artifacts of CT image and storage medium
CN112184884A (en) Three-dimensional model construction method and device, computer equipment and storage medium
CN113220859A (en) Image-based question and answer method and device, computer equipment and storage medium
CN112347865A (en) Bill correction method based on key point detection
CN117195286B (en) User privacy protection method and system based on big data
CN111161239B (en) Medical image analysis method, device, storage medium and computer equipment
CN113592759A (en) Image reconstruction method, image reconstruction device, electronic apparatus, and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination