Detailed Description
Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While certain embodiments of the present disclosure are shown in the drawings, it is to be understood that the disclosure may be embodied in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete. It should be understood that the drawings and embodiments of the disclosure are for illustration purposes only and are not intended to limit the scope of the disclosure.
It should be noted that, for convenience of description, only the portions related to the present invention are shown in the drawings. The embodiments and features of the embodiments in the present disclosure may be combined with each other without conflict.
It should be noted that the terms "first", "second", and the like in the present disclosure are only used for distinguishing different devices, modules or units, and are not used for limiting the order or interdependence of the functions performed by the devices, modules or units.
It is noted that references to "a", "an", and "the" modifications in this disclosure are intended to be illustrative rather than limiting, and that those skilled in the art will recognize that "one or more" may be used unless the context clearly dictates otherwise.
The names of messages or information exchanged between devices in the embodiments of the present disclosure are for illustrative purposes only, and are not intended to limit the scope of the messages or information.
The present disclosure will be described in detail below with reference to the accompanying drawings in conjunction with embodiments.
Fig. 1 illustrates a flow 100 of some embodiments of a back propagation-based face editing method according to the present disclosure. The human face editing method based on the back propagation comprises the following steps:
step 101, obtaining an initial face feature vector of a target face.
In some embodiments, an executing subject (e.g., a server) of the back-propagation-based face editing method may obtain the initial face feature vector of the target face from the image acquisition device through a wired connection manner or a wireless connection manner. It is noted that the wireless connection means may include, but is not limited to, a 3G/4G connection, a WiFi connection, a bluetooth connection, a WiMAX connection, a Zigbee connection, a UWB (ultra wideband) connection, and other wireless connection means now known or developed in the future. The image acquisition device can be a device with a shooting function or a screenshot function. For example, the image capturing devices described above may include, but are not limited to: cell phone, computer, camera. The initial face feature vector may refer to a vector of a photographed image of a target face.
And 102, inputting the initial face feature vector into a pre-trained attribute fitting model to obtain an initial face feature vector attribute value.
In some embodiments, the executing entity may input the initial face feature vector into a pre-trained attribute fitting model to obtain an initial face feature vector attribute value. Here, the attribute fitting model may be a neural network model that takes an initial face feature vector as an input and takes an initial face feature vector attribute value as an output. Here, the initial face feature vector attribute value may refer to an attribute value characterizing a face feature. For example, the initial face feature vector attribute value may characterize an age attribute or a gender attribute. For example, the attribute fitting model may be a CNN Network (Convolutional Neural Network).
Optionally, the attribute fitting model is obtained by training through the following steps:
firstly, a face feature vector set is obtained. In practice, the face feature vector set can be obtained from the face feature vector library in a wired connection mode or a wireless connection mode.
And secondly, inputting the human face feature vector set into a human face editing model to obtain a human face image set. And the face feature vectors in the face feature vector set correspond to the face images in the face image set. The face editing model may be a neural network model that takes a face feature vector as input and takes an edited face image as output. For example, the face editing model may be a GAN network (Generative adaptive Networks). In practice, each face feature vector in the face feature vector set may be input into a face editing model to generate a face image, so as to obtain a face image set.
And thirdly, inputting the face image set into a pre-trained attribute classifier to obtain a face image attribute data set. The face images in the face image set correspond to the face image attribute data in the face image attribute data set. Here, the attribute classifier may be a classifier that takes a face image as an input and takes face image attribute data as an output. For example, the attribute classifier may be a Decision Tree model (DT). In practice, each face image in the face image set may be input to a pre-trained attribute classifier to generate face image attribute data, so as to obtain a face image attribute data set.
And fourthly, combining each face feature vector in the face feature vector set and face image attribute data corresponding to the face feature vector in the face image attribute data set into an initial attribute fitting model sample to obtain an initial attribute fitting model sample set. Here, combining may refer to splicing.
And fifthly, training the initial attribute fitting model based on the initial attribute fitting model sample set to obtain an attribute fitting model.
In practice, the fifth step may include the following sub-steps:
the first substep is to determine the network structure of the initial attribute fitting model and to initialize the network parameters of the initial attribute fitting model.
And a second substep, using the face feature vector included in the initial attribute fitting model sample set as the input of the initial attribute fitting model, using the face image attribute data corresponding to the face feature vector included in the initial attribute fitting model sample set as the expected output of the initial attribute fitting model, and training the initial attribute fitting model by using a deep learning method. The initial attribute fitting model may be a convolutional neural network or a cyclic neural network.
And a third substep of determining the initial attribute fitting model obtained by training as the trained attribute fitting model.
The related contents in the first step to the fifth step are taken as an inventive point of the embodiments of the present disclosure, and the technical problem mentioned in the background art is solved, namely, in the case that the attribute distribution deviation exists in the training set, the attribute entanglement usually exists. The factors that entangle presence attributes tend to be as follows: in the case where there is a deviation in the distribution of attributes in the training set, there is usually an entanglement of attributes. If the above factors are solved, the effect of effectively removing the attribute entanglement can be achieved. To achieve this effect, first, a face feature vector set is obtained. Therefore, a data set of the face editing model is obtained, and data support is provided for training of the face editing model. Secondly, the face feature vector set is input into the face editing model to obtain a face image set. And the face feature vectors in the face feature vector set correspond to the face images in the face image set. Therefore, a data set of the attribute classifier is obtained, and data support is provided for training of the attribute classifier. And then, inputting the face image set into a pre-trained attribute classifier to obtain a face image attribute data set. The face images in the face image set correspond to the face image attribute data in the face image attribute data set. And combining each face feature vector in the face feature vector set and the face image attribute data corresponding to the face feature vector in the face image attribute data set into an initial attribute fitting model sample to obtain an initial attribute fitting model sample set. Thus, data support is provided for training of the initial attribute fitting model. And finally, training the initial attribute fitting model based on the initial attribute fitting model sample set to obtain an attribute fitting model. Therefore, the attribute distribution bias of the data set is improved, and the effect of effectively removing the attribute entanglement is achieved.
And 103, converting the initial face feature vector attribute value and the preset attribute value into a face feature vector gradient value.
In some embodiments, the execution body may convert the initial face feature vector attribute value and the preset attribute value into a face feature vector gradient value. The gradient value of the face feature vector represents a semantic direction for performing attribute editing on the face feature vector.
In practice, the execution subject may determine a loss value between the initial face feature vector attribute value and the preset attribute value based on a preset loss function. For example, the preset loss function may be a cross entropy loss function.
In practice, based on a preset loss function, the execution subject may perform derivation processing on the initial face feature vector through the following formula to generate a face feature vector gradient value:
where Δ z represents the face feature vector gradient value. Z denotes an initial face feature vector. y is
0 Representing the initial face feature vector attribute values. y denotes a preset attribute value. L (y)
0 And y) represents a loss function.
The loss function is represented by taking the derivative of the initial face feature vector.
And step 104, performing gradient updating on the initial face feature vector based on the face feature vector gradient value to generate an updated face feature vector.
In some embodiments, the execution subject may perform gradient update on the initial face feature vector based on the face feature vector gradient value to generate an updated face feature vector.
In practice, based on the gradient value of the face feature vector, the executing entity may generate an updated face feature vector by:
step one, the product of the preset face feature vector updating coefficient and the face feature vector gradient value is determined as a vector gradient value.
And secondly, determining the difference value between the initial face feature vector and the vector gradient value as an updated face feature vector.
Updating the face feature vector may be:
wherein,
representing the updated face feature vector. Z denotes an initial face feature vector. And lambda represents a face feature vector updating coefficient. And deltaz represents gradient values of the characteristic vector of the human face. λ Δ z represents a vector gradient value.
And 105, inputting the updated face feature vector into a pre-trained face editing model to obtain a target face image.
In some embodiments, the execution subject may input the updated face feature vector into a pre-trained face editing model to obtain a target face image. And the target face image is a final face image obtained by editing the attribute of the initial face image. For example, the age attribute of the initial facial image is 25 years, the age attribute of the facial image of 60 years is expected to be obtained, facial feature vector extraction and relevant gradient updating processing are carried out on the initial facial image, the updated facial feature vector is input into a face editing model trained in advance, and the finally obtained facial image of the age attribute of 60 years is the target image. The face editing model may be a GAN network (Generative adaptive Networks).
The above embodiments of the present disclosure have the following advantages: by the face editing method based on back propagation of some embodiments of the present disclosure, modeling of correlation of complex attributes is completed, and the definition of face editing images is improved. In particular, the reason why it is difficult to model the complex attribute correlations is that: semantic directions used by the face editing method are all on an attribute level, and complex attribute correlation is difficult to model, so that the definition of an edited face image is low. Based on this, the method for editing a face based on back propagation according to some embodiments of the present disclosure first obtains an initial face feature vector of a target face. Thus, the target face image information to be edited by different attributes of the face is obtained. Secondly, inputting the initial face feature vector into a pre-trained attribute fitting model to obtain an initial face feature vector attribute value. Therefore, the preparation data for modeling the complex attribute correlation is obtained, and data support is provided for modeling the complex attribute correlation. And then, converting the initial face feature vector attribute value and the preset attribute value into a face feature vector gradient value. The gradient value of the face feature vector represents a semantic direction for performing attribute editing on the face feature vector. Then, based on the gradient value of the face feature vector, the gradient update is carried out on the initial face feature vector to generate an updated face feature vector. Thus, an optimal semantic direction between the attribute level and the instance-specific direction is found. And finally, inputting the updated face feature vector into a face editing model trained in advance to obtain a target face image. Therefore, modeling of correlation of complex attributes is completed, and definition of the face editing image is improved.
With further reference to fig. 2, as an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of a back propagation-based face editing apparatus, which correspond to those shown in fig. 1, and which can be applied in various electronic devices.
As shown in fig. 2, the back propagation-based face editing apparatus 200 of some embodiments includes: an acquisition unit 201, a first input unit 202, a conversion unit 203, an update unit 204, and a second input unit 205. The acquiring unit 201 is configured to acquire an initial face feature vector of a target face; a first input unit 202, configured to input the initial face feature vector into a pre-trained attribute fitting model, so as to obtain an initial face feature vector attribute value; a conversion unit 203 configured to convert the initial face feature vector attribute value and a preset attribute value into a face feature vector gradient value, where the face feature vector gradient value represents a semantic direction for performing attribute editing on a face feature vector; an updating unit 204 configured to perform gradient updating on the initial face feature vector based on the face feature vector gradient value to generate an updated face feature vector; and a second input unit 205 configured to input the updated face feature vector into a pre-trained face editing model, so as to obtain a target face image.
It is to be understood that the units described in the back propagation-based face editing apparatus 200 correspond to the respective steps in the method described with reference to fig. 1. Thus, the operations, features and resulting advantages described above with respect to the method are also applicable to the apparatus 200 and the units included therein, and are not described herein again.
Referring now to FIG. 3, a block diagram of an electronic device (e.g., server) 300 suitable for use in implementing some embodiments of the present disclosure is shown. The electronic device in some embodiments of the present disclosure may include, but is not limited to, a mobile terminal such as a mobile phone, a notebook computer, a digital broadcast receiver, a PDA (personal digital assistant), a PAD (tablet computer), a PMP (portable multimedia player), a vehicle-mounted terminal (e.g., a car navigation terminal), and the like, and a stationary terminal such as a digital TV, a desktop computer, and the like. The electronic device shown in fig. 3 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present disclosure.
As shown in fig. 3, the electronic device 300 may include a processing means (e.g., a central processing unit, a graphics processor, etc.) 301 that may perform various appropriate actions and processes in accordance with a program stored in a Read Only Memory (ROM) 302 or a program loaded from a storage means 308 into a Random Access Memory (RAM) 303. In the RAM 303, various programs and data necessary for the operation of the electronic apparatus 300 are also stored. The processing device 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An input/output (I/O) interface 305 is also connected to bus 304.
Generally, the following devices may be connected to the I/O interface 305: input devices 306 including, for example, a touch screen, touch pad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; an output device 307 including, for example, a Liquid Crystal Display (LCD), a speaker, a vibrator, and the like; storage devices 308 including, for example, magnetic tape, hard disk, etc.; and a communication device 309. The communication means 309 may allow the electronic device 300 to communicate wirelessly or by wire with other devices to exchange data. While fig. 3 illustrates an electronic device 300 having various means, it is to be understood that not all illustrated means are required to be implemented or provided. More or fewer devices may alternatively be implemented or provided. Each block shown in fig. 3 may represent one device or may represent multiple devices, as desired.
In particular, according to some embodiments of the present disclosure, the processes described above with reference to the flow diagrams may be implemented as computer software programs. For example, some embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In some such embodiments, the computer program may be downloaded and installed from a network through the communication device 309, or installed from the storage device 308, or installed from the ROM 302. The computer program, when executed by the processing apparatus 301, performs the above-described functions defined in the methods of some embodiments of the present disclosure.
It should be noted that the computer readable medium described in some embodiments of the present disclosure may be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In some embodiments of the disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In some embodiments of the present disclosure, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, optical cables, RF (radio frequency), etc., or any suitable combination of the foregoing.
In some embodiments, the clients, servers may communicate using any currently known or future developed network Protocol, such as HTTP (Hyper Text Transfer Protocol), and may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), the Internet (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future developed network.
The computer readable medium may be embodied in the electronic device; or may be separate and not incorporated into the electronic device. The computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to: acquiring an initial face feature vector of a target face; inputting the initial face feature vector into a pre-trained attribute fitting model to obtain an initial face feature vector attribute value; converting the initial face feature vector attribute value and a preset attribute value into a face feature vector gradient value, wherein the face feature vector gradient value represents a semantic direction for performing attribute editing on the face feature vector; based on the gradient value of the face feature vector, carrying out gradient update on the initial face feature vector to generate an updated face feature vector; and inputting the updated face feature vector into a pre-trained face editing model to obtain a target face image.
Computer program code for carrying out operations for embodiments of the present disclosure may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, smalltalk, C + +, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The units described in some embodiments of the present disclosure may be implemented by software or hardware. The described units may also be provided in a processor, and may be described as: a processor includes an acquisition unit, a first input unit, a conversion unit, an update unit, and a second input unit. The names of these units do not in some cases form a limitation on the unit itself, and for example, the updating unit may also be described as "a unit that performs gradient update on the initial face feature vector based on the face feature vector gradient value to generate an updated face feature vector".
The functions described herein above may be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), systems on a chip (SOCs), complex Programmable Logic Devices (CPLDs), and the like.
Some embodiments of the present disclosure also provide a computer program product comprising a computer program that, when executed by a processor, implements any of the above-described back propagation-based face editing methods.
The foregoing description is only exemplary of the preferred embodiments of the disclosure and is illustrative of the principles of the technology employed. It will be appreciated by those skilled in the art that the scope of the invention in the embodiments of the present disclosure is not limited to the specific combination of the above-mentioned features, but also encompasses other embodiments in which any combination of the above-mentioned features or their equivalents is made without departing from the inventive concept as defined above. For example, the above features and (but not limited to) technical features with similar functions disclosed in the embodiments of the present disclosure are mutually replaced to form the technical solution.