CN110276811B - Image conversion method and device, electronic equipment and readable storage medium - Google Patents
Image conversion method and device, electronic equipment and readable storage medium Download PDFInfo
- Publication number
- CN110276811B CN110276811B CN201910588382.7A CN201910588382A CN110276811B CN 110276811 B CN110276811 B CN 110276811B CN 201910588382 A CN201910588382 A CN 201910588382A CN 110276811 B CN110276811 B CN 110276811B
- Authority
- CN
- China
- Prior art keywords
- domain
- initial
- network
- coding
- source
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T9/00—Image coding
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Compression Of Band Width Or Redundancy In Fax (AREA)
- Compression Or Coding Systems Of Tv Signals (AREA)
Abstract
The embodiment of the application provides an image conversion method, an image conversion device, an electronic device and a readable storage medium, wherein a source domain image to be converted is obtained, a source domain hidden space code from the source domain image to a hidden space can be obtained through an encoder of a self-coding network, a mapping from the source domain hidden space code to a target domain hidden space code of the hidden space is obtained through a cyclic generation countermeasure network, and finally the target domain hidden space code is input into a target domain decoder to be decoded to obtain a target domain image, so that the conversion between the source domain and the target domain is realized. Therefore, compared with the prior art that the countermeasure generation network training is simply adopted, the method can obtain the target domain image with higher quality, reduces the network depth and the training difficulty, and can better adapt to the requirements under various environments.
Description
Technical Field
The present application relates to the field of image processing, and in particular, to an image conversion method and apparatus, an electronic device, and a readable storage medium.
Background
When trimming a shot image, a user no longer meets the requirement of simply adjusting the values of the contrast, saturation and the like of the image, but needs more advanced adjustment such as conversion of the style type of the image. For example, a real self-portrait image is converted into a cartoon character image, a landscape image is converted into a pencil sketch image, a faceless face image is converted into a face image of a smiling face, an adult image is converted into a child image, and the like.
Most of the current methods for image conversion are realized based on generation of a countermeasure network, and the methods adopt a generator to map a source domain image onto a target image domain to obtain a converted image. Then, the converted image and the image on the target image domain are input into a discriminator, the two are trained simultaneously, then the image mapped to the target domain is mapped back to the source domain through a generator to obtain a secondary generated image, and finally, a certain constraint between the secondary generated image and the original image is minimized to achieve the conversion from the source domain image to the target domain image. However, the inventor finds that the above scheme is often complex in network structure and high in network depth, and training difficulty in generating the countermeasure network is high, so that the image quality after final conversion is often poor.
Disclosure of Invention
In view of the above, an object of the embodiments of the present application is to provide an image conversion method, an image conversion apparatus, an electronic device and a readable storage medium, so as to solve or improve the above problems.
According to an aspect of embodiments of the present application, there is provided an electronic device that may include one or more storage media and one or more processors in communication with the storage media. One or more storage media store machine-executable instructions that are executable by a processor. When the electronic device is running, the processor executes the machine executable instructions to perform the image conversion method.
According to another aspect of embodiments of the present application, there is provided an image conversion method applied to an electronic device, in which a self-coding network and a loop generation countermeasure network are stored, the self-coding network including an encoder and a target domain decoder, the method including:
acquiring a source domain image to be converted;
inputting the source domain image into an encoder of the self-coding network for encoding to obtain a source domain hidden space code;
inputting the source domain hidden space code into a generation network from a source domain to a target domain in the circularly generated countermeasure network for processing to obtain a target domain hidden space code;
and inputting the target domain implicit space code into the target domain decoder for decoding to obtain a target domain image.
In one possible embodiment, the method further comprises:
configuring an initial self-coding network, wherein the initial self-coding network comprises an initial coder, an initial source domain decoder and an initial target domain decoder;
acquiring a training sample set, wherein the training sample set comprises source domain image samples and target domain image samples;
and training the initial encoder, the initial source domain decoder and the initial target domain decoder according to the source domain image samples and the target domain image samples to obtain the self-encoding network, and storing the self-encoding network in the electronic equipment.
In one possible implementation, the step of training the initial encoder, the initial source-domain decoder and the initial target-domain decoder according to the source-domain image samples and the target-domain image samples to obtain the self-coding network includes:
respectively encoding the source domain image sample and the target domain image sample through the initial encoder to obtain a source domain implicit space code corresponding to the source domain image sample and a target domain implicit space code corresponding to the target domain image sample;
decoding the source domain implicit space code through the initial source domain decoder to obtain a source decoded image corresponding to the source domain implicit space code, and decoding the target domain implicit space code through the initial target domain decoder to obtain a target decoded image corresponding to the target domain implicit space code;
updating network parameters of the initial encoder, the initial source domain decoder and the initial target domain decoder according to the source decoded image, the source domain image sample, the target decoded image and the target domain image sample, and returning to the step of respectively encoding the source domain image sample and the target domain image sample by the initial encoder until the initial encoder, the initial source domain decoder and the initial target domain decoder meet a training termination condition, and outputting the self-encoding network obtained by training.
In a possible implementation, the step of updating the network parameters of the initial encoder, the initial source-domain decoder and the initial target-domain decoder according to the source decoded image, the source-domain image samples, the target decoded image and the target-domain image samples comprises:
calculating a first loss function value between the source decoded image and the source domain image samples and a second loss function value between the target decoded image and the target domain image samples;
updating network parameters of the initial encoder, the initial source domain decoder and the initial target domain decoder according to the first loss function value and the second loss function value.
In one possible embodiment, the method further comprises:
configuring an initial cycle generation countermeasure network, wherein the initial cycle generation countermeasure network comprises a first initial generation network and a first initial discrimination network from a source domain to a target domain and a second initial generation network and a second initial discrimination network from the target domain to the source domain;
processing the source domain hidden space code obtained after the source domain image sample is coded by the coder of the input self-coding network through the first initial generation network to obtain a first target domain hidden space code, and processing the first target domain hidden space code through the second initial generation network to obtain a first source domain hidden space code;
processing a target domain hidden space code obtained after an encoder of the input self-coding network encodes a target domain image sample through the second initial generation network to obtain a second source domain hidden space code, and processing the second source domain hidden space code through the first initial generation network to obtain a second target domain hidden space code;
training the initial cycle generation countermeasure network according to the first source domain hidden space encoding, the second source domain hidden space encoding, the first target domain hidden space encoding and the second target domain hidden space encoding to obtain the cycle generation countermeasure network, and storing the cycle generation countermeasure network in the electronic device.
In a possible implementation manner, the processing, by the first initial generation network, a source-domain implicit spatial code obtained by encoding a source-domain image sample by an encoder of the input self-encoding network to obtain a first target-domain implicit spatial code, and processing, by the second initial generation network, the first target-domain implicit spatial code to obtain a first source-domain implicit spatial code includes:
processing a source domain hidden space code obtained after an encoder of the input self-coding network encodes a source domain image sample through the first initial generation network to obtain a coding mapping from the source domain hidden space code to a target domain hidden space code and a weighting coefficient corresponding to the source domain hidden space code on each bit code in the hidden space codes;
obtaining the first target domain hidden space code according to the code mapping from the source domain hidden space code to the target domain hidden space code and the weighting coefficient corresponding to the source domain hidden space code on each bit code in the hidden space code;
processing the input first target domain hidden space coding through the second initial generation network to obtain a coding mapping from the first target domain hidden space coding to a source domain hidden space coding and a weighting coefficient corresponding to the first target domain hidden space coding on each bit of the hidden space coding;
obtaining the first source domain hidden space code according to the code mapping from the first target domain hidden space code to the source domain hidden space code and the weighting coefficient corresponding to the first target domain hidden space code on each bit of the hidden space code;
the step of processing the target domain hidden space code obtained by encoding the target domain image sample by the encoder of the input self-encoding network through the second initial generation network to obtain a second source domain hidden space code, and processing the second source domain hidden space code through the first initial generation network to obtain a second target domain hidden space code includes:
processing a target domain hidden space code obtained after an encoder of the input self-encoding network encodes a target domain image sample through the second initial generation network to obtain a coding mapping from the target domain hidden space code to a source domain hidden space code and a weighting coefficient corresponding to the target domain hidden space code on each bit code in the hidden space codes;
obtaining the second source domain hidden space code according to the coding mapping from the target domain hidden space code to the source domain hidden space code and the weighting coefficient corresponding to the target domain hidden space code on each bit code in the hidden space codes;
processing the input second source domain hidden space coding through the first initial generation network to obtain a weighting coefficient corresponding to the second source domain hidden space coding on each bit of coding mapping from the second source domain hidden space coding to the source domain hidden space coding and the hidden space coding;
and obtaining the second target domain hidden space code according to the coding mapping from the second source domain hidden space code to the source domain hidden space code and a weighting coefficient corresponding to the second source domain hidden space code on each bit of the hidden space code.
In a possible implementation manner, the step of training the initial loop-generated confrontation network according to the first source-domain hidden spatial coding, the second source-domain hidden spatial coding, the first target-domain hidden spatial coding, and the second target-domain hidden spatial coding to obtain the loop-generated confrontation network includes:
distinguishing the first target domain implicit space coding and the target domain implicit space coding through the first initial distinguishing network to obtain a first distinguishing result, and distinguishing the second source domain implicit space coding and the source domain implicit space coding through the second initial distinguishing network to obtain a second distinguishing result;
calculating a third loss function value between a first target domain implicit space coding discrimination value and the target domain implicit space coding discrimination value and a fourth loss function value between a first source domain implicit space coding discrimination value and a source domain implicit space coding discrimination value in the first discrimination result, and calculating a fifth loss function value between a second source domain implicit space coding discrimination value and the source domain implicit space coding discrimination value and a sixth loss function value between the second target domain implicit space coding discrimination value and the target domain implicit space coding discrimination value in the second discrimination result;
and updating network parameters of the initial loop generation countermeasure network according to the first judgment result, the second judgment result, the third loss function value, the fourth loss function value, the fifth loss function value and the sixth loss function value, and returning to the step of processing the source domain implicit space code obtained after the source domain image sample is coded by the encoder of the input self-coding network through the first initial generation network, until the initial loop generation countermeasure network meets the training termination condition, outputting the trained loop generation countermeasure network.
According to another aspect of the embodiments of the present application, there is provided an image conversion apparatus applied to an electronic device, in which a self-coding network and a loop generation countermeasure network are stored, the self-coding network including an encoder and a target domain decoder, the apparatus including:
the acquisition module is used for acquiring a source domain image to be converted;
the coding module is used for inputting the source domain image into a coder of the self-coding network for coding to obtain a source domain hidden space code;
the input module is used for inputting the source domain hidden space code into a generation network from a source domain to a target domain in the circularly generated countermeasure network for processing to obtain a target domain hidden space code;
and the decoding module is used for inputting the target domain implicit space code into the target domain decoder for decoding to obtain a target domain image.
According to another aspect of the embodiments of the present application, there is provided a readable storage medium having stored thereon machine executable instructions, which when executed by a processor, can perform the steps of the image conversion method described above.
Based on any aspect, in the embodiment of the present application, a source domain image to be converted is obtained, a source domain hidden space encoding from the source domain image to a hidden space can be obtained through an encoder of a self-encoding network, a mapping from the source domain hidden space encoding to a target domain hidden space encoding of the hidden space is obtained through a loop generation countermeasure network, and finally the target domain hidden space encoding is input into a target domain decoder for decoding to obtain a target domain image, so that conversion between the source domain and the target domain is realized. Therefore, compared with the prior art that the countermeasure generation network training is simply adopted, the method can obtain the target domain image with higher quality, reduces the network depth and the training difficulty, and can better adapt to the requirements under various environments.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are required to be used in the embodiments will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present application and therefore should not be considered as limiting the scope, and for those skilled in the art, other related drawings can be obtained from the drawings without inventive effort.
Fig. 1 shows one of the flow diagrams of an image conversion method provided by the embodiment of the present application;
fig. 2 illustrates a second flowchart of the image conversion method according to the embodiment of the present application;
FIG. 3 is a schematic diagram illustrating training of a self-coding network provided by an embodiment of the present application;
fig. 4 shows a third flowchart of an image conversion method provided in an embodiment of the present application;
FIG. 5 is a schematic diagram illustrating training of a loop-generated countermeasure network provided by an embodiment of the present application;
fig. 6 shows a block diagram of a structural schematic diagram of an electronic device according to an embodiment of the present application.
Detailed Description
In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it should be understood that the drawings in the present application are for illustrative and descriptive purposes only and are not used to limit the scope of protection of the present application. Additionally, it should be understood that the schematic drawings are not necessarily drawn to scale. The flowcharts used in this application illustrate operations implemented according to some of the embodiments of the present application. It should be understood that the operations of the flow diagrams may be performed out of order, and steps without logical context may be performed in reverse order or simultaneously. In addition, one skilled in the art, under the guidance of the present disclosure, may add one or more other operations to the flowchart, or may remove one or more operations from the flowchart.
In addition, the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. The components of the embodiments of the present application, generally described and illustrated in the figures herein, can be arranged and designed in a wide variety of different configurations. Thus, the following detailed description of the embodiments of the present application, as presented in the figures, is not intended to limit the scope of the claimed application, but is merely representative of selected embodiments of the application. All other embodiments, which can be derived by a person skilled in the art from the embodiments of the present application without making any creative effort, shall fall within the protection scope of the present application.
Fig. 1 shows a schematic flowchart of an image conversion method provided in an embodiment of the present application, and it should be understood that, in other embodiments, the order of some steps in the image conversion method of this embodiment may not be limited by the order in fig. 1 and the following specific embodiments, for example, the steps may be interchanged according to actual needs, or some steps may also be omitted or deleted. The detailed steps of the image conversion method are described below.
Step S110, a source domain image to be converted is obtained.
And step S120, inputting the source domain image into an encoder of a coding network for coding to obtain the source domain hidden space code.
Step S130, the source domain hidden space coding input is circularly generated into a generation network from a source domain to a target domain in a countermeasure network for processing, and the target domain hidden space coding is obtained.
Step S140, the target domain implicit space coding is input into a target domain decoder for decoding, and a target domain image is obtained.
In this embodiment, for step S110, the source domain image to be converted may be image information obtained by the user through any feasible manner such as photographing, network downloading, and the like, for example, the source domain image to be converted may be an image obtained through photographing by an installed camera application, or an image downloaded from a website, a chat log, a cloud service, and the like through a network, and this embodiment does not limit this.
In step S120, the source domain image is input from an encoder of the coding network for encoding, so that mapping from the source domain image to the hidden space, that is, source domain hidden space encoding, can be obtained.
In step S130, a source-domain-to-target-domain generation network in the countermeasure network is generated by inputting the source-domain-to-target-domain implicit spatial coding in a cyclic manner, so as to obtain a coding mapping from the source-domain-to-target-domain-to-implicit spatial coding and a weighting coefficient corresponding to the source-domain-to-implicit spatial coding on each bit of the implicit spatial coding, and obtain the target-domain-to-implicit spatial coding according to the weighting of the weighting coefficient corresponding to the source-domain-to-target-domain-to-implicit spatial coding on each bit of the coding mapping from the source-domain-to-target-domain-to-implicit spatial coding and the weighting of the weighting coefficient corresponding to the source-domain-to-implicit spatial coding on each bit of the implicit spatial coding.
After the target domain implicit space coding is obtained, decoding can be carried out through an input target domain decoder to obtain a target domain image.
Based on the above design, in this embodiment, a source domain image to be converted is obtained, a source domain hidden space encoding from the source domain image to a hidden space is obtained through an encoder of a self-encoding network, a mapping from the source domain hidden space encoding to a target domain hidden space encoding of the hidden space is obtained through a cyclic generation countermeasure network, and finally the target domain hidden space encoding is input to a target domain decoder for decoding to obtain a target domain image, so that conversion between the source domain and the target domain is realized. Compared with the prior art that the confrontation generation network training is simply adopted, the method can obtain the target domain image with higher quality, reduces the network depth and the training difficulty, and can better adapt to the requirements under various environments.
As a possible implementation manner, referring to fig. 2, before the step S110, the image conversion method provided in this embodiment may further include the following steps:
step S101, configuring an initial self-coding network.
In this embodiment, the initial self-encoding network may include an initial encoder, an initial source domain decoder, and an initial target domain decoder.
Step S102, a training sample set is obtained.
In this embodiment, the training sample set includes source domain image samples and target domain image samples. The source domain image sample may be any image that needs to be converted into a target domain, and the target domain image sample is an image converted into a target domain.
Step S103, training an initial encoder, an initial source domain decoder and an initial target domain decoder according to the source domain image samples and the target domain image samples to obtain a self-encoding network, and storing the self-encoding network in the electronic device.
Referring to fig. 3, the training process of the self-coding network in step S103 is exemplarily described below based on fig. 3.
Firstly, respectively encoding a source domain image sample and a target domain image sample through an initial encoder to obtain a source domain implicit space code corresponding to the source domain image sample and a target domain implicit space code corresponding to the target domain image sample. Therefore, the source domain image samples and the target domain image samples can be mapped into the same hidden space through the same initial encoder.
Then, decoding the source domain implicit space code through an initial source domain decoder to obtain a source decoded image corresponding to the source domain implicit space code, and decoding the target domain implicit space code through an initial target domain decoder to obtain a target decoded image corresponding to the target domain implicit space code;
then, updating the network parameters of the initial encoder, the initial source domain decoder and the initial target domain decoder according to the source decoded image, the source domain image sample, the target decoded image and the target domain image sample. For example, as one possible implementation, a first loss function value between the source decoded image and the source domain image samples and a second loss function value between the target decoded image and the target domain image samples may be calculated, and the network parameters of the initial encoder, the initial source domain decoder and the initial target domain decoder may be updated according to the first loss function value and the second loss function value. On the basis, the step of respectively coding the source domain image sample and the target domain image sample by the initial coder is returned until the initial coder, the initial source domain decoder and the initial target domain decoder meet the training termination condition, and the self-coding network obtained by training is output.
The training termination condition may include at least one of the following three conditions:
1) The iterative training times reach the set times; 2) The first loss function value and the second loss function value are lower than a set threshold; 3) The first loss function value and the second loss function value do not decrease.
In the condition 1), in order to save the operation amount, the maximum value of the iteration times may be set, and if the iteration times reaches the set times, the iteration of the iteration cycle may be stopped, and the finally obtained network is used as the self-encoding network. In condition 2), if the first loss function value and the second loss function value are lower than the set threshold, which indicates that the current self-coding network can substantially satisfy the condition, the iteration can be stopped. In condition 3), the first loss function value and the second loss function value no longer decrease, indicating that the optimal self-encoding network has been formed, and the iteration can be stopped.
The above-described iteration stop conditions may be used in combination or alternatively, and for example, the iteration may be stopped when the first loss function value and the second loss function value do not decrease any more, or when the number of iterations reaches a set number, or when the first loss function value and the second loss function value do not decrease any more. Alternatively, the iteration may be stopped when the first and second loss function values are below a set threshold and the first and second loss function values no longer decrease.
In addition, in an actual implementation process, the above example may not be limited to be used as the training termination condition, and a person skilled in the art may design a training termination condition different from the above example according to actual requirements.
Based on the foregoing description, please further refer to fig. 4, after the step S103, the image conversion method provided in this embodiment may further include the following steps:
and step S104, configuring an initial loop to generate a countermeasure network.
In this embodiment, the initial loop generation countermeasure network may include a first initial generation network and a first initial discrimination network from the source domain to the target domain, and a second initial generation network and a second initial discrimination network from the target domain to the source domain.
Referring to fig. 5, the training process of the cycle-generated confrontation network in this embodiment is exemplarily described below based on fig. 5.
Step S105, processing a source domain hidden space code obtained after an encoder of an input self-coding network encodes a source domain image sample through a first initial generation network to obtain a first target domain hidden space code, and processing the first target domain hidden space code through a second initial generation network to obtain the first source domain hidden space code.
For example, as a possible implementation manner, the source-domain implicit spatial coding obtained after the encoder of the input self-coding network encodes the source-domain image sample may be processed by the first initial generation network, so as to obtain a coding mapping from the source-domain implicit spatial coding to the target-domain implicit spatial coding and a weighting coefficient corresponding to the source-domain implicit spatial coding on each bit of the implicit spatial coding. On the basis, a first target domain hidden space code is obtained according to the coding mapping from the source domain hidden space code to the target domain hidden space code and the weighting coefficient corresponding to the source domain hidden space code on each bit code in the hidden space codes, and the input first target domain hidden space code is processed through a second initial generation network to obtain the weighting coefficient corresponding to the first target domain hidden space code on each bit code in the coding mapping from the first target domain hidden space code to the source domain hidden space code and the hidden space code. And then, obtaining the first source domain hidden space code according to the coding mapping from the first target domain hidden space code to the source domain hidden space code and the weighting coefficient corresponding to the first target domain hidden space code on each bit code in the hidden space code.
And step S106, processing the target domain hidden space code obtained after the target domain image sample is coded by the coder of the input self-coding network through the second initial generation network to obtain a second source domain hidden space code, and processing the second source domain hidden space code through the first initial generation network to obtain a second target domain hidden space code.
For example, as a possible implementation manner, the target domain hidden spatial coding obtained after the encoder of the input self-coding network encodes the target domain image sample may be processed by the second initial generation network, so as to obtain a coding mapping from the target domain hidden spatial coding to the source domain hidden spatial coding and a weighting coefficient corresponding to the target domain hidden spatial coding on each bit of the hidden spatial coding. On the basis, a second source domain hidden space code is obtained according to the coding mapping from the target domain hidden space code to the source domain hidden space code and the weighting coefficient corresponding to the target domain hidden space code on each bit code in the hidden space codes, and the input second source domain hidden space code is processed through the first initial generation network to obtain the weighting coefficient corresponding to the second source domain hidden space code on each bit code in the coding mapping from the second source domain hidden space code to the source domain hidden space code and the hidden space code. And then, obtaining a second target domain hidden space code according to the coding mapping from the second source domain hidden space code to the source domain hidden space code and a weighting coefficient corresponding to the second source domain hidden space code on each bit code in the hidden space codes.
Step S107, training an initial cycle generation countermeasure network according to the first source domain hidden space code, the second source domain hidden space code, the first target domain hidden space code and the second target domain hidden space code to obtain a cycle generation countermeasure network, and storing the cycle generation countermeasure network in the electronic equipment.
For example, as a possible implementation manner, a first initial discrimination network may discriminate a first target domain implicit spatial coding and a target domain implicit spatial coding to obtain a first discrimination result, and a second initial discrimination network may discriminate a second source domain implicit spatial coding and a source domain implicit spatial coding to obtain a second discrimination result. On the basis, a third LOSS function value (L2 LOSS) between a first target domain hidden space coding discrimination value and a target domain hidden space coding discrimination value in the first discrimination result and a fourth LOSS function value between a first source domain hidden space coding and a source domain hidden space coding discrimination value are calculated, and a fifth LOSS function value (L2 LOSS) between a second source domain hidden space coding discrimination value and a source domain hidden space coding discrimination value in the second discrimination result and a sixth LOSS function value between a second target domain hidden space coding and a target domain hidden space coding are calculated. And then updating network parameters of the initially generated confrontation network by the initial loop according to the first judgment result, the second judgment result, the third loss function value, the fourth loss function value, the fifth loss function value and the sixth loss function value, returning to the step of processing the source domain implicit space code obtained after the source domain image sample is coded by the encoder of the input self-coding network through the first initially generated network, and outputting the trained circularly generated confrontation network until the initially generated confrontation network meets the training termination condition.
The training termination condition may include at least one of the following three conditions:
1) The iterative training times reach the set times; 2) A total loss function value of the third loss function value, the fourth loss function value, the fifth loss function value, and the sixth loss function value is lower than a set threshold; 3) The total loss function value does not decrease.
Based on the steps, the quality of the hidden space mapping can be improved, and therefore the quality of the subsequently generated target domain image is improved.
Fig. 6 shows a schematic diagram of an electronic device 100 provided in an embodiment of the present application, where in this embodiment, the electronic device 100 may include a storage medium 110, a processor 120, and an image conversion apparatus 130.
The processor 120 may be a general-purpose Central Processing Unit (CPU), a microprocessor, an Application-Specific Integrated Circuit (ASIC), or one or more Integrated circuits for controlling the execution of the program of the image conversion method provided by the above-mentioned method embodiments.
The present application may perform division of function modules on the image conversion apparatus 130 according to the method embodiments, for example, each function module may be divided corresponding to each function, or two or more functions may be integrated into one processing module. The integrated module can be realized in a hardware mode, and can also be realized in a software functional module mode. It should be noted that, the division of the modules in the present application is schematic, and is only a logical function division, and there may be another division manner in actual implementation. For example, in the case of dividing each functional module according to each function, the image conversion apparatus 130 shown in fig. 6 is only a schematic diagram of the apparatus, and the functions of each functional module of the image conversion apparatus 130 will be described in detail below.
An obtaining module 131, configured to obtain a source domain image to be converted. It is understood that the obtaining module 131 can be used to execute the step S110, and for the detailed implementation of the obtaining module 131, reference can be made to the content related to the step S110.
The encoding module 132 is configured to input the source domain image into an encoder of the coding network for encoding, so as to obtain a source domain implicit spatial encoding. It is understood that the encoding module 132 can be used to perform the above step S120, and for the detailed implementation of the encoding module 132, reference can be made to the above description regarding the step S120.
The input module 133 is configured to generate a generation network from a source domain to a target domain in a countermeasure network cyclically for processing the source domain implicit spatial coding input, so as to obtain a target domain implicit spatial coding. It is understood that the input module 133 can be used to execute the step S130, and for the detailed implementation of the input module 133, reference can be made to the above-mentioned content related to the step S130.
And the decoding module 134 is configured to input the target domain implicit space code into a target domain decoder for decoding, so as to obtain a target domain image. It is understood that the decoding module 134 can be used to execute the step S140, and for the detailed implementation of the decoding module 134, reference can be made to the above description about the step S140.
Since the image conversion apparatus 130 provided in the embodiment of the present application is another implementation form of the image conversion method shown in fig. 1, and the image conversion apparatus 130 can be used to execute the method provided in the embodiment shown in fig. 1, the technical effects obtained by the method can refer to the above method embodiment, and are not repeated herein.
Further, based on the same inventive concept, embodiments of the present application also provide a computer-readable storage medium, on which a computer program is stored, and the computer program is executed by a processor to perform the steps of the image conversion method.
In particular, the storage medium can be a general-purpose storage medium, such as a removable disk, a hard disk, etc., on which a computer program can be executed to perform the above-described image conversion method when executed.
Embodiments of the present application are described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (e.g., electronic device 100 of fig. 6), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
While the present application has been described in connection with various embodiments, other variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed application, from a review of the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and the word "a" or "an" does not exclude a plurality. A single processor or other unit may fulfill the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.
The above description is only for various embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily conceive of changes or substitutions within the technical scope of the present application, and all such changes or substitutions are included in the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.
Claims (7)
1. An image conversion method applied to an electronic device, wherein a self-coding network and a loop generation countermeasure network are stored in the electronic device, the self-coding network includes an encoder and a target domain decoder, and the method includes:
acquiring a source domain image to be converted;
inputting the source domain image into an encoder of the self-coding network for encoding to obtain a source domain hidden space code;
inputting the source domain hidden space code into a generation network from a source domain to a target domain in the circularly generated countermeasure network for processing to obtain a target domain hidden space code;
inputting the target domain implicit space code into the target domain decoder for decoding to obtain a target domain image;
the method further comprises the following steps:
configuring an initial self-coding network, wherein the initial self-coding network comprises an initial encoder, an initial source domain decoder and an initial target domain decoder; acquiring a training sample set, wherein the training sample set comprises source domain image samples and target domain image samples; respectively encoding the source domain image sample and the target domain image sample through the initial encoder to obtain a source domain implicit space code corresponding to the source domain image sample and a target domain implicit space code corresponding to the target domain image sample; decoding the source domain implicit space code through the initial source domain decoder to obtain a source decoded image corresponding to the source domain implicit space code, and decoding the target domain implicit space code through the initial target domain decoder to obtain a target decoded image corresponding to the target domain implicit space code; updating network parameters of the initial encoder, an initial source domain decoder and an initial target domain decoder according to the source decoded image, the source domain image sample, the target decoded image and the target domain image sample, and returning to the step of respectively encoding the source domain image sample and the target domain image sample by the initial encoder until the initial encoder, the initial source domain decoder and the initial target domain decoder meet a training termination condition, outputting the self-encoding network obtained by training, and storing the self-encoding network in the electronic device;
configuring an initial loop generation countermeasure network, wherein the initial loop generation countermeasure network comprises a first initial generation network and a first initial discrimination network from a source domain to a target domain and a second initial generation network and a second initial discrimination network from the target domain to the source domain; processing a source domain hidden space code obtained after an encoder of the input self-coding network encodes a source domain image sample through the first initial generation network to obtain a first target domain hidden space code, and processing the first target domain hidden space code through the second initial generation network to obtain a first source domain hidden space code; processing a target domain hidden space code obtained after an encoder of the input self-coding network encodes a target domain image sample through the second initial generation network to obtain a second source domain hidden space code, and processing the second source domain hidden space code through the first initial generation network to obtain a second target domain hidden space code; training the initial cycle generation countermeasure network according to the first source domain hidden space coding, the second source domain hidden space coding, the first target domain hidden space coding and the second target domain hidden space coding to obtain the cycle generation countermeasure network, and storing the cycle generation countermeasure network in the electronic device.
2. The image conversion method according to claim 1, wherein the step of updating the network parameters of the initial encoder, the initial source-domain decoder and the initial target-domain decoder according to the source decoded image, the source-domain image samples, the target decoded image and the target-domain image samples comprises:
calculating a first loss function value between the source decoded image and the source domain image samples and a second loss function value between the target decoded image and the target domain image samples;
updating network parameters of the initial encoder, the initial source domain decoder and the initial target domain decoder according to the first loss function value and the second loss function value.
3. The image conversion method according to claim 1, wherein the step of processing, by the first initial generation network, the source-domain implicit spatial coding obtained by encoding the source-domain image samples by the encoder of the input self-encoding network to obtain a first target-domain implicit spatial coding, and processing, by the second initial generation network, the first target-domain implicit spatial coding to obtain a first source-domain implicit spatial coding comprises:
processing a source domain hidden space code obtained after an encoder of the input self-coding network encodes a source domain image sample through the first initial generation network to obtain a coding mapping from the source domain hidden space code to a target domain hidden space code and a weighting coefficient corresponding to the source domain hidden space code on each bit code in the hidden space codes;
obtaining the first target domain hidden space code according to the code mapping from the source domain hidden space code to the target domain hidden space code and the weighting coefficient corresponding to the source domain hidden space code on each bit of code in the hidden space code;
processing the input first target domain hidden space coding through the second initial generation network to obtain a coding mapping from the first target domain hidden space coding to a source domain hidden space coding and a weighting coefficient corresponding to the first target domain hidden space coding on each bit of the hidden space coding;
obtaining the first source domain hidden space code according to the coding mapping from the first target domain hidden space code to the source domain hidden space code and the weighting coefficient corresponding to the first target domain hidden space code on each bit code in the hidden space codes;
the step of processing, by the second initial generation network, a target domain implicit spatial code obtained after an encoder of the input self-encoding network encodes a target domain image sample to obtain a second source domain implicit spatial code, and processing, by the first initial generation network, the second source domain implicit spatial code to obtain a second target domain implicit spatial code, includes:
processing a target domain hidden space code obtained after an encoder of the input self-encoding network encodes a target domain image sample through the second initial generation network to obtain a coding mapping from the target domain hidden space code to a source domain hidden space code and a weighting coefficient corresponding to the target domain hidden space code on each bit code in the hidden space codes;
obtaining the second source domain hidden space code according to the coding mapping from the target domain hidden space code to the source domain hidden space code and the weighting coefficient corresponding to the target domain hidden space code on each bit code in the hidden space codes;
processing the input second source domain hidden space coding through the first initial generation network to obtain a weighting coefficient corresponding to the second source domain hidden space coding on each bit of coding mapping from the second source domain hidden space coding to the source domain hidden space coding and the hidden space coding;
and obtaining the second target domain hidden space code according to the coding mapping from the second source domain hidden space code to the source domain hidden space code and the weighting coefficient corresponding to the second source domain hidden space code on each bit code in the hidden space codes.
4. The image conversion method according to claim 1, wherein the step of training the initial loop-generated confrontation network according to the first source-domain hidden spatial coding, the second source-domain hidden spatial coding, the first target-domain hidden spatial coding and the second target-domain hidden spatial coding to obtain the loop-generated confrontation network comprises:
distinguishing the first target domain implicit space coding and the target domain implicit space coding through the first initial distinguishing network to obtain a first distinguishing result, and distinguishing the second source domain implicit space coding and the source domain implicit space coding through the second initial distinguishing network to obtain a second distinguishing result;
calculating a third loss function value between a first target domain implicit space coding discrimination value and the target domain implicit space coding discrimination value and a fourth loss function value between a first source domain implicit space coding discrimination value and a source domain implicit space coding discrimination value in the first discrimination result, and calculating a fifth loss function value between a second source domain implicit space coding discrimination value and the source domain implicit space coding discrimination value and a sixth loss function value between the second target domain implicit space coding discrimination value and the target domain implicit space coding discrimination value in the second discrimination result;
and updating network parameters of the initial loop generation countermeasure network according to the first judgment result, the second judgment result, the third loss function value, the fourth loss function value, the fifth loss function value and the sixth loss function value, and returning to the step of processing the source domain implicit space code obtained after the source domain image sample is coded by the encoder of the input self-coding network through the first initial generation network, until the initial loop generation countermeasure network meets the training termination condition, outputting the trained loop generation countermeasure network.
5. An image conversion apparatus applied to an electronic device in which a self-coding network and a loop generation countermeasure network are stored, the self-coding network including an encoder and a target domain decoder, the apparatus comprising:
the acquisition module is used for acquiring a source domain image to be converted;
the coding module is used for inputting the source domain image into a coder of the self-coding network for coding to obtain a source domain hidden space code;
the input module is used for inputting the source domain hidden space code into a generation network from a source domain to a target domain in the circularly generated countermeasure network for processing to obtain a target domain hidden space code;
the decoding module is used for inputting the target domain implicit space code into the target domain decoder for decoding to obtain a target domain image;
the obtaining module is further configured to:
configuring an initial self-coding network, wherein the initial self-coding network comprises an initial encoder, an initial source domain decoder and an initial target domain decoder; acquiring a training sample set, wherein the training sample set comprises source domain image samples and target domain image samples; respectively encoding the source domain image sample and the target domain image sample through the initial encoder to obtain a source domain implicit space code corresponding to the source domain image sample and a target domain implicit space code corresponding to the target domain image sample; decoding the source domain hidden space code through the initial source domain decoder to obtain a source decoded image corresponding to the source domain hidden space code, and decoding the target domain hidden space code through the initial target domain decoder to obtain a target decoded image corresponding to the target domain hidden space code; updating network parameters of the initial encoder, an initial source domain decoder and an initial target domain decoder according to the source decoded image, the source domain image sample, the target decoded image and the target domain image sample, and returning to the step of respectively encoding the source domain image sample and the target domain image sample by the initial encoder until the initial encoder, the initial source domain decoder and the initial target domain decoder meet a training termination condition, outputting the self-encoding network obtained by training, and storing the self-encoding network in the electronic device;
configuring an initial loop generation countermeasure network, wherein the initial loop generation countermeasure network comprises a first initial generation network and a first initial discrimination network from a source domain to a target domain and a second initial generation network and a second initial discrimination network from the target domain to the source domain; processing a source domain hidden space code obtained after an encoder of the input self-coding network encodes a source domain image sample through the first initial generation network to obtain a first target domain hidden space code, and processing the first target domain hidden space code through the second initial generation network to obtain a first source domain hidden space code; processing a target domain hidden space code obtained after an encoder of the input self-coding network encodes a target domain image sample through the second initial generation network to obtain a second source domain hidden space code, and processing the second source domain hidden space code through the first initial generation network to obtain a second target domain hidden space code; training the initial cycle generation countermeasure network according to the first source domain hidden space coding, the second source domain hidden space coding, the first target domain hidden space coding and the second target domain hidden space coding to obtain the cycle generation countermeasure network, and storing the cycle generation countermeasure network in the electronic device.
6. An electronic device, comprising one or more storage media and one or more processors in communication with the storage media, wherein the one or more storage media store processor-executable machine-executable instructions that, when executed by the electronic device, are executed by the processor to implement the image conversion method of any of claims 1-4.
7. A readable storage medium having stored thereon machine executable instructions which, when executed, implement the image conversion method of any one of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910588382.7A CN110276811B (en) | 2019-07-02 | 2019-07-02 | Image conversion method and device, electronic equipment and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910588382.7A CN110276811B (en) | 2019-07-02 | 2019-07-02 | Image conversion method and device, electronic equipment and readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110276811A CN110276811A (en) | 2019-09-24 |
| CN110276811B true CN110276811B (en) | 2022-11-01 |
Family
ID=67963796
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910588382.7A Active CN110276811B (en) | 2019-07-02 | 2019-07-02 | Image conversion method and device, electronic equipment and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110276811B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111539961A (en) * | 2019-12-13 | 2020-08-14 | 山东浪潮人工智能研究院有限公司 | Target segmentation method, device and equipment |
| CN111582449B (en) * | 2020-05-07 | 2023-08-04 | 广州视源电子科技股份有限公司 | Training method, device, equipment and storage medium of target domain detection network |
| CN113761831B (en) * | 2020-11-13 | 2024-05-21 | 北京沃东天骏信息技术有限公司 | Style handwriting generation method, device, equipment and storage medium |
| CN115205694A (en) * | 2021-03-26 | 2022-10-18 | 北京沃东天骏信息技术有限公司 | Image segmentation method, device and computer readable storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108171320A (en) * | 2017-12-06 | 2018-06-15 | 西安工业大学 | A kind of image area switching network and conversion method based on production confrontation network |
| CN108182657A (en) * | 2018-01-26 | 2018-06-19 | 深圳市唯特视科技有限公司 | A kind of face-image conversion method that confrontation network is generated based on cycle |
| CN108875935A (en) * | 2018-06-11 | 2018-11-23 | 兰州理工大学 | Based on the natural image target materials visual signature mapping method for generating confrontation network |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10474929B2 (en) * | 2017-04-25 | 2019-11-12 | Nec Corporation | Cyclic generative adversarial network for unsupervised cross-domain image generation |
-
2019
- 2019-07-02 CN CN201910588382.7A patent/CN110276811B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108171320A (en) * | 2017-12-06 | 2018-06-15 | 西安工业大学 | A kind of image area switching network and conversion method based on production confrontation network |
| CN108182657A (en) * | 2018-01-26 | 2018-06-19 | 深圳市唯特视科技有限公司 | A kind of face-image conversion method that confrontation network is generated based on cycle |
| CN108875935A (en) * | 2018-06-11 | 2018-11-23 | 兰州理工大学 | Based on the natural image target materials visual signature mapping method for generating confrontation network |
Non-Patent Citations (1)
| Title |
|---|
| 基于改进的CycleGAN模型非配对的图像到图像转换;何剑华等;《玉林师范学院学报》;20180401(第02期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110276811A (en) | 2019-09-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110276811B (en) | Image conversion method and device, electronic equipment and readable storage medium | |
| JP7476631B2 (en) | Image coding method and apparatus, and image decoding method and apparatus | |
| CN108038546B (en) | Method and apparatus for compressing neural networks | |
| CN108090521B (en) | Image fusion method and discriminator of generative confrontation network model | |
| CN110704547B (en) | Relation extraction data generation method, model and training method based on neural network | |
| CN111611152B (en) | Test case generation method and device, electronic equipment and readable storage medium | |
| KR102549617B1 (en) | 3d point cloud data encoding/decoding method and apparatus | |
| CN111144457B (en) | Image processing method, device, equipment and storage medium | |
| CN111275054A (en) | Image processing method, image processing device, electronic equipment and storage medium | |
| CN107451106A (en) | Text method and device for correcting, electronic equipment | |
| CN111737134B (en) | Chip testing method and device, electronic equipment and storage medium | |
| JP2019521550A5 (en) | ||
| JP2024517359A (en) | Method, apparatus, computer device and computer program for training an image processing network | |
| CN115830723A (en) | Correlation method and correlation device for training set images | |
| CN116505954B (en) | Huffman coding method, system, device and medium | |
| CN110796115B (en) | Image detection method and device, electronic equipment and readable storage medium | |
| WO2021184380A1 (en) | Point cloud encoding method and decoding method, encoder and decoder, and storage medium | |
| CN115934484B (en) | Diffusion model data enhancement-based anomaly detection method, storage medium and apparatus | |
| CN113032374A (en) | Data processing method, device, medium and equipment | |
| Duda et al. | Image-like 2d barcodes using generalizations of the Kuznetsov–Tsybakov problem | |
| US20170117918A1 (en) | Method and Apparatus for Calculating Estimated Data Compression Ratio | |
| CN113627567B (en) | Picture processing method, text processing method, related device and storage medium | |
| CN115718696B (en) | Source code cryptography misuse detection method and device, electronic equipment and storage medium | |
| CN111652004B (en) | Fusion method and device for machine translation system | |
| CN111274644B (en) | Wall sorting method and device and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |