CN112991191A - Face image enhancement method and device and electronic equipment - Google Patents
Face image enhancement method and device and electronic equipment Download PDFInfo
- Publication number
- CN112991191A CN112991191A CN201911297530.6A CN201911297530A CN112991191A CN 112991191 A CN112991191 A CN 112991191A CN 201911297530 A CN201911297530 A CN 201911297530A CN 112991191 A CN112991191 A CN 112991191A
- Authority
- CN
- China
- Prior art keywords
- image
- local
- face image
- images
- enhancement
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 75
- 238000012545 processing Methods 0.000 claims abstract description 95
- 238000003709 image segmentation Methods 0.000 claims abstract description 31
- 238000012549 training Methods 0.000 claims description 81
- 238000004458 analytical method Methods 0.000 claims description 50
- 230000011218 segmentation Effects 0.000 claims description 27
- 238000004590 computer program Methods 0.000 claims description 13
- 238000003062 neural network model Methods 0.000 claims description 12
- 210000004709 eyebrow Anatomy 0.000 claims description 7
- 210000004209 hair Anatomy 0.000 claims description 6
- 230000003416 augmentation Effects 0.000 claims description 3
- 230000000694 effects Effects 0.000 abstract description 12
- 230000000007 visual effect Effects 0.000 abstract description 12
- 238000010586 diagram Methods 0.000 description 11
- 230000009466 transformation Effects 0.000 description 9
- 230000002708 enhancing effect Effects 0.000 description 8
- 238000004891 communication Methods 0.000 description 7
- 238000013135 deep learning Methods 0.000 description 7
- 230000008569 process Effects 0.000 description 5
- 230000007704 transition Effects 0.000 description 4
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 230000004927 fusion Effects 0.000 description 3
- 238000007499 fusion processing Methods 0.000 description 3
- 230000001815 facial effect Effects 0.000 description 2
- 210000000056 organ Anatomy 0.000 description 2
- 210000000697 sensory organ Anatomy 0.000 description 2
- 230000003044 adaptive effect Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000006835 compression Effects 0.000 description 1
- 238000007906 compression Methods 0.000 description 1
- 238000000802 evaporation-induced self-assembly Methods 0.000 description 1
- 210000000887 face Anatomy 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/04—Context-preserving transformations, e.g. by using an importance map
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4038—Image mosaicing, e.g. composing plane images from plane sub-images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/11—Region-based segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/155—Segmentation; Edge detection involving morphological operators
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30196—Human being; Person
- G06T2207/30201—Face
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Image Analysis (AREA)
- Image Processing (AREA)
Abstract
The invention provides a face image enhancement method, a face image enhancement device and electronic equipment, and relates to the technical field of image processing, wherein the method comprises the following steps: carrying out image segmentation processing on an original face image to be processed to obtain a plurality of target local images; respectively carrying out image enhancement processing on each target local image to obtain a local enhanced image corresponding to each target local image; acquiring an initial enhanced image obtained by performing image enhancement processing on an original face image; and carrying out image splicing processing on the initial enhanced image and each local enhanced image to obtain a target enhanced image corresponding to the original face image. The human face image enhancement method, the human face image enhancement device and the electronic equipment improve the enhancement effect of all parts of the human face in the human face image, so that the overall visual effect of the human face image is improved.
Description
Technical Field
The present invention relates to the field of image processing technologies, and in particular, to a method and an apparatus for enhancing a face image, and an electronic device.
Background
The image enhancement is to enhance useful information of an image, improve the resolution of the image, improve the visual experience effect of the image, purposefully emphasize local and overall features of the image, inhibit uninteresting features, and enable the image to meet specific requirements. Human face image enhancement technologies, such as face beautification, image restoration, old photo beautification and the like, have been adopted in many application scenes, and the human face image enhancement can improve the quality of human face images, so that the human face images are not only clearer in vision, but also the images are more beneficial to the processing and recognition of a computer.
At present, the face image enhancement methods are mainly divided into two types, one type adopts a traditional face image enhancement algorithm, for example, a method of adopting sequence image fusion or a nonlinear transformation algorithm based on gamma transformation and logarithmic transformation, and the other type adopts a face image enhancement algorithm based on deep learning.
However, the feature information extracted by the traditional face image enhancement algorithm is limited, and the general performance is not good enough; however, the existing face image enhancement algorithm based on deep learning can generate too complex feature information when the whole face image is input into the corresponding network model, and can not accurately enhance all parts of the face in the face image at the same time, resulting in poor overall visual effect of the enhanced face image.
Disclosure of Invention
The invention aims to provide a face image enhancement method, a face image enhancement device and electronic equipment, so as to improve the enhancement effect of all parts of a face in a face image and further improve the overall visual effect of the face image.
The embodiment of the invention provides a face image enhancement method, which comprises the following steps:
carrying out image segmentation processing on an original face image to be processed to obtain a plurality of target local images;
respectively carrying out image enhancement processing on each target local image to obtain a local enhanced image corresponding to each target local image;
acquiring an initial enhanced image obtained by performing image enhancement processing on the original face image;
and carrying out image splicing processing on the initial enhanced image and each local enhanced image to obtain a target enhanced image corresponding to the original face image.
Further, the image segmentation processing is performed on the original face image to be processed to obtain a plurality of target local images, and the method includes:
inputting an original face image to be processed into a pre-trained segmentation model to obtain a plurality of target local images output by the segmentation model; the segmentation model is used for carrying out segmentation processing on an input image.
Further, the image segmentation processing is performed on the original face image to be processed to obtain a plurality of target local images, and the method includes:
inputting an original face image to be processed into a pre-trained face analysis model to obtain a plurality of local analysis images output by the face analysis model; wherein the local analytic image comprises one or more of a mouth analytic image, an eye analytic image, a nose analytic image, a hair analytic image, an eyebrow analytic image, and an ear analytic image;
performing closed operation processing on each local analysis image to obtain a plurality of optimized local analysis images;
acquiring regional position information of each optimized local analysis image in the original face image;
and respectively carrying out image interception in the original face image according to the area position information corresponding to each optimized local analysis image to obtain a plurality of target local images.
Further, the obtaining of the region position information of each optimized local analysis image in the original face image includes:
searching a circumscribed rectangle of each optimized local analytic image;
and determining the position coordinate information of the circumscribed rectangle of each optimized local analysis image as the region position information of the optimized local analysis image in the original face image.
Further, the performing image enhancement processing on each of the target local images to obtain a local enhanced image corresponding to each of the target local images includes:
inputting each target local image into a local enhancement model trained in advance to obtain a plurality of local enhancement images output by the local enhancement model; wherein the local augmentation model comprises a deep neural network model.
Further, before performing image enhancement processing on each of the target local images to obtain a locally enhanced image corresponding to each of the target local images, the method further includes:
acquiring a plurality of first training face images;
respectively adjusting the resolution of each first training face image to obtain a second training face image; wherein the resolution of the second training face image is less than the resolution of the first training face image;
respectively carrying out image segmentation processing on each first training face image and each second training face image to obtain a local image corresponding to each first training face image and a local image corresponding to each second training face image;
and training the initial local enhancement model to be trained by using the local images corresponding to the first training face images and the local images corresponding to the second training face images to obtain the trained local enhancement model.
Further, before performing image segmentation processing on each of the second training face images, the method further includes:
and performing noise addition processing and fuzzy processing on each second training face image, and taking the processed second training face image as a second training face image subjected to image segmentation processing.
Further, the image stitching processing on the initial enhanced image and each of the local enhanced images to obtain a target enhanced image corresponding to the original face image includes:
acquiring region position information corresponding to each local enhanced image;
and splicing each local enhanced image in a corresponding region in the initial enhanced image according to the region position information corresponding to each local enhanced image to obtain a target enhanced image corresponding to the original face image.
The embodiment of the invention also provides a face image enhancement device, which comprises:
the segmentation module is used for carrying out image segmentation processing on an original face image to be processed to obtain a plurality of target local images;
the enhancement module is used for respectively carrying out image enhancement processing on each target local image to obtain a local enhanced image corresponding to each target local image;
the acquisition module is used for acquiring an initial enhanced image after the image enhancement processing is carried out on the original face image;
and the splicing module is used for carrying out image splicing processing on the initial enhanced image and each local enhanced image to obtain a target enhanced image corresponding to the original face image.
The embodiment of the invention also provides electronic equipment which comprises a memory and a processor, wherein the memory stores a computer program capable of running on the processor, and the processor realizes the human face image enhancement method when executing the computer program.
The embodiment of the invention also provides a computer readable storage medium, wherein a computer program is stored on the computer readable storage medium, and when the computer program is executed by a processor, the method for enhancing the face image is executed.
In the face image enhancement method, device and electronic equipment provided by the embodiment of the invention, the method comprises the following steps: carrying out image segmentation processing on an original face image to be processed to obtain a plurality of target local images; respectively carrying out image enhancement processing on each target local image to obtain a local enhanced image corresponding to each target local image; acquiring an initial enhanced image obtained by performing image enhancement processing on an original face image; and carrying out image splicing processing on the initial enhanced image and each local enhanced image to obtain a target enhanced image corresponding to the original face image. Therefore, a plurality of target local images which are relatively concerned by a user are obtained through image segmentation processing, targeted enhancement of all parts of the face is achieved through image enhancement processing of the plurality of target local images, and finally the target enhanced images are obtained through image splicing processing of the initial enhanced images and all the local enhanced images, so that natural transition of spliced boundaries in the target enhanced images can be guaranteed. Compared with the prior art that the image enhancement processing is directly carried out on the whole original face image, the face image enhancement method, the face image enhancement device and the electronic equipment provided by the embodiment of the invention improve the enhancement effect of all parts of the face in the face image, thereby improving the overall visual effect of the face image.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts.
Fig. 1 is a schematic flow chart of a method for enhancing a face image according to an embodiment of the present invention;
fig. 2 is a schematic flow chart of another human face image enhancement method according to an embodiment of the present invention;
fig. 3 is a schematic diagram of a training process of a local augmentation model according to an embodiment of the present invention;
fig. 4 is a schematic structural diagram of a face image enhancement device according to an embodiment of the present invention;
fig. 5 is a schematic structural diagram of another face image enhancement device according to an embodiment of the present invention;
fig. 6 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
The technical solutions of the present invention will be described clearly and completely with reference to the following embodiments, and it should be understood that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The existing human face image enhancement algorithm based on deep learning can not accurately enhance all parts of human faces in a human face image, so that the overall visual effect of the enhanced human face image is poor. Based on this, the method, the device and the electronic device for enhancing the face image provided by the embodiment of the invention can improve the enhancement effect of each part of the face in the face image by using the image enhancement technology based on multi-region feature understanding, thereby improving the overall visual effect of the face image. Wherein, the characteristic understanding refers to analyzing the data and the information contained therein, and further understanding the characteristics and the underlying structure of the data.
To facilitate understanding of the embodiment, a detailed description will be first given of a face image enhancement method disclosed in the embodiment of the present invention.
The embodiment of the invention provides a face image enhancement method, which can be executed by an electronic device with image processing capability, wherein the electronic device can be any one of the following devices: desktop computers, notebook computers, tablet computers, smart phones, and the like.
Referring to fig. 1, a schematic flow chart of a method for enhancing a face image mainly includes the following steps S102 to S108:
step S102, image segmentation processing is carried out on an original face image to be processed, and a plurality of target local images are obtained.
The original face image to be processed may be a low-resolution face image, or may be a face image with certain noise and/or blur. By carrying out image segmentation processing on an original face image to be processed, a plurality of target local images which are relatively concerned by a user, such as a mouth image, an eye image, a nose image, an eyebrow image and the like, are obtained.
And step S104, respectively carrying out image enhancement processing on each target local image to obtain a local enhanced image corresponding to each target local image.
The image enhancement processing may be performed on each target local image by using an existing image enhancement method (for example, a method of sequence image fusion, a nonlinear transformation algorithm based on gamma transformation and logarithmic transformation, or a face image enhancement algorithm based on deep learning), so as to obtain a locally enhanced image corresponding to each target local image. This enables a multi-area feature understanding.
And step S106, acquiring an initial enhanced image obtained by performing image enhancement on the original face image.
The original face image can be subjected to image enhancement processing by adopting an existing image enhancement method (for example, a method of sequence image fusion, a nonlinear transformation algorithm based on gamma transformation and logarithmic transformation, or a face image enhancement algorithm based on deep learning), so as to obtain an initial enhanced image. The enhancement effect of the region of the initial enhanced image where the user is more focused is not ideal enough to meet the enhancement requirement of the user, so the initial enhanced image needs to be optimized through the following step S108.
And S108, carrying out image splicing processing on the initial enhanced image and each local enhanced image to obtain a target enhanced image corresponding to the original face image.
The local enhanced images can be spliced in the corresponding area of the initial enhanced image to obtain the target enhanced image corresponding to the original face image, so that the enhancement effect of the local area concerned by a user is enhanced on the basis of the initial enhanced image, and the overall visual effect of the face image is improved.
In addition, boundary fusion processing can be performed during image stitching, and a specific boundary fusion processing method can refer to the related prior art and is not described herein again. The transition of the target enhanced image obtained by the boundary fusion processing at the splicing boundary is more natural, and the overall visual effect of the human face image is further improved.
In the embodiment of the invention, a plurality of target local images which are relatively concerned by a user are obtained through image segmentation processing, targeted enhancement of all parts of a face is realized through image enhancement processing on the plurality of target local images, and finally, the target enhanced images are obtained through image splicing processing on the initial enhanced images and all the local enhanced images, so that natural transition at the splicing boundary in the target enhanced images can be ensured. Compared with the prior art that the image enhancement processing is directly carried out on the whole original face image, the face image enhancement method provided by the embodiment of the invention improves the enhancement effect of each part of the face in the face image, thereby improving the overall visual effect of the face image.
For convenience of understanding, the embodiment of the present invention further provides a specific implementation process of the face image enhancement method, and details are performed on each step in fig. 1. Referring to fig. 2, a schematic flow chart of another method for enhancing a face image, which employs an image enhancement technique based on deep learning, includes the following steps S202 to S210:
step S202, inputting an original face image to be processed into a pre-trained segmentation model to obtain a plurality of target local images output by the segmentation model; the segmentation model is used for carrying out segmentation processing on an input image.
The segmentation model can be obtained by training on the electronic device to which the method is applied, or can be deployed on the electronic device after being trained on other devices.
Step S204, inputting each target local image into a local enhancement model trained in advance to obtain a plurality of local enhancement images output by the local enhancement model; wherein the locally enhanced model comprises a deep neural network model.
The local enhancement model can be obtained by training on the electronic device to which the method is applied, or can be deployed on the electronic device after being trained on other devices. Regional information (each target local image) of different regions in the original face image can be input into the deep neural network model, and an enhanced image (local enhanced image) of each region output by the deep neural network model is obtained. The deep neural network model can be understood to include a plurality of pre-trained enhancement models, the enhancement models correspond to the target local images one to one, and each enhancement model only processes the corresponding target local image.
And step S206, carrying out image enhancement processing on the original face image by adopting a face image enhancement algorithm based on deep learning to obtain an initial enhanced image.
Step S208, acquiring the corresponding region position information of each local enhanced image.
For each locally enhanced image, the region position information corresponding to the optimized locally analyzed image corresponding to the locally enhanced image may be used as the region position information corresponding to the locally enhanced image.
And step S210, splicing each local enhanced image into a corresponding area in the initial enhanced image according to the area position information corresponding to each local enhanced image to obtain a target enhanced image corresponding to the original face image.
Under the condition that the position information of the region corresponding to each local enhanced image is known, each local enhanced image can be used for replacing the corresponding region in the initial enhanced image so as to realize the splicing of each local enhanced image and the initial enhanced image. The target enhanced image obtained in the way is composed of each local enhanced image and the initial enhanced image except the corresponding area of each local enhanced image, and the enhancement effect of the target local image in the original face image is improved.
The human face image enhancement method based on multi-region feature understanding provided by the embodiment of the invention can realize more accurate enhancement of information such as facial features and the like by processing the multi-region feature understanding of the original human face image, and improves the whole visual effect of the human face while enhancing the resolution.
The embodiment of the present invention further provides another possible implementation manner of the step S102, which is as follows: firstly, inputting an original face image to be processed into a pre-trained face analysis model to obtain a plurality of local analysis images output by the face analysis model; wherein the local analytic images comprise one or more of mouth analytic images, eye analytic images, nose analytic images, hair analytic images, eyebrow analytic images and ear analytic images; then, performing closed operation processing on each local analysis image to obtain a plurality of optimized local analysis images; then obtaining the regional position information of each optimized local analysis image in the original face image; and finally, respectively carrying out image interception in the original face image according to the area position information corresponding to each optimized local analysis image to obtain a plurality of target local images.
In specific implementation, the face analysis model can be obtained by training a Helen five-sense organ data set through a Deeplab-v3 segmentation network, wherein the Helen five-sense organ data set is a data set used for a face feature point task, and the Deeplab-v3 segmentation network is a semantic segmentation network. Only the image of the required area is reserved in the local analysis image, and other areas are empty. The local analysis image may have a hole, and therefore the hole in the local analysis image needs to be filled through closed operation processing, so that the five sense organs (optimized local analysis image) obtained through analysis is more accurate.
The region position information of each optimized local analysis image in the original face image can be obtained through the following processes: searching a circumscribed rectangle of each optimized local analytic image; and determining the position coordinate information of the circumscribed rectangle of each optimized local analysis image as the region position information of the optimized local analysis image in the original face image. Specifically, the optimized local analysis image is in a two-dimensional matrix form, and if the position coordinates of each pixel point in the image are set to (x, y), the maximum value and the minimum value of x and the maximum value and the minimum value of y in the optimized local analysis image are detected, so that the circumscribed rectangle of the optimized local analysis image can be obtained (the position coordinates of four vertexes of the circumscribed rectangle are composed of the maximum value and the minimum value of x and the maximum value and the minimum value of y). The region position information corresponding to the optimized local analysis image may include position coordinates of four vertices of the circumscribed rectangle.
According to the image segmentation processing mode based on the face analysis model, when the original face image is segmented, segmentation is more accurate, so that more accurate enhancement of information such as the five sense organs of the follow-up face is facilitated.
In addition, an embodiment of the present invention further provides a training method for a local enhancement model, and referring to a schematic training flow diagram of a local enhancement model shown in fig. 3, the local enhancement model is obtained through training by the following steps:
step S302, a plurality of first training face images are obtained.
A high definition image data set is prepared, the high definition image data set comprising a plurality of first training face images, the first training face images preferably being high resolution (greater than a set resolution threshold) high definition (greater than a set definition threshold). The specific resolution threshold and the sharpness threshold may be set according to actual requirements, and are not limited herein.
Step S304, respectively adjusting the resolution of each first training face image to obtain a second training face image; and the resolution of the second training face image is smaller than that of the first training face image.
The down-sampling processing can be performed on the high-definition image data set to obtain a low-resolution image data set, obviously, the resolution of the image in the low-resolution image data set is smaller than the resolution of the corresponding image in the high-definition image data set, and the image in the low-resolution image data set can be used as a second training face image.
Preferably, in order to improve the training effect of the local enhancement model, various noises (for example, gaussian noise, image compression noise, and the like) may be added to the low-resolution image data set, and gaussian blur with different intensities may be randomly added to obtain a low-definition image data set, and an image in the low-definition image data set is used as a second training face image for performing image segmentation processing subsequently. Based on this, before performing step S306, the method further includes: and carrying out noise addition processing and fuzzy processing on each second training face image, and taking the processed second training face image as a second training face image for image segmentation processing.
Step S306, image segmentation processing is respectively carried out on each first training face image and each second training face image, and a local image corresponding to each first training face image and a local image corresponding to each second training face image are obtained.
For a specific image segmentation method, reference may be made to corresponding contents in the foregoing embodiments, which are not described herein again. In specific implementation, through image segmentation processing, a mouth data set, an eye data set, a nose data set, a hair data set, an ear data set, an eyebrow data set and the like corresponding to a high-definition image data set, and a mouth data set, an eye data set, a nose data set, a hair data set, an ear data set, an eyebrow data set and the like corresponding to a low-definition image data set can be obtained.
Step S308, training the initial local enhancement model to be trained by using the local images corresponding to the first training face images and the local images corresponding to the second training face images to obtain a trained local enhancement model.
And the local images corresponding to the second training face images are input data of the initial local enhancement model to be trained, and the initial local enhancement model continuously adjusts model parameters by comparing the output data with the local images corresponding to the first training face images, so that the training of the initial local enhancement model is realized.
In a specific implementation, the local enhancement model may adopt a deep neural network model, such as an Enhanced Super-Resolution adaptive network (ESRGAN) model. A deep neural network model for generating images is built, a pixel shuffle upsampling method can be adopted in the deep neural network model, a method for connecting a front-layer output and a deep-layer output is combined, and meanwhile, a weight normalization method can be used for designing the deep neural network model. The method for connecting the front-layer output with the deep-layer output means that a deep neural network model can build a plurality of levels, each level has own input data and output data, and the output data of the previous level is generally the input data of the next level. The method for connecting the front-layer output and the deep-layer output by combining the pixel buffer upsampling method means that assuming that the Nth layer finishes one upsampling operation, the output data of the (N-1) th layer is processed by the pixel buffer upsampling method to obtain first output data, and the first output data and the output data of the Nth layer are merged (concat) to be used as input data of the (N + 1) th layer.
Corresponding to the face image enhancement method, the embodiment of the invention also provides a face image enhancement device. Referring to fig. 4, a schematic structural diagram of a face image enhancement apparatus is shown, the apparatus includes:
the segmentation module 42 is configured to perform image segmentation on an original face image to be processed to obtain a plurality of target local images;
the enhancement module 44 is configured to perform image enhancement processing on each target local image to obtain a local enhanced image corresponding to each target local image;
an obtaining module 46, configured to obtain an initial enhanced image obtained by performing image enhancement on an original face image;
and the splicing module 48 is configured to perform image splicing processing on the initial enhanced image and each local enhanced image to obtain a target enhanced image corresponding to the original face image.
In the embodiment of the invention, a plurality of target local images which are relatively concerned by a user are obtained through image segmentation processing, targeted enhancement of all parts of a face is realized through image enhancement processing on the plurality of target local images, and finally, the target enhanced images are obtained through image splicing processing on the initial enhanced images and all the local enhanced images, so that natural transition at the splicing boundary in the target enhanced images can be ensured. Compared with the prior art that the image enhancement processing is directly carried out on the whole original face image, the face image enhancement device provided by the embodiment of the invention improves the enhancement effect of all parts of the face in the face image, thereby improving the overall visual effect of the face image.
In some possible embodiments, the segmentation module 42 is specifically configured to: inputting an original face image to be processed into a pre-trained segmentation model to obtain a plurality of target local images output by the segmentation model; the segmentation model is used for carrying out segmentation processing on an input image.
In other possible embodiments, the segmentation module 42 is specifically configured to: inputting an original face image to be processed into a pre-trained face analysis model to obtain a plurality of local analysis images output by the face analysis model; wherein the local analytic images comprise one or more of mouth analytic images, eye analytic images, nose analytic images, hair analytic images, eyebrow analytic images and ear analytic images; performing closed operation processing on each local analysis image to obtain a plurality of optimized local analysis images; acquiring regional position information of each optimized local analysis image in an original face image; and respectively carrying out image interception in the original face image according to the area position information corresponding to each optimized local analysis image to obtain a plurality of target local images.
Further, the segmentation module 42 is further configured to: searching a circumscribed rectangle of each optimized local analytic image; and determining the position coordinate information of the circumscribed rectangle of each optimized local analysis image as the region position information of the optimized local analysis image in the original face image.
Further, the enhancing module 44 is specifically configured to: inputting each target local image into a local enhancement model trained in advance to obtain a plurality of local enhancement images output by the local enhancement model; wherein the locally enhanced model comprises a deep neural network model.
Further, the splicing module 48 is specifically configured to: acquiring region position information corresponding to each local enhanced image; and splicing each local enhanced image into a corresponding area in the initial enhanced image according to the area position information corresponding to each local enhanced image to obtain a target enhanced image corresponding to the original face image.
Optionally, referring to a schematic structural diagram of another facial image enhancement apparatus shown in fig. 5, on the basis of fig. 4, the apparatus further includes a training module 52, where the training module 52 is configured to: acquiring a plurality of first training face images; respectively adjusting the resolution of each first training face image to obtain a second training face image; the resolution of the second training face image is smaller than that of the first training face image; respectively carrying out image segmentation processing on each first training face image and each second training face image to obtain a local image corresponding to each first training face image and a local image corresponding to each second training face image; and training the initial local enhancement model to be trained by using the local images corresponding to the first training face images and the local images corresponding to the second training face images to obtain a trained local enhancement model.
Further, before performing the image segmentation processing on each second training face image, the training module 52 is further configured to: and carrying out noise addition processing and fuzzy processing on each second training face image, and taking the processed second training face image as a second training face image for image segmentation processing.
The device provided by the embodiment has the same implementation principle and technical effect as the method embodiments, and for the sake of brief description, reference may be made to the corresponding contents in the method embodiments without reference to the device embodiments.
Referring to fig. 6, an embodiment of the present invention further provides an electronic device 100, including: a processor 60, a memory 61, a bus 62 and a communication interface 63, wherein the processor 60, the communication interface 63 and the memory 61 are connected through the bus 62; the processor 60 is arranged to execute executable modules, such as computer programs, stored in the memory 61.
The Memory 61 may include a high-speed Random Access Memory (RAM) and may also include a non-volatile Memory (non-volatile Memory), such as at least one disk Memory. The communication connection between the network element of the system and at least one other network element is realized through at least one communication interface 63 (which may be wired or wireless), and the internet, a wide area network, a local network, a metropolitan area network, and the like can be used.
The bus 62 may be an ISA bus, PCI bus, EISA bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one double-headed arrow is shown in FIG. 6, but that does not indicate only one bus or one type of bus.
The memory 61 is used for storing a program, the processor 60 executes the program after receiving an execution instruction, and the method executed by the apparatus defined by the flow process disclosed in any of the foregoing embodiments of the present invention may be applied to the processor 60, or implemented by the processor 60.
The processor 60 may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the above method may be performed by integrated logic circuits of hardware or instructions in the form of software in the processor 60. The Processor 60 may be a general-purpose Processor, and includes a Central Processing Unit (CPU), a Network Processor (NP), and the like; the device can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components. The various methods, steps and logic blocks disclosed in the embodiments of the present invention may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in connection with the embodiments of the present invention may be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software module may be located in ram, flash memory, rom, prom, or eprom, registers, etc. storage media as is well known in the art. The storage medium is located in a memory 61, and the processor 60 reads the information in the memory 61 and, in combination with its hardware, performs the steps of the above method.
The embodiment of the present invention further provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the computer program performs the face image enhancement method described in the foregoing method embodiment. The computer-readable storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
In all examples shown and described herein, any particular value should be construed as merely exemplary, and not as a limitation, and thus other examples of example embodiments may have different values.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
In the several embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other ways. The above-described embodiments of the apparatus are merely illustrative, and for example, the division of the units is only one logical division, and there may be other divisions when actually implemented, and for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection of devices or units through some communication interfaces, and may be in an electrical, mechanical or other form.
In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit.
Finally, it should be noted that: the above embodiments are only used to illustrate the technical solution of the present invention, and not to limit the same; while the invention has been described in detail and with reference to the foregoing embodiments, it will be understood by those skilled in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some or all of the technical features may be equivalently replaced; and the modifications or the substitutions do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present invention.
Claims (11)
1. A face image enhancement method is characterized by comprising the following steps:
carrying out image segmentation processing on an original face image to be processed to obtain a plurality of target local images;
respectively carrying out image enhancement processing on each target local image to obtain a local enhanced image corresponding to each target local image;
acquiring an initial enhanced image obtained by performing image enhancement processing on the original face image;
and carrying out image splicing processing on the initial enhanced image and each local enhanced image to obtain a target enhanced image corresponding to the original face image.
2. The method according to claim 1, wherein the image segmentation processing on the original face image to be processed to obtain a plurality of target partial images comprises:
inputting an original face image to be processed into a pre-trained segmentation model to obtain a plurality of target local images output by the segmentation model; the segmentation model is used for carrying out segmentation processing on an input image.
3. The method according to claim 1, wherein the image segmentation processing on the original face image to be processed to obtain a plurality of target partial images comprises:
inputting an original face image to be processed into a pre-trained face analysis model to obtain a plurality of local analysis images output by the face analysis model; wherein the local analytic image comprises one or more of a mouth analytic image, an eye analytic image, a nose analytic image, a hair analytic image, an eyebrow analytic image, and an ear analytic image;
performing closed operation processing on each local analysis image to obtain a plurality of optimized local analysis images;
acquiring regional position information of each optimized local analysis image in the original face image;
and respectively carrying out image interception in the original face image according to the area position information corresponding to each optimized local analysis image to obtain a plurality of target local images.
4. The method according to claim 3, wherein the obtaining of the region position information of each optimized local analysis image in the original face image comprises:
searching a circumscribed rectangle of each optimized local analytic image;
and determining the position coordinate information of the circumscribed rectangle of each optimized local analysis image as the region position information of the optimized local analysis image in the original face image.
5. The method according to claim 1, wherein the performing image enhancement processing on each of the target local images to obtain a locally enhanced image corresponding to each of the target local images includes:
inputting each target local image into a local enhancement model trained in advance to obtain a plurality of local enhancement images output by the local enhancement model; wherein the local augmentation model comprises a deep neural network model.
6. The method according to claim 5, before performing image enhancement processing on each of the target local images to obtain a locally enhanced image corresponding to each of the target local images, the method further comprising:
acquiring a plurality of first training face images;
respectively adjusting the resolution of each first training face image to obtain a second training face image; wherein the resolution of the second training face image is less than the resolution of the first training face image;
respectively carrying out image segmentation processing on each first training face image and each second training face image to obtain a local image corresponding to each first training face image and a local image corresponding to each second training face image;
and training the initial local enhancement model to be trained by using the local images corresponding to the first training face images and the local images corresponding to the second training face images to obtain the trained local enhancement model.
7. The method of claim 6, wherein prior to performing image segmentation processing on each of the second training face images, the method further comprises:
and performing noise addition processing and fuzzy processing on each second training face image, and taking the processed second training face image as a second training face image subjected to image segmentation processing.
8. The method according to claim 1, wherein the image stitching processing on the initial enhanced image and each of the local enhanced images to obtain a target enhanced image corresponding to the original face image comprises:
acquiring region position information corresponding to each local enhanced image;
and splicing each local enhanced image in a corresponding region in the initial enhanced image according to the region position information corresponding to each local enhanced image to obtain a target enhanced image corresponding to the original face image.
9. A face image enhancement apparatus, comprising:
the segmentation module is used for carrying out image segmentation processing on an original face image to be processed to obtain a plurality of target local images;
the enhancement module is used for respectively carrying out image enhancement processing on each target local image to obtain a local enhanced image corresponding to each target local image;
the acquisition module is used for acquiring an initial enhanced image after the image enhancement processing is carried out on the original face image;
and the splicing module is used for carrying out image splicing processing on the initial enhanced image and each local enhanced image to obtain a target enhanced image corresponding to the original face image.
10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method of any one of claims 1-8 when executing the computer program.
11. A computer-readable storage medium, having stored thereon a computer program, characterized in that the computer program, when being executed by a processor, is adapted to carry out the method of any one of claims 1-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911297530.6A CN112991191A (en) | 2019-12-13 | 2019-12-13 | Face image enhancement method and device and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911297530.6A CN112991191A (en) | 2019-12-13 | 2019-12-13 | Face image enhancement method and device and electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112991191A true CN112991191A (en) | 2021-06-18 |
Family
ID=76341759
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911297530.6A Pending CN112991191A (en) | 2019-12-13 | 2019-12-13 | Face image enhancement method and device and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112991191A (en) |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103246880A (en) * | 2013-05-15 | 2013-08-14 | 中国科学院自动化研究所 | Human face recognizing method based on multi-level local obvious mode characteristic counting |
| CN106203269A (en) * | 2016-06-29 | 2016-12-07 | 武汉大学 | A kind of based on can the human face super-resolution processing method of deformation localized mass and system |
| CN107516083A (en) * | 2017-08-29 | 2017-12-26 | 电子科技大学 | A kind of remote facial image Enhancement Method towards identification |
| CN107895358A (en) * | 2017-12-25 | 2018-04-10 | 科大讯飞股份有限公司 | The Enhancement Method and system of facial image |
| KR20180109217A (en) * | 2017-03-27 | 2018-10-08 | 삼성전자주식회사 | Method for enhancing face image and electronic device for the same |
| CN109102483A (en) * | 2018-07-24 | 2018-12-28 | 厦门美图之家科技有限公司 | Image enhancement model training method, device, electronic equipment and readable storage medium storing program for executing |
| CN109242760A (en) * | 2018-08-16 | 2019-01-18 | Oppo广东移动通信有限公司 | Processing method, device and the electronic equipment of facial image |
| CN109493297A (en) * | 2018-11-01 | 2019-03-19 | 重庆中科云丛科技有限公司 | Low quality facial image Enhancement Method, system, equipment and storage medium |
| WO2019128508A1 (en) * | 2017-12-28 | 2019-07-04 | Oppo广东移动通信有限公司 | Method and apparatus for processing image, storage medium, and electronic device |
-
2019
- 2019-12-13 CN CN201911297530.6A patent/CN112991191A/en active Pending
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103246880A (en) * | 2013-05-15 | 2013-08-14 | 中国科学院自动化研究所 | Human face recognizing method based on multi-level local obvious mode characteristic counting |
| CN106203269A (en) * | 2016-06-29 | 2016-12-07 | 武汉大学 | A kind of based on can the human face super-resolution processing method of deformation localized mass and system |
| KR20180109217A (en) * | 2017-03-27 | 2018-10-08 | 삼성전자주식회사 | Method for enhancing face image and electronic device for the same |
| CN107516083A (en) * | 2017-08-29 | 2017-12-26 | 电子科技大学 | A kind of remote facial image Enhancement Method towards identification |
| CN107895358A (en) * | 2017-12-25 | 2018-04-10 | 科大讯飞股份有限公司 | The Enhancement Method and system of facial image |
| WO2019128508A1 (en) * | 2017-12-28 | 2019-07-04 | Oppo广东移动通信有限公司 | Method and apparatus for processing image, storage medium, and electronic device |
| CN109102483A (en) * | 2018-07-24 | 2018-12-28 | 厦门美图之家科技有限公司 | Image enhancement model training method, device, electronic equipment and readable storage medium storing program for executing |
| CN109242760A (en) * | 2018-08-16 | 2019-01-18 | Oppo广东移动通信有限公司 | Processing method, device and the electronic equipment of facial image |
| CN109493297A (en) * | 2018-11-01 | 2019-03-19 | 重庆中科云丛科技有限公司 | Low quality facial image Enhancement Method, system, equipment and storage medium |
Non-Patent Citations (4)
| Title |
|---|
| 公维军;吴建军;李晓霞;李晓旭;: "基于深度学习的局部方向人脸识别算法研究", 计算机与数字工程, no. 05 * |
| 吕蕊;宋晓晓;李清;王振;: "基于模糊算法的面部图像处理", 山东工业技术, no. 13, 1 July 2017 (2017-07-01) * |
| 王小明;方晓颖;刘锦高;: "复杂光照下的自适应人脸图像增强", 计算机工程与应用, no. 02, 11 January 2011 (2011-01-11) * |
| 许若波;卢涛;王宇;张彦铎;: "基于组合学习的人脸超分辨率算法", 计算机应用, no. 03 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10614574B2 (en) | Generating image segmentation data using a multi-branch neural network | |
| CN109493350B (en) | Portrait segmentation method and device | |
| KR102419136B1 (en) | Image processing apparatus and method using multiple-channel feature map | |
| CN108010031B (en) | Portrait segmentation method and mobile terminal | |
| CN112132156B (en) | Image saliency target detection method and system based on multi-depth feature fusion | |
| US8103058B2 (en) | Detecting and tracking objects in digital images | |
| CN111507909A (en) | Method and device for clearing fog image and storage medium | |
| CN111275034B (en) | Method, device, equipment and storage medium for extracting text region from image | |
| CN110084238B (en) | Finger vein image segmentation method and device based on LadderNet network and storage medium | |
| CN110675339A (en) | Image restoration method and system based on edge restoration and content restoration | |
| CN114238904B (en) | Identity recognition method, and training method and device of dual-channel hyper-resolution model | |
| CN112308866A (en) | Image processing method, image processing device, electronic equipment and storage medium | |
| CN116363261A (en) | Training method of image editing model, image editing method and device | |
| CN110619334A (en) | Portrait segmentation method based on deep learning, architecture and related device | |
| CN114022497A (en) | Image processing method and device | |
| CN111833360A (en) | Image processing method, device, equipment and computer readable storage medium | |
| CN111815606A (en) | Image quality evaluation method, storage medium, and computing device | |
| CN109615620B (en) | Image compression degree identification method, device, equipment and computer readable storage medium | |
| CN116485944A (en) | Image processing method and device, computer readable storage medium and electronic equipment | |
| CN115830362A (en) | Image processing method, apparatus, device, medium, and product | |
| CN113012030A (en) | Image splicing method, device and equipment | |
| CN112991191A (en) | Face image enhancement method and device and electronic equipment | |
| CN113221835B (en) | Scene classification method, device and equipment for surface review video and storage medium | |
| CN115471413A (en) | Image processing method and device, computer readable storage medium and electronic device | |
| CN114913588A (en) | Face image restoration and recognition method applied to complex scene |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |