CN111488836B - Face contour correction method, device, equipment and storage medium - Google Patents
Face contour correction method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN111488836B CN111488836B CN202010285509.0A CN202010285509A CN111488836B CN 111488836 B CN111488836 B CN 111488836B CN 202010285509 A CN202010285509 A CN 202010285509A CN 111488836 B CN111488836 B CN 111488836B
- Authority
- CN
- China
- Prior art keywords
- initial key
- key point
- point
- search
- correction
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000012937 correction Methods 0.000 title claims abstract description 149
- 238000000034 method Methods 0.000 title claims abstract description 55
- 230000001815 facial effect Effects 0.000 claims description 17
- 238000004590 computer program Methods 0.000 claims description 8
- 210000000697 sensory organ Anatomy 0.000 claims description 8
- 238000010606 normalization Methods 0.000 claims description 4
- 210000000056 organ Anatomy 0.000 claims description 2
- 239000000523 sample Substances 0.000 claims 1
- 238000001514 detection method Methods 0.000 abstract description 29
- 238000004364 calculation method Methods 0.000 abstract description 14
- 238000010586 diagram Methods 0.000 description 14
- 230000008901 benefit Effects 0.000 description 9
- 210000001508 eye Anatomy 0.000 description 8
- 210000000214 mouth Anatomy 0.000 description 6
- 210000001331 nose Anatomy 0.000 description 6
- 238000004422 calculation algorithm Methods 0.000 description 5
- 210000000887 face Anatomy 0.000 description 5
- 238000012545 processing Methods 0.000 description 3
- 230000008859 change Effects 0.000 description 2
- 238000005034 decoration Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 210000004709 eyebrow Anatomy 0.000 description 2
- 230000014509 gene expression Effects 0.000 description 2
- 210000004209 hair Anatomy 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000003796 beauty Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 239000002537 cosmetic Substances 0.000 description 1
- 210000005069 ears Anatomy 0.000 description 1
- 230000008921 facial expression Effects 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000008707 rearrangement Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/168—Feature extraction; Face representation
- G06V40/171—Local features and components; Facial parts ; Occluding parts, e.g. glasses; Geometrical relationships
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
Landscapes
- Engineering & Computer Science (AREA)
- Health & Medical Sciences (AREA)
- Oral & Maxillofacial Surgery (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Theoretical Computer Science (AREA)
- General Health & Medical Sciences (AREA)
- Human Computer Interaction (AREA)
- Image Analysis (AREA)
- Image Processing (AREA)
Abstract
The embodiment of the invention discloses a face contour correction method, a face contour correction device, face contour correction equipment and a storage medium. The method comprises the following steps: acquiring initial key points of a face contour in a face image, carrying out region division on the initial key points to obtain a plurality of initial key point sets, acquiring gradient information of the face image, correcting the initial key points in the current initial key point set by adopting a correction strategy corresponding to the current initial key point set based on the gradient information aiming at each initial key point set to obtain corresponding correction key points, and determining a face contour correction result according to the correction key points, wherein the correction strategy is related to the face contour region characteristics corresponding to the current initial key point set. According to the technical scheme provided by the embodiment of the invention, the calculated amount of gradient calculation is small, the correction efficiency can be effectively improved, the initial key points are divided according to the characteristics of the face contour region, the corresponding correction strategies are adopted for correction respectively, and the accuracy of key point detection can be improved.
Description
Technical Field
The embodiment of the invention relates to the technical field of computer vision, in particular to a face contour correction method, a face contour correction device, face contour correction equipment and a storage medium.
Background
The detection of key points of human faces is used as a fundamental research problem in the field of computer vision, has important application value in the fields of video application, identity verification, monitoring security and the like, and is therefore widely focused in academia and industry. The main task of the face key point detection is to locate the key region positions of the face, including eyebrows, eyes, nose, mouth, face contours and the like, and provide accurate face signals for subsequent other processing.
Compared with key points of other areas, the face outline has richer changes, and is a difficult point in face key point detection. In order to improve the accuracy of the detection of the key points of the human face contour (also called as the prediction of the key points of the human face contour), the primary detection result is generally required to be corrected, and two methods are generally adopted at present. One is to introduce a cascade structure, and correct key points of the face step by step from thick to thin through a multi-stage carefully designed neural network; the other method adopts a multitask learning method, utilizes attribute information (such as gender and the like) of the human face, and simultaneously processes regression tasks of key points and classification tasks of the human face attribute in a joint learning mode. However, both methods are computationally intensive and inefficient. Therefore, the existing face contour correction schemes are still not perfect enough and need improvement.
Disclosure of Invention
The embodiment of the invention provides a face contour correction method, a face contour correction device, face contour correction equipment and a storage medium, which can optimize the existing face contour correction scheme.
In a first aspect, an embodiment of the present invention provides a face contour correction method, including:
acquiring initial key points of a face outline in a face image, and carrying out region division on the initial key points to obtain a plurality of initial key point sets;
acquiring gradient information of the face image;
and correcting the initial key points in the current initial key point set by adopting a correction strategy corresponding to the current initial key point set based on the gradient information aiming at each initial key point set to obtain corresponding correction key points, wherein the correction strategy is related to the characteristics of the face contour region corresponding to the current initial key point set.
In a second aspect, an embodiment of the present invention provides a face contour correction apparatus, including:
the initial key point dividing module is used for acquiring initial key points of the face outline in the face image, and dividing the initial key points into areas to obtain a plurality of initial key point sets;
the gradient information acquisition module is used for acquiring gradient information of the face image;
and the key point correction module is used for correcting the initial key points in the current initial key point set by adopting a correction strategy corresponding to the current initial key point set based on the gradient information aiming at each initial key point set to obtain corresponding correction key points, wherein the correction strategy is related to the face contour region characteristics corresponding to the current initial key point set.
In a third aspect, an embodiment of the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, where the processor implements a face contour correction method as provided in the embodiment of the present invention when executing the computer program.
In a fourth aspect, an embodiment of the present invention provides a computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements a face contour correction method as provided by the embodiment of the present invention.
According to the face contour correction scheme provided by the embodiment of the invention, initial key points of the face contour in the face image are obtained, the initial key points are subjected to region division to obtain a plurality of initial key point sets, gradient information of the face image is obtained, and for each initial key point set, the initial key points in the current initial key point set are corrected by adopting a correction strategy corresponding to the current initial key point set based on the gradient information to obtain corresponding correction key points, wherein the correction strategy is related to the face contour region characteristics corresponding to the current initial key point set. By adopting the technical scheme, the calculation amount introduced by gradient calculation is small, the correction efficiency can be effectively improved, the initial key points are divided according to the characteristics of the face contour region, the corresponding correction strategies are adopted for correction respectively, and the accuracy of key point detection can be improved.
Drawings
Fig. 1 is a flow chart of a face contour correction method according to an embodiment of the present invention;
fig. 2 is a flow chart of another face contour correction method according to an embodiment of the present invention;
fig. 3 is a schematic diagram of a face key point provided in an embodiment of the present invention;
fig. 4 is a schematic diagram of face contour key point correction according to an embodiment of the present invention;
fig. 5 is a schematic diagram of a face contour correction key point detection flow provided in an embodiment of the present invention;
fig. 6 is a schematic diagram before face contour key point correction according to an embodiment of the present invention;
fig. 7 is a schematic diagram of a face contour key point after correction according to an embodiment of the present invention;
fig. 8 is a block diagram of a face contour correction device according to an embodiment of the present invention;
fig. 9 is a block diagram of a computer device according to an embodiment of the present invention.
Detailed Description
The invention is described in further detail below with reference to the drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting thereof. It should be further noted that, for convenience of description, only some, but not all of the structures related to the present invention are shown in the drawings. Furthermore, embodiments of the invention and features of the embodiments may be combined with each other without conflict.
Fig. 1 is a flow chart of a face contour correction method according to an embodiment of the present invention, which may be performed by a face contour correction device, where the device may be implemented by software and/or hardware, and may be generally integrated in a computer device. As shown in fig. 1, the method includes:
The computer device may include mobile terminal devices such as mobile phones, tablet computers, notebook computers, and personal digital assistants, and may also include other devices such as desktop computers. The embodiment of the invention can effectively improve the detection calculation efficiency while ensuring the detection accuracy of the key points of the face contour, and effectively control the calculation complexity, thus being widely applicable to mobile calculation platforms and other platforms with limited calculation resources.
The scheme provided by the embodiment of the invention can be applied to various application scenes needing face contour correction, and can be seamlessly accessed into the existing face key point frame. Optionally, the method can be applied to the functions of special cosmetic effects, beauty, stickers, virtual puppets, 3D expressions and the like in video live broadcast application or short video application and the like.
The face image may be understood as an image containing a face, and may be specifically acquired by an acquisition device such as a camera in a computer device, or may be transmitted to the computer device through a network, or may be an image locally stored in the computer device, or the like, where a specific source is not limited. In addition, the number of faces included in the face image is not limited, and may be one or more.
By way of example, face frames and key points of all faces in a face image can be obtained through face detection and a key point positioning network, and position information (such as coordinates) of the key points can be obtained, key points corresponding to face contours are extracted from the key points, and initial key points of the face contours are obtained. The process can be understood as rough detection of key points of the face contour, and the rough detection result is corrected in the subsequent step to obtain an accurate face contour detection result. The specific type and network structure of the face detection and key point positioning network are not limited. The number of key points of the face image can be set according to actual requirements, and the number of initial key points of the face contour can also be set according to actual requirements, and the method is not limited in detail.
In the embodiment of the invention, the initial key points are divided into areas, and the specific dividing mode can be determined according to actual requirements, such as application scenes, detection accuracy, facial expressions, facial movement conditions and the like. Illustratively, the area division is performed on the initial key points to obtain a plurality of initial key point sets, which may include: and carrying out region division on the initial key points according to the left end point, the left cheek, the chin, the right cheek and the right end point to obtain at least five initial key point sets. For example, it may be divided into five initial sets of key points corresponding to the left end point, the left cheek region, the chin region, the right cheek region, and the right end point, respectively. Considering that the motion degree of a specific part is different, the chin can be further subdivided, and when the motion degree of the chin is larger, the chin can be further divided into a left chin area, a right chin area and a right chin area.
Optionally, before the area division of the initial keypoint, the method may further include: and recognizing the face state in the face image, and carrying out region division on the initial key points according to the recognition result. The face states may include expression types or facial motion conditions, and different dividing modes may be configured for different face states in advance.
For example, gradient information of the face image may be calculated using a preset gradient operator. The preset gradient operator can be a Sobel operator, and gradient information of the face image can be calculated rapidly and accurately. In addition, other gradient operators are possible, such as the Roberts (Roberts) operator, the Probiot (Prewitt) operator, and the Laplace (Laplace) operator. The gradient information may in particular be a gradient value.
In some embodiments, acquiring the facial key points in the facial image may also be included. The step can comprise the steps of calculating initial gradient information of the face image by using a preset gradient operator, and setting the gradient of the image area corresponding to the five sense organs key point to zero on the basis of the initial gradient information to obtain the gradient information of the face image. The advantage of this arrangement is that by erasing the gradients in these regions, the main gradients of the face are distributed near the facial contour region, which can eliminate the possible interference caused by the gradients in the facial region, especially when the face in the facial image is a side face, and can avoid searching the facial contour according to the maximum gradient principle, so that the search result is easy to sink into the region with obvious contour, such as the mouth, nose, eyes and other facial parts, and thus the critical point positioning is seriously deviated. The five sense organs in the five sense organ key points can be all parts (eyebrows, eyes, ears, nose and mouth) of the five sense organs, or part of the five sense organs, and can be selected according to actual conditions, for example, left eye, right eye, nose and mouth. Optionally, after the facial feature key points in the facial image are acquired, the image areas corresponding to the facial feature key points are removed from the facial image, and then gradient information of the facial image after removal processing is calculated by using a preset gradient operator. The specific acquisition mode of the gradient information can be set according to the preset gradient operators and other related requirements.
The inventor notes that the gradient value of the face contour area is usually larger, so that sufficient bottom features such as edge textures can be provided for contour key point positioning, meanwhile, the calculation amount introduced by a gradient operator is smaller, the algorithm efficiency can be effectively controlled, and the deployment of a mobile terminal is facilitated, so that the inventor decides to correct the key points based on gradient information. In addition, the inventor also considers that the movement conditions of different areas of the facial contour may be different under different conditions, for example, when the face is in a smile state, the change in the horizontal direction of the key points of the cheek area is more obvious than the key points of the chin area, and thus, the different areas need to be processed separately.
For example, corresponding correction strategies may be preset according to the characteristics of each face contour region, and different correction strategies may be different correction algorithms, or may also adopt the same correction algorithm and different correction parameter values (that is, the formulas corresponding to the correction algorithms are the same, but the values of one or more parameters in the formulas are different).
For example, the region division modes corresponding to different face states may be different, and the correction strategies corresponding to the multiple initial key point sets divided by the different division modes may be different. For example, the first division method divides the area a, the area B, the area C, the area D, and the area E into corresponding correction policies a, B, C, D, E; the second division method divides the area F, the area G, the area H, the area I and the area J into corresponding correction policies F, G, H, I and J respectively. Optionally, the step may further include: and determining correction strategies corresponding to the multiple initial key point sets respectively according to the region division mode. The method has the advantage that the appropriate correction strategy corresponding to each region can be selected according to the current face state more flexibly.
By way of example, after the initial key points in all the initial key point sets are corrected and the corresponding corrected key points are obtained, all the corrected key points are summarized, and the corrected result of the face contour, namely the final face contour detection result, can be obtained.
According to the face contour correction method provided by the embodiment of the invention, initial key points of the face contour in the face image are obtained, the initial key points are subjected to region division to obtain a plurality of initial key point sets, gradient information of the face image is obtained, and for each initial key point set, the initial key points in the current initial key point set are corrected by adopting a correction strategy corresponding to the current initial key point set based on the gradient information to obtain corresponding correction key points, wherein the correction strategy is related to the face contour region characteristics corresponding to the current initial key point set. By adopting the technical scheme, the calculation amount introduced by gradient calculation is small, the correction efficiency can be effectively improved, the initial key points are divided according to the characteristics of the face contour region, the corresponding correction strategies are adopted for correction respectively, and the accuracy of key point detection can be improved.
In some embodiments, the correcting the initial key points in the current initial key point set by using a correction policy corresponding to the current initial key point set based on the gradient information includes: and for each initial key point in the current initial key point set, correcting the current initial key point by adopting a correction strategy corresponding to the current initial key point set based on the gradient information and two initial key points adjacent to the current initial key point in the left-right direction. The advantage of this arrangement is that it is possible to target eachThe initial key points to be corrected are corrected in a small range by referring to adjacent initial key points, so that excessive correction and deviation from a real contour are prevented. It should be noted that, for the first initial keypoint and the last initial keypoint in the current initial keypoint set, the initial keypoints adjacent to the left or right may be the initial keypoints in the other initial keypoint sets, or the initial keypoints may not exist. For the case that the initial key point does not exist, a preset algorithm can be adopted to calculate the missing initial key point. For example, for the left endpoint initial keypoint p i The left endpoint initial key point p can be utilized i And right adjacent initial key point p i+1 To calculate the coordinates of the left adjacent initial key point p i-1 ＝2*p i -p i+1 . As another example, for right endpoint initial keypoint p i The right endpoint initial key point p can be utilized i And left adjacent initial key point p i-1 To calculate the coordinates of the left adjacent initial key point p i+1 ＝2*p i -p i-1 。
In some embodiments, the correcting the current initial key point by using a correction policy corresponding to the current initial key point set based on the gradient information and two initial key points adjacent to the current initial key point in the left-right direction includes: and under the condition that the current initial key point and two initial key points adjacent to the current initial key point in the left-right direction are both visible, correcting the current initial key point by adopting a correction strategy corresponding to the current initial key point set based on the gradient information and the two initial key points adjacent to the current initial key point in the left-right direction. The advantage of this arrangement is that the inventor considers that the visibility of key points of the facial contour may change, for example, the key points may be blocked by accessory decorations (such as collars or scarves) near the face, and when the key points are blocked, if the key points are still corrected by using gradient information, the reliability of prediction is significantly reduced, and the correction effect is affected. Optionally, the face frames and key points of all faces in the face image are obtained through the face detection and key point positioning network, and meanwhile, the visibility of the key points can be obtained. Optionally, under the condition that it is determined that the current initial key point and any one of two initial key points adjacent to the current initial key point in the left-right direction are not visible, the current initial key point is directly incorporated into the face contour correction result, that is, the current initial key point is not corrected.
In some embodiments, the correcting the current initial key point by using a correction policy corresponding to the current initial key point set based on the gradient information and two initial key points adjacent to the current initial key point in the left-right direction includes: determining the direction of the contour line according to two initial key points adjacent to the current initial key point in the left-right direction; and searching a target point in the vertical direction of the contour line direction based on the gradient information by adopting a correction strategy corresponding to the current initial key point set, and taking the target point as a correction key point corresponding to the current initial key point, wherein the correction strategy corresponding to the current initial key point set comprises a search range. The perpendicular direction to the contour line direction is understood to be the direction perpendicular to the contour line direction. The setting has the advantages that key points which are more attached to the outline of the real face can be found in the direction perpendicular to the outline, and correction efficiency is improved under the condition that correction accuracy is guaranteed. Wherein the contour direction may be represented by a contour direction vector, which may be represented as v, for example ic ＝p i+1 -p i-1 Wherein p is i+1 For right adjacent initial key point, p i-1 Is the left adjacent initial keypoint.
In some embodiments, the searching the target point in the vertical direction of the contour line direction based on the gradient information by adopting the correction strategy corresponding to the current initial key point set includes: determining an initial search point in a search range, and determining other search points according to a search step length on the basis of the initial search point; and determining the score of each search point by adopting a preset score determining mode based on the gradient information, and determining a target point according to the score. The setting has the advantages that the searching points are determined by setting reasonable searching step length, and each searching point is scored according to gradient information in a certain scoring mode, so that the target point is found rapidly and accurately.
Wherein the starting search point may be determined based on the boundaries of the search range. Illustratively, the initial search point is determined according to the current initial key point, the search range in the contour line direction, the search range in the vertical direction, a first ratio of a first distance of the initial search point and the current initial key point in the contour line direction to the search range in the contour line direction, and a second ratio of a second distance of the initial search point and the current initial key point in the vertical direction to the search range in the vertical direction. By setting the first proportion and the second proportion, the relative position relation between the search area and the current initial key point can be reasonably determined, and different first proportions and/or second proportions can be configured for the corresponding correction strategies according to the characteristics of the face contour area, so that the correction strategies are more suitable for the characteristics of the corresponding face contour area, and more targeted correction results are obtained. For example, different first proportions and/or second proportions may be set according to the difference in the movement condition of the corresponding portion of the face contour region, and taking the movement condition at the lower jaw as an example, a condition such as "double chin" may be generated, and thus, for this condition, the value of the different second proportions may be specifically selected.
Optionally, the second ratio is set so that the center of the search range in the vertical direction is biased toward the interior of the face outline, and in the vertical direction, since gradient interference of the collar, the hair, and the like on the exterior of the face is often greater than that on the interior of the face, the search range on the exterior of the face in the vertical direction is smaller than that on the interior of the face, rather than searching in the same range on the interior and the exterior of the face.
In some embodiments, the determining the score of each search point by using a preset score determining manner based on the gradient information includes: and for each search point, determining the score of the current search point according to the projection size of the current search point in the depth direction and the distance between the current search point and the current initial key point, wherein the projection size of the current search point in the depth direction is related to gradient information corresponding to the current search point. The method has the advantages that the target point is found more quickly and accurately by utilizing gradient information and a reasonable grading mode, the distance between the current searching point and the current initial key point is considered when the grading is determined, the searching point can be limited not to be far away from the current initial key point, and excessive correction is avoided.
In some embodiments, the determining the target point from the score comprises: calculating total scores corresponding to different search steps in the vertical direction according to the scores; determining the corresponding target searching step number by taking the highest total score as a target; and determining a target point according to the target searching step number and the initial searching point. The advantage of this arrangement is that the scores of the search points can be summarized, the comparison is performed by using the total scores of the different search steps in the vertical direction, the comparison result is obtained quickly, and the target point in the vertical direction is obtained according to the comparison result.
In some embodiments, the starting search point within the search range is determined using the following formula:
the score for the current search point is determined using the following formula:
determining a target search step number using the following formula:
wherein b i Representing a starting search point;
representing a search range in the contour line direction; />
Representing a search range in the vertical direction; m represents a first ratio; n represents a second ratio; v ic Representing a contour line direction vector calculated from two initial keypoints left and right adjacent to the current initial keypoint; v ip Representation according to v ic A calculated vertical direction vector; />
Representing a current search point, wherein i corresponds to a current initial key point, j represents the number of search steps in the contour line direction, and k represents the number of search steps in the vertical direction; />
Representing a search step in the contour direction; />
Representing a search step in the vertical direction; p is p i Representing a current initial key point; />
Representing the gradient corresponding to the current search point; />
Representation->
At v ip The numerical product of the projections in the direction, 255 being a normalization factor; alpha represents a balance factor; k (k) * Indicating the number of target search steps.
Fig. 2 is a flow chart of another face contour correction method according to an embodiment of the present invention, where the method is optimized based on the above-mentioned alternative embodiments, as shown in fig. 2, and may include:
Among them, the five sense organ key points may include key points of left eye, right eye, nose and mouth.
And 203, setting the gradient of the image area corresponding to the five sense organ key points to zero on the basis of the initial gradient information to obtain the gradient information of the face image.
And 204, carrying out region division on the initial key points to obtain a plurality of initial key point sets.
The method comprises the steps of dividing a left endpoint, a right endpoint, a left cheek key point, a right cheek key point and a chin key point to obtain five initial key point sets. Fig. 3 is a schematic diagram of a face key point provided by the embodiment of the present invention, and as shown in the drawing, 106 face key points are obtained, wherein the total number of face contour points is 33, the left end point of the face contour is the key point with the number of 0, the right end point is the key point with the number of 32, the left cheek key point is the key point with the number of 1-7, the right cheek key point is the key point with the number of 25-31, and the chin key point is the key point with the number of 8-24. Fig. 3 is only a schematic illustration, and the specific number and division may be set according to practical situations.
For each initial key point of the left cheek, if the initial key point and two adjacent initial key points on the left and right are visible, the initial key point and the two adjacent initial key points are utilized to correct the initial key points by using a gradient correction strategy corresponding to the left cheek, otherwise, the original coordinates are maintained.
For each initial key point of the right cheek, if the initial key point and two adjacent initial key points on the left and right are visible, correcting by using the gradient correction strategy corresponding to the left cheek by utilizing the initial key point and the two adjacent initial key points, otherwise, keeping the original coordinates.
For each initial key point of the chin, if the initial key point and two adjacent initial key points on the left and right are visible, correcting by using the gradient correction strategy corresponding to the chin by utilizing the initial key point and the two adjacent initial key points, otherwise, keeping the original coordinates.
For the initial key point of the left endpoint, if the initial key point p i And its right adjacent initial key point p i+1 All can be seen, the coordinates of the initial key point and the right adjacent initial key point are utilized to calculate and obtain a left adjacent initial key point p i-1 ＝2*p i -p i+1 And correcting by using the initial key point and two adjacent initial key points and using a gradient correction strategy corresponding to the left end point, otherwise, maintaining the original coordinates.
For the initial key point of the right endpoint, if the initial key point p i And its left adjacent initial key point p i-1 All can see that the coordinates of the initial key point and the left adjacent initial key point are utilized to calculate and obtain the right adjacent initial key point p i+1 ＝2*p i -p i-1 Using the initial key point and two adjacent initial key points to correct by using a gradient correction strategy corresponding to the right endpoint, otherwise, keepingOriginal coordinates.
The following describes an example of a correction procedure for an initial key point of the chin area. Fig. 4 is a schematic diagram of face contour key point correction according to an embodiment of the present invention.
As shown in fig. 4, for the initial key point p i Calculating the left adjacent initial key point p i-1 To the right adjacent initial key point p i+1 Is the contour line direction vector v ic ＝p i+1 -p i-1 Its vector sitting is marked as
The vertical vector of the contour line direction is +.>
Our goal is to search the optimal key point p of the most fitting face contour line in the vertical vector direction i * I.e. the target point.
Setting a starting search point
Wherein->
Is the contour line direction vector +>
Search scope of->
Is a vertical vector +.>
Search range f out Control b i And p is as follows i Distance in the vertical vector direction. Can be provided with b i And p is as follows i The distance of the vector in the direction of the contour line is +.>
I.e. theThe first ratio was 0.5. And for a second ratio f out If set f out =0.5, then b i And p is as follows i The distance in the vertical vector direction is +.>
But in the vertical vector direction, since the gradient disturbance outside the hair face tends to be greater than the gradient disturbance inside the face, f can be set out < 0.5 such that the search range outside the face in the vertical vector direction is smaller than that inside the face, instead of searching in the same range inside and outside the face.
Is arranged in a unit vertical vector
The search step in direction is +.>
For searching the total number of steps, the vector +.>
The search step in direction is +.>
To search for the total number of steps. When searching the kth step in the vertical vector direction and searching the jth step in the contour line direction vector direction, the searched points are as follows:
point(s)
Score S of (2) ijk The calculation formula is as follows:
wherein alpha is a balance factor,
for->
Gradient calculated by Sobel operator, +.>
Is->
At v ip The numerical product of the projections in the direction, 255, is a normalization factor. Wherein the first term of the first formula +.>
Representing the projection size in the depth direction; the second term is mainly used to limit (punish) the point of the search from p i Too far, ||v ic The I is used for normalization.
Score S when searching for step number kth in vertical vector direction ik ＝∑ j S ijk Thus for the initial key point p i Optimum number of search steps
According to k * Obtaining a corrected key point (namely a target point):
the method for correcting the key points of the human face based on the gradient information aims at the contour line direction vector
Vertical vector
Different search ranges can be set, and different application scenes can be adapted more flexibly. Because the movement conditions of different parts of the outline key points are different, different f can be used out To correct for different locations. For example, in practical situations, the chin key point of the face is more easily disturbed by gradients of decorations such as collars and scarves, so in the correction of the chin key point, f out May be set smaller than the left and right cheek keypoints and the left and right end points.
And 206, determining a face contour detection result according to the corrected key points.
According to the face contour correction method provided by the embodiment of the invention, the initial key points are partitioned, the initial key points are corrected by combining the visibility and the gradient information, gradient noise interference, the visibility of the key points and priori knowledge of the face contour are fully considered, compared with the scheme of optimizing the key points by utilizing a plurality of cascade networks and the like, the face contour correction method has the advantages of greatly improving the accuracy of face contour key point detection in terms of time consumption, enabling the predicted result of the contour key points to be attached to the real face contour to be higher, and still accurately positioning the positions of the contour key points under the condition that the face posture and the face orientation are changed. Meanwhile, the method is less in time consumption, and the key point detection operation requirements of live broadcasting and other real-time scenes are met.
Fig. 5 is a schematic diagram of a face contour correction key point detection flow provided in an embodiment of the present invention, which can be further understood by combining with fig. 5. And inputting the original face image into a face detection and key point positioning network to obtain the face frames, key point coordinates and the visibility of key points of all faces in the picture. And then calculating the gradient of the face image by using a Sobel gradient calculation module, removing the image gradients of eyes, noses and mouths, and correcting the contour key points by using the key point coordinates, the visibility of the key points and the gradient information after gradient removal processing to obtain a final detection result.
Fig. 6 is a schematic diagram before face contour key point correction provided by an embodiment of the present invention, fig. 7 is a schematic diagram after face contour key point correction provided by an embodiment of the present invention, and fig. 7 shows a final detection result obtained by using a face contour correction method of an embodiment of the present invention. As can be seen from comparing fig. 6 and fig. 7, the detection result of the contour key points is more fit with the real face contour.
Fig. 8 is a block diagram of a face contour correction device according to an embodiment of the present invention, where the device may be implemented by software and/or hardware, and may be generally integrated in a computer device, and may perform face contour correction by performing a face contour correction method. As shown in fig. 8, the apparatus includes:
the initial key point dividing module 801 is configured to obtain initial key points of a face contour in a face image, and perform region division on the initial key points to obtain a plurality of initial key point sets;
a gradient information obtaining module 802, configured to obtain gradient information of the face image;
and a key point correction module 803, configured to correct, for each initial key point set, an initial key point in the current initial key point set by using a correction policy corresponding to the current initial key point set based on the gradient information, so as to obtain a corresponding correction key point, where the correction policy is related to a face contour region feature corresponding to the current initial key point set.
According to the face contour correction device provided by the embodiment of the invention, the initial key points of the face contour in the face image are obtained, the initial key points are subjected to region division to obtain a plurality of initial key point sets, gradient information of the face image is obtained, the initial key points in the current initial key point set are corrected by adopting a correction strategy corresponding to the current initial key point set based on the gradient information aiming at each initial key point set, and the corresponding correction key points are obtained, wherein the correction strategy is related to the face contour region characteristics corresponding to the current initial key point set. By adopting the technical scheme, the calculation amount introduced by gradient calculation is small, the correction efficiency can be effectively improved, the initial key points are divided according to the characteristics of the face contour region, the corresponding correction strategies are adopted for correction respectively, and the accuracy of key point detection can be improved.
The embodiment of the invention provides a computer device, and the face contour correction device provided by the embodiment of the invention can be integrated in the computer device. Fig. 9 is a block diagram of a computer device according to an embodiment of the present invention. The computer device 900 comprises a memory 901, a processor 902 and a computer program stored in the memory 901 and capable of running on the processor 902, wherein the processor 902 implements the face contour correction method provided by the embodiment of the invention when executing the computer program.
The embodiment of the invention also provides a storage medium containing computer executable instructions, which when executed by a computer processor, are used for executing the face contour correction method provided by the embodiment of the invention.
The face contour correction device, the face contour correction equipment and the face contour correction storage medium provided by the embodiment can execute the face contour correction method provided by any embodiment of the invention, and have the corresponding functional modules and beneficial effects of executing the method. Technical details not described in detail in the above embodiments may be referred to the face contour correction method provided in any embodiment of the present invention.
Note that the above is only a preferred embodiment of the present invention. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, while the invention has been described in connection with the above embodiments, the invention is not limited to the embodiments, but may be embodied in many other equivalent forms without departing from the spirit or scope of the invention, which is set forth in the following claims.
Claims (14)
1. The face contour correction method is characterized by comprising the following steps of:
acquiring initial key points of a face outline in a face image, and carrying out region division on the initial key points to obtain a plurality of initial key point sets;
acquiring gradient information of the face image;
for each initial key point set, correcting initial key points in the current initial key point set by adopting a correction strategy corresponding to the current initial key point set based on the gradient information to obtain corresponding correction key points, wherein the correction strategy is related to the face contour region characteristics corresponding to the current initial key point set;
the correcting the initial key points in the current initial key point set by adopting a correction strategy corresponding to the current initial key point set based on the gradient information comprises the following steps:
and for each initial key point in the current initial key point set, correcting the current initial key point by adopting a correction strategy corresponding to the current initial key point set based on the gradient information and two initial key points adjacent to the current initial key point in the left-right direction.
2. The method according to claim 1, wherein the correcting the current initial key point with the correction policy corresponding to the current initial key point set based on the gradient information and two initial key points adjacent to the current initial key point in the left-right direction includes:
and under the condition that the current initial key point and two initial key points adjacent to the current initial key point in the left-right direction are both visible, correcting the current initial key point by adopting a correction strategy corresponding to the current initial key point set based on the gradient information and the two initial key points adjacent to the current initial key point in the left-right direction.
3. The method according to claim 1 or 2, wherein the correcting the current initial key point by using a correction policy corresponding to the current initial key point set based on the gradient information and two initial key points adjacent to the current initial key point in the left-right direction includes:
determining the direction of the contour line according to two initial key points adjacent to the current initial key point in the left-right direction;
and searching a target point in the vertical direction of the contour line direction based on the gradient information by adopting a correction strategy corresponding to the current initial key point set, and taking the target point as a correction key point corresponding to the current initial key point, wherein the correction strategy corresponding to the current initial key point set comprises a search range.
4. The method of claim 3, wherein the searching for the target point in the direction perpendicular to the contour line direction based on the gradient information using the correction strategy corresponding to the current initial set of key points comprises:
determining an initial search point in a search range, and determining other search points according to a search step length on the basis of the initial search point;
and determining the score of each search point by adopting a preset score determining mode based on the gradient information, and determining a target point according to the score.
5. The method of claim 4, wherein determining the score of each search point based on the gradient information in a preset score determination manner comprises:
and for each search point, determining the score of the current search point according to the projection size of the current search point in the depth direction and the distance between the current search point and the current initial key point, wherein the projection size of the current search point in the depth direction is related to gradient information corresponding to the current search point.
6. The method of claim 5, wherein the determining a target point from the score comprises:
calculating total scores corresponding to different search steps in the vertical direction according to the scores;
determining the corresponding target searching step number by taking the highest total score as a target;
and determining a target point according to the target searching step number and the initial searching point.
7. The method of claim 4, wherein determining the starting search point within the search range comprises:
and determining an initial search point according to the current initial key point, the search range in the contour line direction, the search range in the vertical direction, the first proportion of the first distance between the initial search point and the current initial key point in the contour line direction to the search range in the contour line direction, and the second proportion of the second distance between the initial search point and the current initial key point in the vertical direction to the search range in the vertical direction.
8. The method of claim 7, wherein the second ratio is valued such that a center of the search range in the vertical direction is biased toward the interior of the face contour.
9. The method of claim 7, wherein the step of determining the position of the probe is performed,
the starting search point within the search range is determined using the following formula:
the score for the current search point is determined using the following formula:
determining a target search step number using the following formula:
wherein b i Representing a starting search point;
representing a search range in the contour line direction; />
Representing a search range in the vertical direction; m represents a first ratio; n represents a second ratio; x is x ic Representing a contour line direction vector calculated from two initial keypoints left and right adjacent to the current initial keypoint; v ip Representation according to v ic A calculated vertical direction vector; />
Representing the current search point, i corresponds to the current initial key point, j represents theThe number of search steps in the contour line direction, k representing the number of search steps in the vertical direction; />
Representing a search step in the contour direction; />
Representing a search step in the vertical direction; p is p i Representing a current initial key point; />
Representing the gradient corresponding to the current search point; />
Representation->
At v ip The numerical product of the projections in the direction, 255 being a normalization factor; alpha represents a balance factor; k (k) * Indicating the number of target search steps.
10. The method of claim 1, wherein the performing region division on the initial keypoints to obtain a plurality of initial keypoint sets includes:
and carrying out region division on the initial key points according to the left end point, the left cheek, the chin, the right cheek and the right end point to obtain at least five initial key point sets.
11. The method as recited in claim 1, further comprising: acquiring five sense organ key points in the face image:
the acquiring gradient information of the face image comprises the following steps:
calculating initial gradient information of the face image by using a preset gradient operator;
and setting the gradient of the image area corresponding to the five-sense organ key points to zero on the basis of the initial gradient information to obtain the gradient information of the face image.
12. A face contour correction apparatus, comprising:
the initial key point dividing module is used for acquiring initial key points of the face outline in the face image, and dividing the initial key points into areas to obtain a plurality of initial key point sets;
the gradient information acquisition module is used for acquiring gradient information of the face image;
the key point correction module is used for correcting the initial key points in the current initial key point set by adopting a correction strategy corresponding to the current initial key point set based on the gradient information to obtain corresponding correction key points, wherein the correction strategy is related to the face contour region characteristics corresponding to the current initial key point set;
the key point correction module is specifically configured to:
and for each initial key point in the current initial key point set, correcting the current initial key point by adopting a correction strategy corresponding to the current initial key point set based on the gradient information and two initial key points adjacent to the current initial key point in the left-right direction.
13. A facial contour correction apparatus comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the method of any of claims 1-11 when executing the computer program.
14. A computer readable storage medium, on which a computer program is stored, characterized in that the program, when being executed by a processor, implements the method according to any one of claims 1-11.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010285509.0A CN111488836B (en) | 2020-04-13 | 2020-04-13 | Face contour correction method, device, equipment and storage medium |
| PCT/CN2021/085385 WO2021208767A1 (en) | 2020-04-13 | 2021-04-02 | Facial contour correction method and apparatus, and device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010285509.0A CN111488836B (en) | 2020-04-13 | 2020-04-13 | Face contour correction method, device, equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111488836A CN111488836A (en) | 2020-08-04 |
| CN111488836B true CN111488836B (en) | 2023-06-02 |
Family
ID=71812733
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010285509.0A Active CN111488836B (en) | 2020-04-13 | 2020-04-13 | Face contour correction method, device, equipment and storage medium |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN111488836B (en) |
| WO (1) | WO2021208767A1 (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111488836B (en) * | 2020-04-13 | 2023-06-02 | 广州市百果园信息技术有限公司 | Face contour correction method, device, equipment and storage medium |
| CN112837336B (en) * | 2021-02-23 | 2022-02-22 | 浙大宁波理工学院 | Method and system for estimating and acquiring room layout based on heat map correction of key points |
Family Cites Families (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP4414401B2 (en) * | 2006-02-10 | 2010-02-10 | 富士フイルム株式会社 | Facial feature point detection method, apparatus, and program |
| CN103824087A (en) * | 2012-11-16 | 2014-05-28 | 广州三星通信技术研究有限公司 | Detection positioning method and system of face characteristic points |
| CN106156692B (en) * | 2015-03-25 | 2019-12-13 | 阿里巴巴集团控股有限公司 | method and device for positioning human face edge feature points |
| CN105139007B (en) * | 2015-09-30 | 2019-04-16 | 小米科技有限责任公司 | Man face characteristic point positioning method and device |
| CN107358207A (en) * | 2017-07-14 | 2017-11-17 | 重庆大学 | A kind of method for correcting facial image |
| CN107958235B (en) * | 2017-12-28 | 2021-10-26 | 泰康保险集团股份有限公司 | Face image detection method, device, medium and electronic equipment |
| CN109409262A (en) * | 2018-10-11 | 2019-03-01 | 北京迈格威科技有限公司 | Image processing method, image processing apparatus, computer readable storage medium |
| CN109376684B (en) * | 2018-11-13 | 2021-04-06 | 广州市百果园信息技术有限公司 | Face key point detection method and device, computer equipment and storage medium |
| CN110175558B (en) * | 2019-05-24 | 2021-02-05 | 北京达佳互联信息技术有限公司 | Face key point detection method and device, computing equipment and storage medium |
| CN110348331B (en) * | 2019-06-24 | 2022-01-14 | 深圳数联天下智能科技有限公司 | Face recognition method and electronic equipment |
| CN111488836B (en) * | 2020-04-13 | 2023-06-02 | 广州市百果园信息技术有限公司 | Face contour correction method, device, equipment and storage medium |
-
2020
- 2020-04-13 CN CN202010285509.0A patent/CN111488836B/en active Active
-
2021
- 2021-04-02 WO PCT/CN2021/085385 patent/WO2021208767A1/en active Application Filing
Also Published As
| Publication number | Publication date |
|---|---|
| WO2021208767A1 (en) | 2021-10-21 |
| CN111488836A (en) | 2020-08-04 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2021139484A1 (en) | Target tracking method and apparatus, electronic device, and storage medium | |
| US11403874B2 (en) | Virtual avatar generation method and apparatus for generating virtual avatar including user selected face property, and storage medium | |
| JP5755012B2 (en) | Information processing apparatus, processing method thereof, program, and imaging apparatus | |
| US20220400246A1 (en) | Gaze correction of multi-view images | |
| EP2863335A1 (en) | Method, device and storage medium for locating feature points on human face | |
| JP7230939B2 (en) | Information processing device, information processing method and information processing program | |
| JP2006260397A (en) | Eye opening degree estimating device | |
| US9213897B2 (en) | Image processing device and method | |
| CN111488836B (en) | Face contour correction method, device, equipment and storage medium | |
| JP2007164720A (en) | Head detecting device, head detecting method, and head detecting program | |
| WO2020238374A1 (en) | Method, apparatus, and device for facial key point detection, and storage medium | |
| JP2008250999A (en) | Object tracing method, object tracing device and object tracing program | |
| CN110531853B (en) | Electronic book reader control method and system based on human eye fixation point detection | |
| EP3506149A1 (en) | Method, system and computer program product for eye gaze direction estimation | |
| CN110598638A (en) | Model training method, face gender prediction method, device and storage medium | |
| JP2000357221A (en) | Method and device for image processing and recording medium with image processing program recorded | |
| CN112597872A (en) | Gaze angle estimation method and device, storage medium, and electronic device | |
| JPWO2018078857A1 (en) | Gaze estimation apparatus, gaze estimation method, and program recording medium | |
| CN111898571A (en) | Action recognition system and method | |
| JP2018081402A (en) | Image processing system, image processing method, and program | |
| JPWO2017061106A1 (en) | Information processing apparatus, image processing system, image processing method, and program | |
| CN107341460A (en) | Face tracking method and device | |
| US20240104769A1 (en) | Information processing apparatus, control method, and non-transitory storage medium | |
| EP2998928B1 (en) | Apparatus and method for extracting high watermark image from continuously photographed images | |
| WO2020232697A1 (en) | Online face clustering method and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20231012 Address after: 31a, 15 / F, building 30, maple mall, bangrang Road, Brazil, Singapore Patentee after: Baiguoyuan Technology (Singapore) Co.,Ltd. Address before: 5-13 / F, West Tower, building C, 274 Xingtai Road, Shiqiao street, Panyu District, Guangzhou, Guangdong 510000 Patentee before: GUANGZHOU BAIGUOYUAN INFORMATION TECHNOLOGY Co.,Ltd. |
|
| TR01 | Transfer of patent right |