CN109635659B - Face key point positioning method and device, storage medium and electronic equipment - Google Patents
Face key point positioning method and device, storage medium and electronic equipment Download PDFInfo
- Publication number
- CN109635659B CN109635659B CN201811341716.2A CN201811341716A CN109635659B CN 109635659 B CN109635659 B CN 109635659B CN 201811341716 A CN201811341716 A CN 201811341716A CN 109635659 B CN109635659 B CN 109635659B
- Authority
- CN
- China
- Prior art keywords
- key point
- coordinates
- key points
- target
- face
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 53
- 238000005457 optimization Methods 0.000 claims abstract description 137
- 239000013598 vector Substances 0.000 claims description 54
- 230000009471 action Effects 0.000 claims description 24
- 238000004590 computer program Methods 0.000 claims description 8
- 230000006870 function Effects 0.000 claims description 3
- 238000004364 calculation method Methods 0.000 abstract description 13
- 238000005516 engineering process Methods 0.000 abstract description 3
- 230000008569 process Effects 0.000 description 7
- 238000004891 communication Methods 0.000 description 6
- 238000010586 diagram Methods 0.000 description 6
- 238000012549 training Methods 0.000 description 5
- 238000004422 calculation algorithm Methods 0.000 description 3
- 238000013459 approach Methods 0.000 description 2
- 230000001419 dependent effect Effects 0.000 description 2
- 230000004807 localization Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000013136 deep learning model Methods 0.000 description 1
- 238000006073 displacement reaction Methods 0.000 description 1
- 210000004709 eyebrow Anatomy 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/161—Detection; Localisation; Normalisation
- G06V40/165—Detection; Localisation; Normalisation using facial parts and geometric relationships
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/168—Feature extraction; Face representation
- G06V40/171—Local features and components; Facial parts ; Occluding parts, e.g. glasses; Geometrical relationships
Landscapes
- Engineering & Computer Science (AREA)
- Health & Medical Sciences (AREA)
- Oral & Maxillofacial Surgery (AREA)
- Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- Human Computer Interaction (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Theoretical Computer Science (AREA)
- Geometry (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Image Analysis (AREA)
Abstract
The disclosure relates to a face key point positioning method, a face key point positioning device, a storage medium and electronic equipment, and aims to solve the problem that the existing face key point positioning technology is too high in calculation cost. The method comprises the following steps: according to a preset face template, pre-generating initial key points on a face image; performing the following updating operation aiming at the target key points on the face image: determining a key point set comprising the target key points, and carrying out local optimization on coordinates of all key points in the key point set; determining a functional relation for carrying out global optimization on the target key points according to coordinates before and after local optimization of all key points in the key point set; and substituting the coordinates before the local optimization updating of the target key point into the functional relation to obtain the coordinates after the global optimization of the target key point, and taking the coordinates after the global optimization of the target key point as the result of the current coordinate updating of the target key point.
Description
Technical Field
The present disclosure relates to the field of image processing technologies, and in particular, to a method and an apparatus for locating a face key point, a storage medium, and an electronic device.
Background
In the field of computer vision, the positioning of key points of a face image has very important significance, and the core of the positioning is to quickly and accurately position key points with strong semantic meaning in the face image, such as canthus, nose tip, mouth corner and the like in the face image.
In the related art, the key points of the face are mainly located by the following two ways:
the first method is a method for determining key points by using the combination of basic features of an image, namely, texture features of a human face and position constraints among feature points, and the classical algorithms include an ASM (Active shape model) algorithm and an AAM (Active application Models) algorithm.
The second way is to autonomously learn key points in the face image by using a deep learning model.
Although the two methods can determine the positions of the face key points, the face key points need to be determined by learning a large number of face texture features and shape features, and the early-stage calculation cost is high.
Disclosure of Invention
The present disclosure aims to provide a method, an apparatus, a storage medium and an electronic device for locating a face key point, so as to solve the problem of too high computation cost of the existing face key point locating technology.
In order to achieve the above object, in a first aspect, the present disclosure provides a method for locating a face keypoint, the method including:
according to a preset face template, pre-generating initial key points on a face image;
performing the following updating operation aiming at the target key points on the face image:
determining a key point set comprising the target key points, and carrying out local optimization on coordinates of all key points in the key point set;
determining a functional relation for carrying out global optimization on the target key points according to coordinates before and after local optimization of all key points in the key point set;
substituting the coordinates before the local optimization updating of the target key points into the function relation to obtain the coordinates after the global optimization of the target key points, and taking the coordinates after the global optimization of the target key points as the result of the current coordinate updating of the target key points;
wherein the target key point is any key point on the face image.
Optionally, the locally optimizing the coordinates of all the keypoints in the set of keypoints includes:
performing the following for each keypoint of the set of keypoints:
calculating the comprehensive action vector of all pixel points in the current neighborhood range of the key point to the key point by the following formula:
wherein,
for the face key point o in the preset neighborhood psioThe resultant vector of action, w, received internallypTo said predetermined neighborhood ΨoThe distances from other key points p except the face key point o to the face key point o are normalized and calculated to obtain a null spaceThe inter-weight value d (p, o) is the distance between the key point o of the face and the other key points p, rhopThe gradient amplitude of the other key points p;
and carrying out local optimization on the coordinates of the key points according to the calculated comprehensive action vector.
Optionally, the determining, according to coordinates before and after local optimization of all the keypoints in the keypoint set, a functional relation for performing global optimization on the target keypoint includes:
establishing a global topological constraint equation according to the coordinates of all the key points in the key point set before local optimization and the coordinates after local optimization;
and solving the global topological constraint equation by a least square method to obtain the functional relation.
Optionally, the method further comprises:
when the sum of the squares of the errors of the coordinates before the local optimization of the target key points and the coordinates after the global optimization of the target key points is less than a preset threshold value, stopping updating the target key points;
and when the sum of the squares of the errors of the coordinates before the local optimization of the target key point and the coordinates after the global optimization of the target key point is not less than a preset threshold value, executing the updating operation on the target key point again.
In a second aspect, the present disclosure also provides a face keypoint device, including:
the generating module is used for pre-generating initial key points on the face image according to a preset face template;
an updating module, configured to perform the following updating operations on the target key points on the face image, where the updating module includes:
the local optimization submodule is used for determining a key point set comprising the target key points and carrying out local optimization on the coordinates of all key points in the key point set;
the global optimization submodule is used for determining a functional relation used for carrying out global optimization on the target key point according to coordinates before and after local optimization of all key points in the key point set, substituting the coordinates before local optimization updating of the target key point into the functional relation to obtain the coordinates after global optimization of the target key point, and taking the coordinates after global optimization of the target key point as a result of the current coordinate updating of the target key point;
wherein the target key point is any key point on the face image.
Optionally, the local optimization submodule is configured to:
performing the following for each keypoint of the set of keypoints:
calculating the comprehensive action vector of all pixel points in the current neighborhood range of the key point to the key point by the following formula:
wherein,
for the face key point o in the preset neighborhood psioThe resultant vector of action, w, received internallypTo said predetermined neighborhood ΨoThe distances from other key points p except the face key point o to the face key point o are normalized and calculated to obtain a space weight, d (p, o) is the distance from the face key point o to the other key points p, and rhopThe gradient amplitude of the other key points p;
and carrying out local optimization on the coordinates of the key points according to the calculated comprehensive action vector.
Optionally, the global optimization submodule is configured to:
establishing a global topological constraint equation according to the coordinates of all the key points in the key point set before local optimization and the coordinates after local optimization;
and solving the global topological constraint equation by a least square method to obtain the functional relation.
Optionally, the update module is further configured to:
when the sum of the squares of the errors of the coordinates before the local optimization of the target key points and the coordinates after the global optimization of the target key points is less than a preset threshold value, stopping updating the target key points;
and when the sum of the squares of the errors of the coordinates before the local optimization of the target key point and the coordinates after the global optimization of the target key point is not less than a preset threshold value, executing the updating operation on the target key point again.
In a third aspect, the present disclosure also provides a computer-readable storage medium having a computer program stored thereon, where the program is to be executed by a processor to implement the steps of the method according to any one of the first aspect.
In a fourth aspect, the present disclosure also provides an electronic device, including:
a memory having a computer program stored thereon;
a processor for executing the computer program in the memory to implement the steps of the method of any of the first aspects.
By the technical scheme, the initial key points can be pre-generated on the face image according to the preset face template, a large number of training samples do not need to be marked, and the calculation cost is saved. Moreover, the technical scheme can update the target key points on the face image, realize local optimization and global optimization of the target key points, and further improve the accuracy of positioning the face key points. The technical scheme provided by the disclosure saves the calculation cost on the basis of ensuring the accuracy of the positioning of the key points of the human face.
Additional features and advantages of the disclosure will be set forth in the detailed description which follows.
Drawings
The accompanying drawings, which are included to provide a further understanding of the disclosure and are incorporated in and constitute a part of this specification, illustrate embodiments of the disclosure and together with the description serve to explain the disclosure without limiting the disclosure. In the drawings:
FIG. 1 is a flow chart illustrating a method of face keypoint localization according to an exemplary embodiment of the present disclosure;
fig. 2 is a schematic diagram illustrating a preset face model according to an exemplary embodiment of the present disclosure;
FIG. 3 is a schematic diagram illustrating the acquisition of width, length, and center point coordinates of a binocular region and a mouth region according to an exemplary embodiment of the present disclosure;
FIG. 4 is another flowchart illustrating a face keypoint location method according to an exemplary embodiment of the present disclosure;
FIG. 5 is a schematic process diagram of an iterative update using a face keypoint localization method according to an exemplary embodiment of the present disclosure;
FIG. 6 is a block diagram illustrating a face keypoint locating apparatus according to an exemplary embodiment of the present disclosure;
fig. 7 is a block diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure.
Detailed Description
The following detailed description of specific embodiments of the present disclosure is provided in connection with the accompanying drawings. It should be understood that the detailed description and specific examples, while indicating the present disclosure, are given by way of illustration and explanation only, not limitation.
Fig. 1 is a flowchart illustrating a face keypoint locating method according to an exemplary embodiment of the present disclosure, and referring to fig. 1, the face keypoint locating method includes the following steps:
step S101, according to a preset face template, an initial key point is pre-generated on a face image.
In this disclosure, the preset face template may be determined according to an average face, and the preset face template may include 58 face key points, may also include other numbers of face key points, and the like, which is not limited in this disclosure.
Further, the following update operations from step S1021 to step S1023 are performed for the target key points on the face image:
step S1021, a key point set comprising target key points is determined, and local optimization is carried out on coordinates of all key points in the key point set.
The target key point is any one key point on the face image, and may be one key point or multiple key points, which is not limited in the embodiment of the present disclosure.
Step S1022, determining a functional relation for performing global optimization on the target keypoint according to the coordinates before and after local optimization of all the keypoints in the keypoint set.
And S1023, substituting the coordinates before the local optimization updating of the target key point into the function relation to obtain the coordinates after the global optimization of the target key point, and taking the coordinates after the global optimization of the target key point as the result of the current coordinate updating of the target key point.
By the technical scheme, the initial key points can be pre-generated on the face image according to the preset face template, a large number of training samples do not need to be marked, and the calculation cost is saved. Moreover, the technical scheme can update the target key points on the face image, realize local optimization and global optimization of the target key points, and further improve the accuracy of positioning the face key points. The technical scheme provided by the disclosure saves the calculation cost on the basis of ensuring the accuracy of the positioning of the key points of the human face.
In order to make those skilled in the art understand the technical solutions provided by the embodiments of the present disclosure, the following steps are exemplified in detail.
First, in an exemplary embodiment, the preset face template may be as shown in fig. 2, in which 58 face key points with corresponding serial numbers are included, and the correspondence relationship between the coordinates of the part of face key points in fig. 2 and the serial numbers thereof may refer to table 1:
TABLE 1
| Serial number | Coordinates of the object | Serial number | Coordinates of the object |
| 1 | (-0.438we,pe+0.625he) | 35 | (-0.125wm,pm+hm) |
| 2 | (-0.344we,pe+0.688he) | 36 | (-0.125wm,pm+1.25hm) |
| 3 | (-0.250we,pe+0.75he) | 37 | (-0.125wm,pe-0.25he) |
| 4 | (-0.156we,pe+0.688he) | 38 | (-0.375wm,pm+0.25hm) |
| 5 | (-0.063we,pe+0.5he) | 39 | (-0.125wm,pm+0.375hm) |
| 11 | (-0.375we,pe) | 40 | (0,pm+0.313hm) |
| 12 | (-0.313we,pe+0.2he) | 44 | (0,pm-0.1hm) |
| 13 | (-0.250we,pe+0.25he) | 45 | (-0.125wm,pm-0.1hm) |
| 14 | (-0.188we,pe+0.2he) | 58 | (-0.438wm,pm+0.3hm) |
| 15 | (-0.125we,pe) | 57 | (-0.381wm,pm+0.05hm) |
| 16 | (-0.188we,pe-0.2he) | 56 | (-0.325wm,pm-0.17hm) |
| 17 | (-0.250we,pe-0.25he) | 55 | (-0.25wm,pm-0.4hm) |
| 18 | (-0.313we,pe-0.2he) | 54 | (-0.175wm,pm-0.6hm) |
| 32 | (0,pm+0.625hm) | 53 | (-0.1wm,pm-0.769hm) |
| 33 | (-0.125wm,pm+0.75hm) | 52 | (0,pm-0.833hm) |
| 34 | (-0.25wm,pm+0.75hm) |
Wherein, we、he、peRespectively being eyes in face imagesWidth, height and center point coordinates of the region, wm、hm、pmThe width, the height and the center point coordinate of a mouth area in the face image are respectively shown, wherein the eye area is a rectangular area comprising eyes, and the mouth area comprises a rectangular area of mouth.
It should be understood that, referring to fig. 2, the face key points with serial numbers 1-5 are respectively symmetrical to the face key points with serial numbers 6-10, the coordinates of the face key points with serial numbers 6-10 can be determined according to the coordinates of the face key points with serial numbers 1-5, and the coordinates of the face key points with serial numbers 19-26 can be determined according to the face key points with serial numbers 11-18, and so on. Therefore, the correspondence between the coordinates of the key points of the partial face and the serial numbers thereof is listed in table 1.
According to a preset face template, pre-generating an initial key point on a face image, firstly, acquiring a face region in the face image, respectively extracting a binocular region and a mouth region in the face region, respectively determining a length value, a width value and a center point coordinate of the binocular region and the mouth region, and then generating the initial key point according to the length value, the width value and the center point coordinate of the binocular region, the length value, the width value and the center point coordinate of the mouth region and the preset face template.
For example, extracting the binocular region and the mouth region in the face region may be extracting by using a binocular extractor and a mouth extractor provided by OpenCV (Open source computer Vision Library), and accordingly, determining the length value and the width value of the binocular region and the mouth region may be determining the length value and the width value of the binocular region and the mouth extractor, and the center point coordinates of the binocular region and the mouth region may be determined according to the center points of the binocular extractor and the mouth extractor and a preset coordinate system established on the face region.
For example, for a face image, a face region in the face image is extracted, and a binocular region and a mouth region in the face region are extracted by a binocular extractor and a mouth extractor, respectively, then the binocular extractor and the mouth extractor may be used to extract the face region and the mouth regionThe width value and the length value of the extractor are respectively determined as the width value and the length value of the binocular region and the mouth region. Meanwhile, referring to fig. 3, the central points of the binocular extractor and the mouth extractor are determined, and then a preset coordinate system is established with the midpoint of the line connecting the two central points as the origin, so that the central point coordinates of the binocular region and the mouth region can be determined according to the preset coordinate system and the central points of the binocular extractor and the mouth extractor, for example, the width value of the binocular region can be recorded as weThe length of the binocular region is recorded as heAnd the coordinate of the central point of the two-eye area is recorded as peThe width of the mouth region is denoted as wmLength of mouth area is denoted as hmThe coordinate of the center point of the mouth area is recorded as pm。
Then, referring to table 1, the initial key points of the face image are calculated according to the widths, lengths and center point coordinates of the binocular region and the mouth region.
Through the method, the initial key points can be pre-generated on the face image according to the preset face template, a large number of training samples are not required to be marked, the calculation cost is saved, and the initial key point positioning result of a single face image can be quickly obtained.
After the initial key points are obtained, the target key points that need to be positioned and updated may be determined, for example, the initial key points that need to be positioned and updated are selected by a user operation, or a coordinate range of the initial key points that need to be positioned and updated, for example, a coordinate range of the eye region, is preset, so as to update the coordinates of the initial key points of the eye region.
Specifically, the embodiments of the present disclosure may specifically select the set of key points according to actual computing power and accuracy requirements. For example, in implementation, the key points in the same region may be numbered sequentially, for example, all the key points in the right eye region shown in fig. 2 are numbered 11, 12, 13, 14, 15, 16, 17, and 18, so that when updating is performed on the target key point, a plurality of adjacent numbered key points may be selected as the key point set. A set of keypoints that includes the target keypoints may also be randomly selected. All the key points (for example, 58 key points in total) on the preset face template can also be used as the key point set.
Further, after determining the set of keypoints, the coordinates of all the keypoints in the set of keypoints can be locally optimized. For example, the local optimization updating of the coordinates of all the key points in the key point set may be to locally optimize the coordinates of each key point in the key point set according to a comprehensive action vector of all pixel points in a neighborhood range where the key point is currently located to the key point.
The neighborhood range can be used to represent the iterative update speed of the key point, and the smaller the neighborhood range is determined to be, the faster the iterative update speed of the key point is, however, it should be noted that the small neighborhood range easily causes the key point positioning result to be locally optimal, which results in low accuracy of the key point positioning, and therefore, the neighborhood range cannot be set too small. In the embodiment of the present disclosure, the neighborhood range may be determined by obtaining a response map of the key point, or may be determined according to the accuracy requirement of the user for positioning the key point, and the like.
The comprehensive action vector is used to represent the influence of all pixel points in the current neighborhood range of the key point on the position coordinates of the key point, or, the comprehensive action vector is the position offset of the key point in the current neighborhood range.
In the embodiment of the present disclosure, the comprehensive action vector of all pixel points in the current neighborhood range of the key point to the key point may be calculated by the following formula:
wherein,
for the face key point o in the preset neighborhood psioThe resultant vector of action, w, received internallypTo said predetermined neighborhood ΨoThe distances from other key points p except the face key point o to the face key point o are normalized and calculated to obtain a space weight, wherein d (p, o) is the distance from the face key point o to the other key points p, and rhopThe gradient magnitude of the other keypoints p.
In the above formula, the gradient magnitude ρ of other key points ppMay be determined by the following formula:
ρp＝cos(θ)[hp,x,hp,y]T(2)
wherein, [ h ]p,x,hp,y]TIs the gradient vector of other key points p, theta is the vector
The gradient vectors of other key points p.
Space weight wpMay be determined by the following formula:
wherein z is a preset normalization coefficient, xpAnd ypRespectively the abscissa and ordinate, sigma, of other key points pwThe specific calculation method for the standard deviation of the distances from other key points p to the face key point o is similar to the related art method, and is not described herein again.
After the comprehensive action vector is calculated through the formula, the coordinates of the key points can be updated according to the calculated comprehensive action vector, that is, the coordinates corresponding to the comprehensive action vector can be superposed on the basis of the key points of the face, so that the key points of the face move towards the edge direction. When a face keypoint moves to a flat edge, the gradient vector [ h ] is due to the other keypoints p being located on the edgep,x,hp,y]TPerpendicular to the edge, then vector
And gradient vector [ hp,x,hp,y]TVertically, cos (θ) will approach to 0, then the displacement of the face key point due to other key points will approach to 0, that is, the synthetic action vector of all pixel points in the neighborhood range where the face key point is currently located to the face key point is 0, and at this time, the distribution of the face feature points will tend to be stable, so that the local optimization process for the face key point can be realized.
After obtaining the locally updated coordinates of each keypoint in the keypoint set, in a possible manner, step S1022 may be to first establish a global topological constraint equation according to the coordinates of all keypoints in the keypoint set before local optimization and the locally optimized coordinates, and then solve the global topological constraint equation by a least square method to obtain the functional relation.
For example, the abscissa and the ordinate of the coordinate position before the local optimization update of all the key points in the key point set may be respectively used as a vector X and a vector Y, the abscissa and the ordinate of the coordinate position after the local optimization update of all the key points in the key point set may be respectively used as a vector X 'and a vector Y', then the vector X and the vector Y are used as independent variables, the vector X 'and the vector Y' are used as dependent variables, a global topological constraint equation is established, and finally the global topological constraint equation is solved by a least square method to obtain a functional relational expression for indicating the coordinate mapping relationship before and after the update of the key points.
In the embodiment of the present disclosure, the edge contour of the key point of the human face can be limited within a reasonable range by obtaining a global topological constraint equation through the least square solution, so as to implement global optimization updating of the key point of the human face, and further improve the accuracy of positioning the key point of the human face.
It should be understood that, in the embodiment of the present disclosure, the established global topological constraint equation may be a binary quadratic polynomial, a ternary cubic polynomial, or the like, which is not limited by the embodiment of the present disclosure.
For example, the global topological constraint equation is a binary quadratic polynomial, and the coefficients to be determined of the binary quadratic polynomial are vector a and vector B with dimension 6, respectively, denoted as a ═ a1,a2,...,a6]TAnd B ═ B1,b2,...,b6]TMeanwhile, a set of key points is determined, which includes k key points in the neighborhood range of the target key point. Storing the horizontal and vertical coordinates of the k key points before local optimization updating into a vector X and a vector Y respectively, and recording the X as X ═ X1,x2,...,xk]TAnd Y ═ Y1,y2,...,yk]TAnd storing the horizontal and vertical coordinates of the k key points after local optimization updating into a vector X ' and a vector Y ', and recording the horizontal and vertical coordinates as X ' ═ X1',x2',...,xk']TAnd Y ═ Y1',y2',...,yk']TThen, taking the vector X and the vector Y as independent variables, and taking the vector X 'and the vector Y' as dependent variables, a global topological constraint equation can be obtained:
X'＝a1+a2X+a3Y+a4X·Y+a5X2+a6Y2(4)
Y'＝b1+b2X+b3Y+b4X·Y+b5X2+b6Y2(5)
finally, the coefficient vector a to be determined in (4) and equation (5) [ a ] can be determined by the least square method1,a2,a3,a4,a5,a6]Sum vector B ═ B1,b2,b3,b4,b5,b6]And solving to obtain a functional relation for carrying out global optimization updating on the target key points.
It should be noted that, when the global topological constraint equation is established, the locally optimized coordinate of the key point may be a coordinate of the key point after performing one-time local optimization, or a coordinate of the key point after performing multiple times of local optimization, which is not limited in this embodiment of the disclosure. However, it should be understood that if the coordinates of the keypoint after local optimization are the coordinates of the keypoint after multiple local optimizations, then the seat before the keypoint local optimization should be the coordinates of the keypoint before the first local optimization when the topological constraint equation is established.
Further, in the example that the global topological constraint equation is formulas (4) and (5), when the vector a and the vector B are known, the abscissa of the target key point is taken as X, the ordinate is taken as Y, the formulas (4) and (5) are substituted, and the obtained vector X 'and vector Y' are the coordinate positions after global optimization updating of the target key point, so that the inaccuracy of the local optimization result of the face key point is avoided, and the accuracy of face key point positioning is further ensured.
The above update operation is an explanation of a one-time update process of the target key point, and in specific implementation, when the sum of squares of errors of the coordinates before the local optimization of the target key point and the coordinates after the global optimization of the target key point is not less than the preset threshold, the update operation of the target key point may be executed again, and when the sum of squares of errors of the coordinates before the local optimization of the target key point and the coordinates after the global optimization of the target key point is less than the preset threshold, the update of the target key point is stopped.
The sum of the squares of the errors of the coordinates before the local optimization of the target key point and the coordinates after the global optimization may be obtained by summing the differences between the coordinates before the local optimization of the target key point and the coordinates after the global optimization after performing the square calculation, and the specific calculation method is similar to the method for calculating the sum of the squares of the errors in the related art and is not described herein again.
The preset threshold may be set in advance by a user according to a requirement for accuracy of the positioning result of the key point, and the like, which is not limited by the present disclosure. For example, the preset threshold may be set to 0.3, and when the sum of squared errors of the updated coordinate positions is less than 0.3, the update of the target keypoints is stopped, otherwise, the update operation of the target keypoints is performed again.
In addition, when the update operation on the target key point is executed again, the coordinates of other key points in the key point set corresponding to the key point, except for the target key point, may be the coordinates after the last local optimization. However, it should be understood that if the global optimization update was performed on other keypoints in the set of keypoints during the last update process, the coordinates of the other keypoints in the set of keypoints may be the coordinates of the other keypoints that were updated by the last global optimization at the time of updating the target keypoint.
The following describes a method for locating key points of a human face according to the present disclosure in a complete embodiment.
Referring to fig. 4, the method for locating face key points includes the following steps:
step S401, a face region in the face image is acquired.
Step S402, respectively extracting the binocular region and the mouth region in the face region, and respectively determining the length value, the width value and the center point coordinate of the binocular region and the mouth region.
Step S403, generating an initial key point according to the length value, the width value, and the center point coordinate of the binocular region, the length value, the width value, and the center point coordinate of the mouth region, and a preset face model.
Step S404, a key point set comprising the target key points is determined, and each key point in the key point set is locally optimized.
Step S405, establishing a functional relation based on the coordinates before and after each key point in the key point set is locally updated, and performing global optimization on the target key point according to the functional relation.
The process of the local optimization update and the global optimization update may specifically refer to the above detailed description, and will not be described herein again.
And step S406, taking the coordinates after the global optimization of the target key point as the result of the current coordinate updating of the target key point.
Step S407, determining whether the sum of squares of errors of the coordinates before the target key point local optimization and the coordinates after the target key point global optimization is less than a preset threshold.
If the sum of the squares of the errors of the coordinates before the local optimization of the target key points and the coordinates after the global optimization of the target key points is not less than the preset threshold, returning to execute the step S404, otherwise, entering the step S408.
In step S408, the update of the target key point is stopped.
By the method, the initial key points are pre-generated on the face image according to the preset face template, a large number of training samples are not required to be marked, the calculation cost is saved, and the accuracy of positioning the key points on the face image is ensured by updating the coordinates of the initial key points. Referring to fig. 5, fig. 5 shows a process of coordinate change of key points in an iterative updating process, so that it can be clearly seen that the key points near the right eyebrow of the face image are positioned more and more accurately.
Based on the same inventive concept, referring to fig. 6, an embodiment of the present disclosure further provides a face key point positioning device 60, including:
the generating module 61 is configured to pre-generate an initial key point on the face image according to a preset face template;
an updating module 62, configured to perform the following updating operations on target key points on the face image, where the updating module 62 includes:
the local optimization submodule 621 is configured to determine a key point set including the target key point, and perform local optimization on coordinates of all key points in the key point set;
a global optimization submodule 622, configured to determine a functional relation used for performing global optimization on the target key point according to coordinates before and after local optimization of all key points in the key point set, substitute the coordinates before local optimization and update of the target key point into the functional relation, obtain coordinates after global optimization of the target key point, and use the coordinates after global optimization of the target key point as a result of current coordinate update of the target key point;
wherein the target key point is any key point on the face image.
Optionally, the local optimization submodule 621 is configured to:
performing the following for each keypoint of the set of keypoints:
calculating the comprehensive action vector of all pixel points in the current neighborhood range of the key point to the key point by the following formula:
wherein,
for the face key point o in the preset neighborhood psioThe resultant vector of action, w, received internallypTo said predetermined neighborhood ΨoThe distances from other key points p except the face key point o to the face key point o are normalized and calculated to obtain a space weight, d (p, o) is the distance from the face key point o to the other key points p, and rhopThe gradient amplitude of the other key points p;
and carrying out local optimization on the coordinates of the key points according to the calculated comprehensive action vector.
Optionally, the global optimization submodule 622 is configured to:
establishing a global topological constraint equation according to the coordinates of all the key points in the key point set before local optimization and the coordinates after local optimization;
and solving the global topological constraint equation by a least square method to obtain the functional relation.
Optionally, the updating module 62 is configured to stop updating the target key point when a sum of squares of errors of the coordinates before the target key point local optimization and the coordinates after the target key point global optimization is less than a preset threshold, and perform the updating operation on the target key point again when the sum of squares of errors of the coordinates before the target key point local optimization and the coordinates after the target key point global optimization is not less than the preset threshold.
The device can pre-generate the initial key points on the face image according to the preset face template, does not need to label a large number of training samples, and saves the calculation cost. And the target key points on the face image can be updated, so that the local optimization and the global optimization of the target key points are realized, and the accuracy of positioning the face key points is improved. Namely, on the basis of ensuring the accuracy of positioning the key points of the human face, the calculation cost is saved.
With regard to the apparatus in the above-described embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.
Based on the same inventive concept, the present disclosure also provides an electronic device, comprising:
a memory having a computer program stored thereon;
a processor for executing the computer program in the memory to implement the steps of any of the above-mentioned face key point positioning methods.
In an exemplary embodiment, a block diagram of the electronic device may be as shown in FIG. 7. Referring to fig. 7, the electronic device 700 may include: a processor 701 and a memory 702. The electronic device 700 may also include one or more of a multimedia component 703, an input/output (I/O) interface 704, and a communication component 705.
The processor 701 is configured to control the overall operation of the electronic device 700 to complete all or part of the steps of the above-mentioned face keypoint location method. The memory 702 is used to store various types of data to support operation at the electronic device 700, such as instructions for any application or method operating on the electronic device 700 and application-related data, such as the coordinates of local optimization updated keypoints, global optimization updated keypoints, and so forth. The Memory 702 may be implemented by any type of volatile or non-volatile Memory device or combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic Memory, flash Memory, magnetic disk, or optical disk.
The I/O interface 704 provides an interface between the processor 701 and other interface modules, such as a keyboard, mouse, buttons, etc. These buttons may be virtual buttons or physical buttons. The communication component 705 is used for wired or wireless communication between the electronic device 700 and other devices. Wireless Communication, such as Wi-Fi, bluetooth, Near Field Communication (NFC), 2G, 3G, 4G, or 5G, or a combination of one or more of them, so that the corresponding Communication component 705 may include: Wi-Fi module, bluetooth module, NFC module.
In an exemplary embodiment, the electronic Device 700 may be implemented by one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components, for performing the above-mentioned face key point positioning method.
In another exemplary embodiment, there is also provided a computer readable storage medium comprising program instructions which, when executed by a processor, implement the steps of the above-mentioned face keypoint location method. For example, the computer readable storage medium may be the memory 702 described above comprising program instructions executable by the processor 701 of the electronic device 700 to perform the face keypoint location method described above.
The preferred embodiments of the present disclosure are described in detail with reference to the accompanying drawings, however, the present disclosure is not limited to the specific details of the above embodiments, and various simple modifications may be made to the technical solution of the present disclosure within the technical idea of the present disclosure, and these simple modifications all belong to the protection scope of the present disclosure.
It should be noted that, in the foregoing embodiments, various features described in the above embodiments may be combined in any suitable manner, and in order to avoid unnecessary repetition, various combinations that are possible in the present disclosure are not described again.
In addition, any combination of various embodiments of the present disclosure may be made, and the same should be considered as the disclosure of the present disclosure, as long as it does not depart from the spirit of the present disclosure.
Claims (8)
1. A method for locating key points of a human face is characterized by comprising the following steps:
according to a preset face template, pre-generating initial key points on a face image;
performing the following updating operation aiming at the target key points on the face image:
determining a key point set comprising the target key points, and carrying out local optimization on coordinates of all key points in the key point set;
determining a functional relation for carrying out global optimization on the target key points according to coordinates before and after local optimization of all key points in the key point set;
substituting the coordinates before the local optimization updating of the target key points into the function relation to obtain the coordinates after the global optimization of the target key points, and taking the coordinates after the global optimization of the target key points as the result of the current coordinate updating of the target key points;
wherein the target key point is any key point on the face image;
the locally optimizing coordinates of all keypoints in the set of keypoints comprises:
performing the following for each keypoint of the set of keypoints:
calculating the comprehensive action vector of all pixel points in the current neighborhood range of the key point to the key point by the following formula:
wherein,
for the face key point o in the preset neighborhood psioThe resultant vector of action, w, received internallypTo said predetermined neighborhood ΨoThe distances from other key points p except the face key point o to the face key point o are normalized and calculated to obtain a space weight, d (p, o) is the distance from the face key point o to the other key points p, and rhopThe gradient amplitude of the other key points p;
and carrying out local optimization on the coordinates of the key points according to the calculated comprehensive action vector.
2. The method according to claim 1, wherein determining a functional relation for global optimization of the target keypoint based on coordinates before and after local optimization of all keypoints in the set of keypoints comprises:
establishing a global topological constraint equation according to the coordinates of all the key points in the key point set before local optimization and the coordinates after local optimization;
and solving the global topological constraint equation by a least square method to obtain the functional relation.
3. The method according to claim 1 or 2, characterized in that the method further comprises:
when the sum of the squares of the errors of the coordinates before the local optimization of the target key points and the coordinates after the global optimization of the target key points is less than a preset threshold value, stopping updating the target key points;
and when the sum of the squares of the errors of the coordinates before the local optimization of the target key point and the coordinates after the global optimization of the target key point is not less than a preset threshold value, executing the updating operation on the target key point again.
4. A face keypoint apparatus, the apparatus comprising:
the generating module is used for pre-generating initial key points on the face image according to a preset face template;
an update module, configured to perform an update operation on a target keypoint on the face image, where the update module includes:
the local optimization submodule is used for determining a key point set comprising the target key points and carrying out local optimization on the coordinates of all key points in the key point set;
the global optimization submodule is used for determining a functional relation used for carrying out global optimization on the target key point according to coordinates before and after local optimization of all key points in the key point set, substituting the coordinates before local optimization updating of the target key point into the functional relation to obtain the coordinates after global optimization of the target key point, and taking the coordinates after global optimization of the target key point as a result of the current coordinate updating of the target key point;
wherein the target key point is any key point on the face image;
the local optimization submodule is configured to:
performing the following for each keypoint of the set of keypoints:
calculating the comprehensive action vector of all pixel points in the current neighborhood range of the key point to the key point by the following formula:
wherein,
for face key point o in advanceLet neighborhood ΨoThe resultant vector of action, w, received internallypTo said predetermined neighborhood ΨoThe distances from other key points p except the face key point o to the face key point o are normalized and calculated to obtain a space weight, d (p, o) is the distance from the face key point o to the other key points p, and rhopThe gradient amplitude of the other key points p;
and carrying out local optimization on the coordinates of the key points according to the calculated comprehensive action vector.
5. The apparatus of claim 4, wherein the global optimization submodule is configured to:
establishing a global topological constraint equation according to the coordinates of all the key points in the key point set before local optimization and the coordinates after local optimization;
and solving the global topological constraint equation by a least square method to obtain the functional relation.
6. The apparatus according to claim 4 or 5, wherein the updating module is further configured to stop updating the target keypoint when a sum of squared errors of the coordinates before the target keypoint local optimization and the coordinates after the target keypoint global optimization is smaller than a preset threshold;
and when the sum of the squares of the errors of the coordinates before the local optimization of the target key point and the coordinates after the global optimization of the target key point is not less than a preset threshold value, executing the updating operation on the target key point again.
7. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the method according to any one of claims 1 to 3.
8. An electronic device, comprising:
a memory having a computer program stored thereon;
a processor for executing the computer program in the memory to carry out the steps of the method of any one of claims 1 to 3.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811341716.2A CN109635659B (en) | 2018-11-12 | 2018-11-12 | Face key point positioning method and device, storage medium and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811341716.2A CN109635659B (en) | 2018-11-12 | 2018-11-12 | Face key point positioning method and device, storage medium and electronic equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109635659A CN109635659A (en) | 2019-04-16 |
| CN109635659B true CN109635659B (en) | 2020-10-30 |
Family
ID=66067811
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811341716.2A Active CN109635659B (en) | 2018-11-12 | 2018-11-12 | Face key point positioning method and device, storage medium and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109635659B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113033257B (en) * | 2019-12-24 | 2023-08-18 | 杭州海康威视数字技术股份有限公司 | Fingerprint identification method, fingerprint identification device, electronic equipment and readable storage medium |
| CN111667403B (en) * | 2020-07-02 | 2023-04-18 | 北京爱笔科技有限公司 | Method and device for generating human face image with shielding |
| CN112101106B (en) * | 2020-08-07 | 2024-05-28 | 深圳数联天下智能科技有限公司 | Face key point determining method, device and storage medium |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105354531B (en) * | 2015-09-22 | 2019-05-21 | 成都通甲优博科技有限责任公司 | A kind of mask method of face key point |
| CN106339693A (en) * | 2016-09-12 | 2017-01-18 | 华中科技大学 | Positioning method of face characteristic point under natural condition |
| CN107066932A (en) * | 2017-01-16 | 2017-08-18 | 北京龙杯信息技术有限公司 | The detection of key feature points and localization method in recognition of face |
-
2018
- 2018-11-12 CN CN201811341716.2A patent/CN109635659B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN109635659A (en) | 2019-04-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109635659B (en) | Face key point positioning method and device, storage medium and electronic equipment | |
| US11734471B2 (en) | Topology optimization for subtractive manufacturing techniques | |
| TW201828105A (en) | Word vector processing method and apparatus | |
| CN108830385B (en) | Deep learning model training method and device and computer readable storage medium | |
| CN104268591A (en) | Face key point detecting method and device | |
| CN112085033B (en) | Template matching method and device, electronic equipment and storage medium | |
| US11842262B2 (en) | Techniques for analyzing vehicle design deviations using deep learning with neural networks | |
| CN110796135B (en) | Target positioning method and device, computer equipment and computer storage medium | |
| CN112307876B (en) | Method and device for detecting node | |
| CN110806211A (en) | Method and device for robot to autonomously explore and establish graph and storage medium | |
| CN103646278A (en) | Application of particle swarm algorithm based on adaptive strategy in robot path planning | |
| CN108846855A (en) | Method for tracking target and equipment | |
| KR102282730B1 (en) | Method and system for object detection and posture estimation based on object-customized image feature detection algorithm | |
| CN111460234A (en) | Graph query method and device, electronic equipment and computer readable storage medium | |
| CN108122202A (en) | A kind of map Linear element multi-scale information derived method based on Shape context | |
| CN106802958A (en) | Conversion method and system of the CAD data to GIS data | |
| CN107506572B (en) | Method and device for acquiring height of target point | |
| CN106202247B (en) | A kind of collision checking method based on longitude and latitude | |
| CN105741345A (en) | Point cloud normal vector adjusting method and system | |
| WO2021007194A1 (en) | Anti-collision well trajectory design | |
| US9928331B2 (en) | Method and control device for circuit layout migration | |
| CN106934141B (en) | Acceleration method for calculating resistance based on long-edge cutting | |
| CN112424628A (en) | Positioning device | |
| CN110728359A (en) | Method, device, equipment and storage medium for searching model structure | |
| CN116229116A (en) | Process multiplexing processing method and system based on similar parts and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |