WO2019233229A1 - Image fusion method, apparatus, and storage medium - Google Patents

Image fusion method, apparatus, and storage medium Download PDF

Info

Publication number
WO2019233229A1
WO2019233229A1 PCT/CN2019/085841 CN2019085841W WO2019233229A1 WO 2019233229 A1 WO2019233229 A1 WO 2019233229A1 CN 2019085841 W CN2019085841 W CN 2019085841W WO 2019233229 A1 WO2019233229 A1 WO 2019233229A1
Authority
WO
WIPO (PCT)
Prior art keywords
face
dimensional
image
model
fused
Prior art date
Application number
PCT/CN2019/085841
Other languages
French (fr)
Chinese (zh)
Inventor
周景锦
程培
钱梦仁
傅斌
Original Assignee
腾讯科技（深圳）有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 腾讯科技（深圳）有限公司 filed Critical 腾讯科技（深圳）有限公司
Publication of WO2019233229A1 publication Critical patent/WO2019233229A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/77—Retouching; Inpainting; Scratch removal
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/60—Rotation of whole images or parts thereof
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30196—Human being; Person
    • G06T2207/30201—Face

Definitions

  • the present application relates to the field of image processing technologies, and in particular, to an image fusion method, device, and storage medium.
  • Image fusion technology is a key technology in computer vision and has a wide range of application scenarios. For example, face change, average face, crazy face change, etc. are popular with users.
  • the key technology is face fusion technology.
  • Face fusion is an application of image fusion technology, which mainly fuses the face area in the face image containing the user's face into the template image containing the face to replace the face in the template image, for example To migrate the pixels of the face area in the face image to the corresponding face area in the template image.
  • Embodiments of the present application provide an image fusion method, device, and storage medium, which are intended to improve the effect of fusing faces in an image.
  • An image fusion method includes:
  • An image fusion device includes:
  • a construction unit configured to obtain a two-dimensional image to be fused, and construct a three-dimensional face model according to a face in the two-dimensional image to be fused;
  • An obtaining unit configured to obtain a template image, and obtain a rotation parameter of a face in the template image in a three-dimensional space
  • a rotation unit configured to rotate the three-dimensional face model based on the rotation parameters to obtain a rotated three-dimensional face model
  • a converting unit configured to convert the three-dimensional model of the rotated face into a two-dimensional face image
  • a fusion unit is configured to fuse the two-dimensional face image and the template image.
  • a storage medium stores a plurality of instructions, and the instructions are suitable for loading by a processor to execute steps in any image fusion method provided by the implementation of the present application.
  • FIG. 1 is a schematic diagram of an image fusion method according to an embodiment of the present application
  • FIG. 2 is a schematic flowchart of an image fusion method according to an embodiment of the present application.
  • FIG. 3 (a) is a schematic diagram of a two-dimensional image to be fused provided by an embodiment of the present application
  • FIG. 3 (b) is a schematic diagram of a template image provided by an embodiment of the present application.
  • FIG. 4 (a) is a schematic diagram of a three-dimensional face model provided by an embodiment of the present application.
  • FIG. 4 (b) is a schematic diagram of a rotated three-dimensional human face model provided by an embodiment of the present application.
  • FIG. 5 is a schematic diagram of a two-dimensional face image provided by an embodiment of the present application.
  • FIG. 6 (a) is a schematic diagram of obtaining a facial feature point by detecting a facial feature point according to an embodiment of the present application
  • FIG. 6 (b) is a schematic diagram of obtaining a triangle region of a human face by dividing a triangle according to an embodiment of the present application
  • FIG. 7 is a schematic diagram of a fused image provided by an embodiment of the present application.
  • FIG. 8 is another schematic flowchart of an image fusion method according to an embodiment of the present application.
  • FIG. 9 is a schematic structural diagram of an image fusion apparatus according to an embodiment of the present application.
  • FIG. 10 is another schematic structural diagram of an image fusion apparatus according to an embodiment of the present application.
  • FIG. 11 is a schematic structural diagram of a terminal according to an embodiment of the present application.
  • the inventors of the present application found that some face fusion schemes are only used when the orientation of the face in the template image and the face image are the same (for example, all face faces or all side faces).
  • a good fusion effect can be obtained, and in general, the face direction of a face image taken by a user is difficult to be consistent with the face direction in the template image. For example, if you ask the user to take a face image that includes a side face, and try to maintain the same side face pose as in the template image, it is difficult for the user's side face pose and angle to reach the Side faces are consistent.
  • the orientation of the face in the template image and the face image are inconsistent, the feature points of the face in the template image and the face image will deviate greatly, and the fusion can only be forced.
  • the face in the template image is a side face
  • the face in the face image is a positive face
  • some areas of the side face in the template image are not visible, such as blocking one side of the nose
  • the face All the positive face areas in the image will be fused into the template image.
  • the facial feature points between the side face in the template image and the positive face in the face image will be biased. Forcibly performing fusion will cause distortion and bias. Shift etc.
  • the face direction (such as posture and angle) in the face image provided by the user is difficult to be consistent with the template image, so the obtained face fusion effect is compared. difference.
  • the orientation of the face in the template image and the face image are inconsistent (for example, the face in the template image is a side face and the face in the face image is a positive face), forcing the fusion will cause errors such as distortion and offset. , Resulting in very poor face fusion.
  • the embodiments of the present application provide an image fusion method, device, and storage medium, which can improve the effect of fusing faces in an image.
  • FIG. 1 is a schematic diagram of a scene of an image fusion method provided by an embodiment of the present application.
  • the image fusion device may be specifically integrated in a computing device such as a tablet, a mobile phone, a notebook computer, and a desktop computer that has a storage unit and a microprocessor and has a computing capability, such as a terminal.
  • the terminal may be used to receive an image fusion instruction input by a user, and obtain a two-dimensional image to be fused based on the image fusion instruction, where the two-dimensional image to be fused contains a user's face.
  • the terminal may send an image acquisition request to the server, and receive the server based on The returned two-dimensional image to be fused; or the user's image is collected through a camera to obtain the two-dimensional image to be fused, and so on.
  • a three-dimensional face model may be constructed based on the faces in the two-dimensional image to be fused.
  • a model acquisition request may be sent to the server, and a preset face model returned by the server based on the model acquisition request may be received according to the preset face.
  • the model obtains the 3D coordinate points of the face in the 2D image to be fused in the 3D plane, and constructs a 3D face model based on the 3D coordinate points.
  • the 3D face model can be uploaded to the server for processing. storage.
  • a template image may be obtained, and the template image includes a human face.
  • a template image may be obtained from a server.
  • a rotation parameter of the human face in the template image in a three-dimensional space may be obtained.
  • the three-dimensional face model is rotated based on the rotation parameters to obtain a rotated three-dimensional face model, and the rotated three-dimensional face model is converted into a two-dimensional face image.
  • the two-dimensional face image can be fused with the template image.
  • the two-dimensional face image and the template image can be detected separately for the feature points of the face, and the corresponding corresponding feature points of the face can be obtained. Fusion the face in the two-dimensional face image to the area of the face in the template image to obtain the fused image; and so on.
  • the scene schematic diagram of the image fusion method shown in FIG. 1 is only an example, and the scenario of the image fusion method described in the embodiment of the present application is to more clearly illustrate the technical solution of the embodiment of the present application, and does not constitute Limitations of the technical solutions provided in the embodiments of the present application. With the evolution of image fusion methods and the emergence of new business scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.
  • the image fusion device can be specifically integrated into a tablet computer, a mobile phone, a notebook computer, and a desktop computer, which have a storage unit and a microprocessor and have computing capabilities. In the terminal.
  • An image fusion method includes: obtaining a two-dimensional image to be fused, and constructing a three-dimensional face model based on a face in the two-dimensional image to be fused; obtaining a template image, and obtaining rotation of a human face in the template image in a three-dimensional space Parameters; rotating the 3D face model based on the rotation parameters to obtain a rotated 3D face model; converting the rotated 3D face model into a 2D face image; and fusing the 2D face image with a template image.
  • FIG. 2 is a schematic flowchart of an image fusion method according to an embodiment of the present application.
  • the image fusion method may include:
  • step S101 a two-dimensional image to be fused is acquired, and a three-dimensional human face model is constructed according to a human face in the two-dimensional image to be fused.
  • the two-dimensional image to be fused may include one or more faces, and may also include other objects, wherein the face may be a frontal face (that is, a front face), or an arbitrary angled face (that is, a side face) Face) and so on.
  • the two-dimensional image to be fused may include a front face.
  • the two-dimensional image to be fused can be an image collected by a camera during the image fusion device's photographing process; the two-dimensional image to be fused can also be a pre-stored image of the image fusion device, and the two-dimensional image to be fused can also be downloaded on the Internet. Search or image obtained from the image database; of course, the two-dimensional image to be fused may also be an image obtained by other methods, and the specific acquisition method is not limited here.
  • the image fusion device may construct a three-dimensional face model according to the human face in the two-dimensional image to be fused.
  • the step of constructing a three-dimensional face model based on a face in a two-dimensional image to be fused may include:
  • the preset face model may be a three-dimensional average face model, and the three-dimensional average face model may be preset.
  • the image fusion device may receive a setting instruction input by the user and set a preset face model based on the setting instruction; or, the image fusion device may send a model acquisition request to the server and receive the preset face model returned by the server based on the model acquisition request. After obtaining the preset face model, the image fusion device can obtain the three-dimensional coordinate points of the face in the two-dimensional image to be fused in the three-dimensional plane according to the preset face model, and construct a three-dimensional face model according to the three-dimensional coordinate points.
  • the step of obtaining a three-dimensional coordinate point of a face in a two-dimensional image to be fused in a three-dimensional plane according to a preset face model, and constructing a three-dimensional face model according to the three-dimensional coordinate point may include:
  • the face in the process of the image fusion device constructing a three-dimensional face model based on the faces in the two-dimensional image to be fused, the face can be extracted from the two-dimensional image to be fused, and the face projection in the two-dimensional image to be fused is obtained.
  • the first projection parameter may be a coordinate point of a human face in a two-dimensional plane.
  • the image fusion device obtains a second projection parameter of the preset face model projected in the two-dimensional plane, and the second projection parameter may be a coordinate point of the preset face model in the two-dimensional plane.
  • the position information of the three-dimensional average point model of the three-dimensional average face model can be changed to generate three-dimensional face models of users with different expressions and different identities, that is, all three-dimensional face models can be obtained by The offset is added to the three-dimensional coordinate points to represent it. Therefore, the three-dimensional face model of the user's face can be expressed by the following formula (1):
  • M in the formula (1) represents a three-dimensional face model of a human face, Represents a preset face model (ie, a three-dimensional average face model), N is the number of points of the three-dimensional coordinate point included in the three-dimensional average face model;
  • a id P id represents an identity offset term, A id is the identity base of m id , P id is the identity parameter;
  • a exp P exp represents the expression offset term, A exp is the expression base of m exp dimension, P exp is the expression parameter.
  • the image fusion device may according to the first projection parameter and the second projection parameter, Obtain the three-dimensional coordinate points of the human face in the three-dimensional plane, for example, as shown in the following formula (2) and formula (3):
  • X 3d represents the three-dimensional coordinate point of the preset face model
  • M keypoint represents the second projection parameter of the preset face model in the two-dimensional plane
  • p is a projection function
  • X 2d is a projection of the face in the two-dimensional plane.
  • the projection may include orthogonal projection, perspective projection, and the like. Taking orthogonal projection for 3D modeling as an example, orthogonal projection can be performed by the following formula (4):
  • formula (2) can be solved iteratively by formula (1), formula (3), and formula (4), and the identity parameter can be obtained P id and expression parameter P exp , scaling factor S, rotation coefficient R, and two-dimensional translation component T.
  • P id , P exp , S, R, T a three-dimensional coordinate point is obtained, and the three-dimensional coordinate point can be generated.
  • this three-dimensional face model is a three-dimensional face model that needs to be obtained.
  • a three-dimensional face model constructed based on a face in a two-dimensional image to be fused is shown in FIG. 4 (a).
  • the three-dimensional face model may also be obtained through other methods, and the specific construction method is not limited here.
  • the image fusion device may arbitrarily extract a face from the two-dimensional image to be fused to construct a three-dimensional face model, or the image fusion device may Output a selection item and receive a user to select a face from the two-dimensional image to be fused based on the selection item to construct a three-dimensional face model, or the image fusion device may extract all human faces from the two-dimensional image to be fused, and Construct a three-dimensional face model based on each face; etc.
  • step S102 a template image is obtained, and a rotation parameter of a human face in the template image in a three-dimensional space is obtained.
  • step S102 may be performed before step S101, or may be performed after step S101.
  • the specific execution order may be flexibly set according to actual needs, which is not limited here.
  • the template image may include one or more human faces, and may also include other objects, where the human face may be a front human face (that is, a front face), or a human face at any angle (that is, a side face). .
  • the template image may include a side face.
  • the template image can be an image collected by the image fusion device through a camera, or an image preset by the image fusion device, or an image obtained by searching on the Internet or obtained from an image database; of course, the template image is also It can be obtained through other methods, and the specific acquisition method is not limited here.
  • the image fusion device can obtain the rotation parameters of the face in the template image in the three-dimensional space, and the rotation parameter can be the turning of the face in the X-axis, Y-axis, and Z-axis directions in the three-dimensional space.
  • the rotation parameter of the normal face in the template image in three-dimensional space can be set to 0;
  • the human face in the template image is a side face, the The rotation parameters of the side face with respect to the front face are: an x angle rotated in the X axis direction, a y angle rotated in the Y axis direction, and a z angle rotated in the Z axis direction.
  • the image fusion device may receive a rotation parameter input by a user and set the rotation parameter as a rotation parameter of a face in a template image in a three-dimensional space, that is, the rotation parameter may be manually set.
  • the step of obtaining the rotation parameters of the human face in the template image in the three-dimensional space may include: constructing a three-dimensional model based on the human face in the template image; and acquiring the human face in the template image in the three-dimensional space based on the three-dimensional model. Rotation parameters.
  • the step of constructing a three-dimensional model based on a face in a template image may include:
  • the image fusion device may extract a human face from the template image, and obtain a third projection parameter of the human face in the template image projected in the two-dimensional plane, and the third projection parameter may be that the human face is in the two-dimensional plane. Coordinate point.
  • the image fusion device can obtain a fourth projection parameter of a preset face model (the preset face model is consistent with the above-mentioned preset face model, that is, a three-dimensional average face model) in a two-dimensional plane.
  • the fourth projection parameter may be a coordinate point of a preset face model in a two-dimensional plane.
  • the above formulas (1), (2), (3), and (4) can be used to calculate the target three-dimensional coordinate points of the human face in the template image in the three-dimensional plane according to the third projection parameter and the fourth projection parameter. , And build a 3D model based on the target 3D coordinate points.
  • the rotation parameter of the face in the three-dimensional space in the template image can be obtained based on the three-dimensional model, and the rotation parameter is the rotation coefficient R in the above formula (4).
  • step S103 the three-dimensional face model is rotated based on the rotation parameters to obtain a rotated three-dimensional face model.
  • the image fusion device can apply the rotation parameters to the 3D face model, and control the 3D face model to rotate in the X, Y, and Z directions in the 3D space. , So that the direction of the face (including pose and angle, etc.) in the three-dimensional face model is consistent with the direction of the face in the template image.
  • the 3D face model is rotated based on the rotation parameters, and the rotated 3D face model is obtained as shown in FIG. 4 (b).
  • the template may be based on the template.
  • the rotation parameters of the side face relative to the front face in the three-dimensional space are used to rotate the front face in the three-dimensional face model, and the face in the three-dimensional face model after rotation is obtained as the side face.
  • the face in the three-dimensional face model is a side face and the face in the template image is a positive face
  • the rotation parameters of the positive face in the template image relative to the side face in the three-dimensional space are used to rotate the side face in the three-dimensional face model to obtain that the face in the three-dimensional face model after rotation is the positive face.
  • the image fusion device can pre-process the rotated 3D face model, for example, by adjusting the brightness, so that the light on the face in the rotated 3D face model is illuminated.
  • the environment remains comparable to the template map.
  • step S104 the three-dimensional model of the face after the rotation is converted into a two-dimensional face image.
  • the image fusion device may convert the three-dimensional model of the rotated face into a two-dimensional face image.
  • the two-dimensional face image may be as shown in FIG. 5.
  • a screenshot instruction may be received.
  • the screenshot instruction directly captures the image displayed on the display interface of the rotated three-dimensional model of the face to obtain a two-dimensional face image; or, according to the method of constructing a three-dimensional face model based on the face described above, the inverse mapping is performed to convert the rotated three-dimensional face model
  • the inverse mapping is a two-dimensional face image.
  • step S105 the two-dimensional face image and the template image are fused.
  • the image fusion device can first perform face feature point detection, then divide the face triangle area according to the face feature points, then perform affine transformation according to the face triangle area, and finally each person in the face in the two-dimensional face image
  • the triangular area of the face is fused to the corresponding position of the face in the template image, and some post-processing operations such as edge softening can also be performed to complete the face image fusion.
  • the image fusion device may separately construct a three-dimensional face model according to each face in the two-dimensional image to be fused, and obtain a template image.
  • the rotation parameters of each face in the three-dimensional space are based on each rotation parameter to rotate each three-dimensional face model to obtain a rotated three-dimensional face model; and then convert each rotated three-dimensional model into a two-dimensional face Image, and fuse each face in the two-dimensional face image with each face in the template image.
  • a three-dimensional person may be constructed based on the first face and the second face, respectively.
  • the face model, and the rotation parameters of the third face and the fourth face in the three-dimensional space may be constructed based on the first face and the second face, respectively.
  • the three-dimensional face model corresponding to the first face is rotated, and the fourth face correspondence is based on Rotate the 3D face model corresponding to the second face to obtain a rotated 3D face model; then convert each rotated 3D model into a 2D face image, and finally convert the 2D face image into The first face in the image is fused with the third face in the template image, and the second face in the two-dimensional face image is fused with the fourth face in the template image.
  • the step of fusing the two-dimensional face image with the template image may include:
  • the face in the two-dimensional face image is fused to the area where the face in the template image is located.
  • the image fusion device can use face recognition technology to detect feature points on each face organ such as eyes, nose, eyebrows, and mouth in a two-dimensional face image to generate a first face feature point, the first person
  • the face feature point may be position coordinate information of a key point corresponding to each face organ.
  • the first face feature point may be an outer contour of the face and an edge or a center of each face organ.
  • the number can be 106, 83, or 68. The specific number can be flexibly set according to actual needs.
  • the image fusion device may use face recognition technology to detect feature points on each face organ such as eyes, nose, eyebrows, and mouth in the template image to generate a second face feature point, and the second face
  • the feature point may be position coordinate information of a key point corresponding to each face organ, and the second face feature point may be in the outer contour of the face and the edge or center of each face organ, etc.
  • the number can be 106, 83, or 68. The specific number can be flexibly set according to actual needs. In order to be able to accurately fuse the two-dimensional face image and the template image, the number of feature points of the first face and the number of feature points of the second face can be set to be the same.
  • face feature point detection is performed to obtain a face image containing the face feature points, as shown in FIG. 6 (a).
  • each feature point may be numbered or identified in other ways, for example, feature points numbered 56, 62, and 64.
  • the image fusion device may fuse the face in the two-dimensional face image to the area where the face in the template image is located according to the first face feature point and the second face feature point.
  • the step of fusing a face in a two-dimensional face image to a region where the face is in a template image according to the first face feature point and the second face feature point includes:
  • the image fusion device may perform triangle division on the face in the two-dimensional face image according to the first face feature point to obtain the first face triangle area.
  • each face feature point is connected according to a preset algorithm. Lines, so that the face can be divided into multiple triangular regions, and the preset algorithm can be flexibly set according to actual needs.
  • the face in the template image is triangle-divided according to the second face feature point to obtain a second face triangle area.
  • the face is triangle-divided according to the face feature point to obtain a face including the face triangle area.
  • the image is shown in Figure 6 (b).
  • the image fusion device may perform affine transformation based on the first face triangle area corresponding to the two-dimensional face image and the second face triangle area corresponding to the template image to obtain the target face triangle area. For example, the first face triangle area and the second face face triangle area at the corresponding position of the face triangle area can be compared. If the first face triangle area and the second face face triangle area correspond to the corresponding face triangle area. The shape and size of the target face are inconsistent. At this time, you can perform operations such as zooming, rotating, or panning on the face triangle area at the position in the first face triangle area to obtain the target face triangle area, making the target face triangle area and the second face area The shape and size of the face triangle area at the corresponding position in the face triangle area are the same.
  • the triangle area of the target face can be fused to the area of the face in the template image to obtain a fused image.
  • the fused image can be shown in FIG. 7.
  • you can use fusion algorithms such as alpha-blend or seamless clone.
  • a three-dimensional human face model is constructed based on the positive face in the two-dimensional image to be fused.
  • the face in the three-dimensional face model is a front face, and a rotation parameter of the side face in the template image relative to the front face in the three-dimensional space is obtained.
  • the rotation parameter is the side face relative to the positive face in the three-dimensional space in the template image. Face rotation parameters.
  • the three-dimensional face model is rotated based on the rotation parameters to obtain a rotated three-dimensional face model.
  • the method may include: rotating the positive face in the three-dimensional face model based on the rotation parameters to obtain a rotated three-dimensional face model.
  • the face in the three-dimensional face model is a side face.
  • the step of fusing the two-dimensional face image with the template image may include: fusing the rotated side face in the two-dimensional face image to an area where the side face in the template image is located.
  • the user In order to solve the problem of difficult side-face fusion by constructing a three-dimensional face model, the user only needs to take a normal face image normally, construct a three-dimensional face model from the front face image, and then automatically rotate the three-dimensional face model to the side of the template image. The face directions are the same. Finally, the rotated 3D face model is converted into a 2D face image, and a side face fusion is performed between the 2D face image and the template image. For example, as long as the face in the two-dimensional image to be fused is a normal face, side-face fusion with the template image of each side face can be realized, which is convenient, fast, and effective.
  • the image fusion device may first obtain a two-dimensional image to be fused, and construct a three-dimensional face model based on the face in the two-dimensional image to be fused; obtain a template image, and construct a three-dimensional model based on the face in the template image, and Based on the three-dimensional model, the rotation parameters of the face in the template image in three-dimensional space are obtained; then, the three-dimensional face model is rotated based on the rotation parameter to obtain the rotated three-dimensional face model; secondly, the person in the rotated three-dimensional face model is The face area is fused with the face area in the 3D model to obtain a fused 3D face model, and finally the fused 3D face model is converted into a 2D face image; and so on.
  • the embodiment of the present application can obtain a two-dimensional image to be fused, and construct a three-dimensional face model based on the human face in the two-dimensional image to be fused, and obtain the template image and the human face in the template image in the three-dimensional space.
  • Rotation parameters and then rotate the 3D face model based on the rotation parameters to obtain a rotated 3D face model, and then convert the rotated 3D face model into a 2D face image.
  • the 2D face image and The template image is fused, for example, the face in the two-dimensional face image is fused to the area where the face in the template image is located.
  • This solution constructs and rotates a three-dimensional face model, so that a two-dimensional face image can be obtained based on the rotated three-dimensional face model, so that the two-dimensional face image is consistent with the direction of the face in the template image, so that the two-dimensional face image
  • the face in the image is accurately fused to the area where the face in the template image is located, which improves the effect of merging the face in the image.
  • an image fusion device is used as a terminal.
  • the human face in the two-dimensional image to be fused is a positive face
  • the human face in the template image is a side face.
  • this embodiment is only for the convenience of describing the examples given, and should not be construed as limiting the direction and number of faces in the two-dimensional image and the template image, but it does not matter
  • the direction and the number of faces in the two-dimensional image and the template image are fused.
  • the image fusion process is similar and can be understood according to this example.
  • FIG. 8 is a schematic flowchart of an image fusion method according to an embodiment of the present application.
  • the method flow may include:
  • the terminal obtains a two-dimensional image and a template image to be fused.
  • the two-dimensional image to be fused may include a front face, as shown in FIG. 3 (a); the template image may include a side face, as shown in FIG. 3 (b).
  • the two to be fused two The dimensional image and the template image may also include other objects.
  • the two-dimensional image to be fused and the template image can be the image collected by the camera during the photographing process of the terminal, or the image preset by the terminal, or the image obtained by searching on the Internet or obtaining from the image database, etc. ; Of course, it can also be an image obtained by other methods, and the specific acquisition method is not limited here.
  • the two-dimensional image to be fused may be an image obtained by a user taking a self-portrait
  • the template image may be a preset image.
  • the terminal obtains a three-dimensional coordinate point of a face in a two-dimensional image to be fused in a three-dimensional plane according to a preset face model, and constructs a three-dimensional face model according to the three-dimensional coordinate point.
  • the preset face model may be a three-dimensional average face model, and the three-dimensional average face model may be preset, and different expressions and different identities may be generated by changing the position information of the three-dimensional coordinate points of the three-dimensional average face model.
  • the user's three-dimensional face model that is, all three-dimensional face models can be represented by adding offsets to the three-dimensional coordinate points of the three-dimensional average face model.
  • the terminal can obtain the three-dimensional coordinate points of the normal face in the two-dimensional image to be fused in the three-dimensional plane according to the preset face model, and construct a three-dimensional face model according to the three-dimensional coordinate points.
  • the terminal may obtain the first projection parameter of the human face in the two-dimensional image to be fused in a two-dimensional plane according to the above formula (1), formula (2), formula (3), and formula (4), and obtain Preset the second projection parameters of the face model projected in the two-dimensional plane, and then obtain the three-dimensional coordinate points of the face in the two-dimensional image to be fused in the three-dimensional plane according to the first projection parameter and the second projection parameter.
  • the three-dimensional coordinate points are used to construct a three-dimensional face model; and so on, the three-dimensional face model may be as shown in FIG. 4 (a).
  • the terminal constructs a three-dimensional model according to the face in the template image, and obtains a rotation parameter of the face in the three-dimensional space based on the three-dimensional model.
  • S203 may be performed before step S202, or may be performed after step S202.
  • the terminal may obtain the third projection parameter of the face projected in the template image in a two-dimensional plane according to the above formula (1), formula (2), formula (3), and formula (4); obtain a preset face model, And preset a fourth projection parameter in which the face model is projected in a two-dimensional plane; and then according to the third projection parameter and the fourth projection parameter, obtain the target three-dimensional coordinate point of the face in the template image in the three-dimensional plane, and according to the target 3D coordinate points to build a 3D model; etc.
  • the rotation parameter of the side face in the three-dimensional space relative to the front face in the template image can be obtained based on the three-dimensional model, and the rotation parameter is the rotation coefficient R in the above formula (4).
  • the rotation parameter may include an x-angle rotated in the X-axis direction, a y-angle rotated in the Y-axis direction, a z-angle rotated in the Z-axis direction, and the like.
  • the rotation parameter may also be set by a user.
  • the terminal rotates the three-dimensional face model based on the rotation parameters to obtain a rotated three-dimensional face model.
  • the terminal can apply the rotation parameters to the 3D face model, and control the 3D face model to rotate in the X, Y, and Z directions in the 3D space, so that the direction of the face and the template in the 3D face model Face orientation is the same in the picture.
  • the front face in the three-dimensional face model can be rotated to obtain the face in the rotated three-dimensional face model as the side face.
  • the three-dimensional face model is shown in Figure 4 (b).
  • the terminal converts the three-dimensional face model after rotation into a two-dimensional face image.
  • the terminal may receive a screenshot instruction, and directly capture an image displayed on the display interface of the rotated three-dimensional model of the face according to the screenshot instruction to obtain a two-dimensional face image; or, perform inverse mapping in accordance with the method for constructing a three-dimensional face model based on a face,
  • the three-dimensional model of the rotated face is inversely mapped into a two-dimensional face image.
  • the three-dimensional model of the rotated face can also be converted into a two-dimensional face image by other methods.
  • the two-dimensional face image may be as shown in FIG. 5, and the face in the obtained two-dimensional face image is a side face.
  • the terminal performs face feature point detection on the two-dimensional face image and the template image, respectively, and obtains the first face feature point and the second face feature point correspondingly.
  • the terminal can use face recognition technology to detect feature points on each face organ such as eyes, nose, eyebrows, and mouth in the two-dimensional face image, generate first face feature points, and separately in the template image Face points such as eyes, nose, eyebrows, and mouth of a human face are detected to generate feature points to generate a second face feature point.
  • the first facial feature point and the second facial feature point may be position coordinate information of a key point corresponding to each facial organ, and the first facial feature point and the second facial feature point may be on the outer contour of the human face. And the edges or centers of various facial organs.
  • the number of the first face feature points and the second face feature points may be 106, 83, or 68, and the specific number may be flexibly set according to actual needs. For example, face feature point detection is performed to obtain a face image containing the face feature points, as shown in FIG. 6 (a).
  • the terminal fuses the face in the two-dimensional face image to the area where the face in the template image is located according to the first face feature point and the second face feature point.
  • the terminal may perform triangle division on the face in the two-dimensional face image according to the first face feature point to obtain a triangle region of the first face, and perform triangle division on the face in the template image according to the second face feature point.
  • a second triangular area of the face for example, triangulating the face according to the facial feature points to obtain a face image containing the triangular area of the face, as shown in FIG. 6 (b).
  • the terminal may perform affine transformation based on the first face triangle area corresponding to the two-dimensional face image and the second face triangle area corresponding to the template image to obtain the target face triangle area.
  • the connection between the face feature points 56, 62, and 64 in the two-dimensional face image of Fig. 6 (a) constitutes a triangle area A1 of the face (as shown in Fig. 6 (b)).
  • the line between the face feature points 56, 62 and 64 (not shown) constitutes a face triangle area A2 (not shown).
  • the face triangle area A1 is compared with the face triangle area A2.
  • the shape and size of the area A1 and the face triangle area A2 are not consistent.
  • the face triangle area A1 can be zoomed, rotated, or translated to obtain the target face triangle area A.
  • the target face triangle area A can be
  • the face triangle area A2 has the same shape and size, and other face triangle areas also perform the same processing.
  • the terminal may fuse the triangle area of the target face to the area of the face in the template image to obtain a fused image.
  • the fused image may be as shown in FIG. 7.
  • the embodiment of the present application constructs a three-dimensional face model by using a front face in a two-dimensional image to be fused, and automatically rotates the three-dimensional face model according to a rotation parameter of a side face in a template image, so that the front face in the three-dimensional face model and the side of the template image Face directions are the same. Finally, the rotated 3D face model is converted into a 2D face image, and the 2D face image and the template image are subjected to side-face fusion. This solution improves the two-dimensional side-face fusion effect by constructing a three-dimensional face model.
  • the face in the two-dimensional image to be fused is a normal face, it can be perfectly fused with the side face at any angle in the template image to obtain the fusion.
  • the fusion effect of the post-image is very good, the face after fusion is naturally beautiful, and the whole process is automated, convenient and quick. For example, it can perfect the application scene of crazy face change, increasing the fun and playability.
  • the embodiment of the present application further provides a device based on the above image fusion method.
  • the meanings of the nouns are the same as in the above image fusion method. For specific implementation details, refer to the description in the method embodiment.
  • FIG. 9 is a schematic structural diagram of an image fusion apparatus according to an embodiment of the present application.
  • the image fusion apparatus may include a construction unit 301, an acquisition unit 302, a rotation unit 303, a conversion unit 304, and a fusion unit 305.
  • the construction unit 301 is configured to obtain a two-dimensional image to be fused, and construct a three-dimensional face model according to a face in the two-dimensional image to be fused.
  • the two-dimensional image to be fused may include one or more human faces, and may also include other objects, wherein the human face may be a front face or a side face.
  • the two-dimensional image to be fused may include a front face.
  • the two-dimensional image to be fused may be an image collected by a camera during the photographing process; it may also be an image pre-stored by an image fusion device; it may also be an image obtained by searching on the Internet or obtained from an image database, etc .; of course
  • the two-dimensional image to be fused may also be an image obtained by other methods, and the specific acquisition method is not limited here.
  • the construction unit 301 may include a first acquisition subunit, a first construction subunit, and the like, which may specifically be as follows:
  • a first acquisition subunit configured to acquire a preset face model
  • the first construction subunit is configured to obtain a three-dimensional coordinate point of a face in a two-dimensional image to be fused in a three-dimensional plane according to a preset face model, and construct a three-dimensional face model according to the three-dimensional coordinate point.
  • the preset face model may be a three-dimensional average face model, and the three-dimensional average face model may be preset.
  • the first obtaining subunit may receive a setting instruction and set a preset face model based on the setting instruction; or, the first obtaining subunit may send a model obtaining request to the server and receive the preset face model returned by the server based on the model obtaining request. .
  • the first construction subunit can obtain the three-dimensional coordinate points of the face in the two-dimensional image to be fused in the three-dimensional plane according to the preset face model, and construct a three-dimensional face based on the three-dimensional coordinate points. model.
  • the first constructing subunit may be specifically configured to: obtain a first projection parameter of a human face projected in a two-dimensional image to be fused in a two-dimensional plane; and obtain a preset face model projected in a two-dimensional plane.
  • a second projection parameter in the image obtaining a three-dimensional coordinate point of a face in a two-dimensional image to be fused in a three-dimensional plane according to the first projection parameter and the second projection parameter; and constructing a three-dimensional face model according to the three-dimensional coordinate point.
  • the human face in the process of constructing a three-dimensional face model based on the faces in the two-dimensional image to be fused by the first construction subunit, the human face can be extracted from the two-dimensional image to be fused, and the person in the two-dimensional image to be fused is obtained.
  • a first projection parameter of the face projection in the two-dimensional plane, and the first projection parameter may be a coordinate point of the human face in the two-dimensional plane.
  • the first construction subunit may obtain a second projection parameter of the preset face model projected in the two-dimensional plane, and the second projection parameter may be a coordinate point of the preset face model in the two-dimensional plane.
  • the three-dimensional face model may also be obtained through other methods, and the specific construction method is not limited here.
  • the first construction subunit may arbitrarily extract a face from the two-dimensional image to be fused to construct a three-dimensional face model, or the first The construction sub-unit may output selection items and receive a user to select a face from the two-dimensional image to be fused based on the selection items to construct a three-dimensional face model. Alternatively, the first construction sub-unit may be extracted from the two-dimensional image to be fused. Make all faces, and build 3D face models based on each face; and so on.
  • the obtaining unit 302 is configured to obtain a template image, and obtain a rotation parameter of a human face in the template image in a three-dimensional space.
  • the template image may include one or more human faces, and may also include other objects, where the human face may be a front face or a side face at any angle.
  • the template image may include a side face.
  • the template image can be an image collected through a camera, or a preset image, or an image obtained by searching on the Internet or obtained from an image database; of course, the template image can also be obtained by other methods The specific acquisition method is not limited here.
  • the obtaining unit 302 may obtain a rotation parameter of a face in a three-dimensional space in the template image, and the rotation parameter may be a turning direction of the face in the X-axis, Y-axis, and Z-axis directions in the three-dimensional space.
  • the rotation parameter of the normal face in the template image in the three-dimensional space can be set to 0; when the face in the template image is a side face, the side face in the template image can be set relative to the positive face.
  • the rotation parameters of the face are: an x angle rotated in the X axis direction, a y angle rotated in the Y axis direction, and a z angle rotated in the Z axis direction.
  • the obtaining unit 302 may receive a rotation parameter input by a user and set the rotation parameter as a rotation parameter of a face in a template image in a three-dimensional space, that is, the rotation parameter may be manually set.
  • the obtaining unit 302 may include a second construction subunit, a second obtaining subunit, and the like, and may specifically be as follows:
  • a second construction subunit configured to construct a three-dimensional model according to a face in the template image
  • the second obtaining subunit is configured to obtain a rotation parameter of the face in the template image in the three-dimensional space based on the three-dimensional model.
  • the second construction subunit may be specifically used for:
  • the second construction sub-unit may extract a human face from the template image, and obtain a third projection parameter of the human face in the template image projected in the two-dimensional plane, and the third projection parameter may be the human face in the two-dimensional plane. Coordinate points in the plane.
  • the image fusion device may obtain a fourth projection parameter of the preset face model projected in the two-dimensional plane, and the fourth projection parameter may be a coordinate point of the preset face model in the two-dimensional plane.
  • the above formulas (1), (2), (3), and (4) can be used to calculate the target three-dimensional coordinate points of the human face in the template image in the three-dimensional plane according to the third projection parameter and the fourth projection parameter.
  • the second acquisition subunit may obtain a rotation parameter of the face in the three-dimensional space in the template image based on the three-dimensional model, and the rotation parameter is the rotation coefficient R in the above formula (4).
  • the rotation unit 303 is configured to rotate the three-dimensional face model based on the rotation parameters to obtain a rotated three-dimensional face model.
  • the rotation unit 303 can apply the rotation parameters to the three-dimensional face model, and control the three-dimensional face model to rotate in the directions of X-axis, Y-axis, and Z-axis in the three-dimensional space. , So that the direction of the face in the three-dimensional face model is consistent with the direction of the face in the template image.
  • the 3D face model is rotated based on the rotation parameters, and the rotated 3D face model is obtained as shown in FIG. 4 (b).
  • the template may be based on the template.
  • the rotation parameters of the side face relative to the front face in the three-dimensional space are used to rotate the front face in the three-dimensional face model, and the face in the three-dimensional face model after rotation is obtained as the side face.
  • the face in the three-dimensional face model is a side face and the face in the template image is a positive face
  • the rotation parameters of the positive face in the template image relative to the side face in the three-dimensional space are used to rotate the side face in the three-dimensional face model to obtain that the face in the three-dimensional face model after rotation is the positive face.
  • the conversion unit 304 is configured to convert the three-dimensional model of the face after the rotation into a two-dimensional face image.
  • the conversion unit 304 may convert the three-dimensional model of the rotated face into a two-dimensional face image.
  • the two-dimensional face image may be as shown in FIG. 5.
  • a screenshot instruction may be received.
  • the screenshot instruction directly captures the image displayed on the display interface of the rotated three-dimensional model of the face to obtain a two-dimensional face image; or, according to the method of constructing a three-dimensional face model based on the face described above, the inverse mapping is performed to convert the rotated three-dimensional face model
  • the inverse mapping is a two-dimensional face image.
  • a fusion unit 305 is configured to fuse a two-dimensional face image and a template image.
  • the fusion unit 305 may first perform face feature point detection, then divide the face triangle area according to the face feature point, and then perform affine transformation based on the face triangle area, and finally each person in the face in the two-dimensional face image
  • the triangular area of the face is fused to the corresponding position of the face in the template image, and some post-processing operations such as edge softening can also be performed to complete the face image fusion.
  • the fusion unit 305 may separately construct a three-dimensional face model according to each face in the two-dimensional image to be fused, and obtain a template image.
  • the rotation parameters of each face in the three-dimensional space are based on each rotation parameter to rotate each three-dimensional face model to obtain a rotated three-dimensional face model; and then convert each rotated three-dimensional model into a two-dimensional face Image, and fuse each face in the two-dimensional face image with each face in the template image.
  • a three-dimensional person may be constructed based on the first face and the second face, respectively.
  • the face model, and the rotation parameters of the third face and the fourth face in the three-dimensional space may be constructed based on the first face and the second face, respectively.
  • the three-dimensional face model corresponding to the first face is rotated, and the fourth face correspondence is based on Rotate the 3D face model corresponding to the second face to obtain a rotated 3D face model; then convert each rotated 3D model into a 2D face image, and finally convert the 2D face image into The first face in the image is fused with the third face in the template image, and the second face in the two-dimensional face image is fused with the fourth face in the template image.
  • the fusion unit 305 may include a first detection sub-unit 3051, a second detection sub-unit 3052, and a fusion sub-unit 3053, etc., which may specifically be as follows:
  • a first detection sub-unit 3051 configured to perform face feature point detection on a two-dimensional face image to obtain a first face feature point
  • a second detection subunit 3052 configured to detect a facial feature point on the template image to obtain a second facial feature point
  • a fusion subunit 3053 is configured to fuse the face in the two-dimensional face image to the region where the face is in the template image according to the first face feature point and the second face feature point.
  • the first detection sub-unit 3051 may use face recognition technology to detect feature points on each face organ such as eyes, nose, eyebrows, and mouth in a two-dimensional face image to generate a first face feature point.
  • the first face feature point may be position coordinate information of a key point corresponding to each face organ, and the first face feature point may be in the outer contour of the face and the edge or center of each face organ.
  • the number of facial feature points can be 106, 83, or 68. The specific number can be flexibly set according to actual needs.
  • the second detection sub-unit 3052 can use face recognition technology to detect feature points on each face organ such as eyes, nose, eyebrows, and mouth in the template image to generate a second face feature point.
  • the second person The face feature point may be position coordinate information of a key point corresponding to each face organ, and the second face feature point may be in the outer contour of the face and the edge or center of each face organ, etc.
  • the second face feature point The number can be 106, 83, or 68. The specific number can be flexibly set according to actual needs. In order to be able to accurately fuse the two-dimensional face image and the template image, the number of feature points of the first face and the number of feature points of the second face can be set to be the same. For example, face feature point detection is performed to obtain a face image containing the face feature points, as shown in FIG. 6 (a).
  • the fusion subunit 3053 can fuse the face in the two-dimensional face image to the area where the face in the template image is located according to the first face feature point and the second face feature point.
  • the fusion subunit 3053 may be specifically configured to:
  • the face in the two-dimensional face image is triangle-divided to obtain the first face triangle area
  • the face in the template image is triangle-divided to obtain The second face triangle area
  • the affine transformation is performed based on the first face triangle area and the second face triangle area to obtain the target face triangle area
  • the target face triangle area is fused to the face area in the template image.
  • the fusion sub-unit 3053 may triangulate the face in the two-dimensional face image according to the first face feature point to obtain a first face triangle area.
  • each face feature point is performed according to a preset algorithm. Connect the lines so that the face can be divided into multiple triangular areas.
  • the preset algorithm can be flexibly set according to actual needs.
  • the face in the template image is triangle-divided according to the second face feature point to obtain a second face triangle area.
  • the face is triangle-divided according to the face feature point to obtain a face including the face triangle area.
  • the image is shown in Figure 6 (b).
  • the fusion subunit 3053 may perform affine transformation based on the first face triangle area corresponding to the two-dimensional face image and the second face triangle area corresponding to the template image to obtain the target face triangle area. For example, the first face triangle area and the second face face triangle area at the corresponding position of the face triangle area can be compared. If the first face triangle area and the second face face triangle area correspond to the corresponding face triangle area. The shape and size of the target face are inconsistent. At this time, you can perform operations such as zooming, rotating, or panning on the face triangle area at the position in the first face triangle area to obtain the target face triangle area, making the target face triangle area and the second face area The shape and size of the face triangle area at the corresponding position in the face triangle area are the same. At this time, the triangle area of the target face can be fused to the area of the face in the template image to obtain a fused image. For example, the fused image can be shown in FIG. 7.
  • a three-dimensional human face model is constructed based on the positive face in the two-dimensional image to be fused.
  • the face in the three-dimensional face model is a front face, and a rotation parameter of the side face in the template image relative to the front face in the three-dimensional space is obtained.
  • the rotation parameter is the side face relative to the positive face in the three-dimensional space in the template image. Face rotation parameters.
  • the rotation unit 303 may be specifically configured to rotate the positive face in the three-dimensional face model based on the rotation parameters to obtain a rotated three-dimensional face model, and the face in the rotated three-dimensional face model is a side face.
  • the fusion unit 305 may be specifically configured to fuse the rotated side face in the two-dimensional face image to the region where the side face is located in the template image.
  • the user In order to solve the problem of difficult side-face fusion by constructing a three-dimensional face model, the user only needs to take a normal face image normally, construct a three-dimensional face model from the front face image, and then automatically rotate the three-dimensional face model to the side of the template image. The face directions are the same. Finally, the rotated 3D face model is converted into a 2D face image, and a side face fusion is performed between the 2D face image and the template image. For example, as long as the face in the two-dimensional image to be fused is a normal face, side-face fusion with the template image of each side face can be realized, which is convenient, fast, and effective.
  • the construction unit 301 can obtain the two-dimensional image to be fused, and the three-dimensional face model is constructed based on the face in the two-dimensional image to be fused, and the template image and the template image are obtained by the acquisition unit 302.
  • This solution constructs and rotates a three-dimensional face model, so that a two-dimensional face image can be obtained based on the rotated three-dimensional face model, so that the two-dimensional face image is consistent with the direction of the face in the template image, so that the two-dimensional face image
  • the face in the image is accurately fused to the area where the face in the template image is located, which improves the effect of merging the face in the image.
  • an embodiment of the present application further provides a terminal.
  • the terminal may include a radio frequency (RF) circuit 601, a memory 602 including one or more computer-readable storage media, and an input unit. 603, a display unit 604, a sensor 605, an audio circuit 606, a wireless fidelity (WiFi) module 607, a processor 608 including one or more processing cores, a power supply 609, and other components.
  • RF radio frequency
  • the terminal structure shown in FIG. 11 does not constitute a limitation on the terminal, and may include more or fewer components than shown in the figure, or some components may be combined, or different component arrangements. among them:
  • the RF circuit 601 can be used to receive and send signals during the transmission and reception of information or during a call. In particular, after receiving the downlink information of the base station, it is processed by one or more processors 608. In addition, the uplink-related data is sent to the base station. .
  • the RF circuit 601 includes, but is not limited to, an antenna, at least one amplifier, a tuner, one or more oscillators, a subscriber identity module (SIM, Subscriber Identity Module) card, a transceiver, a coupler, and a low noise amplifier (LNA, Low Noise Amplifier), duplexer, etc.
  • SIM Subscriber identity module
  • LNA Low Noise Amplifier
  • the RF circuit 601 can also communicate with a network and other devices through wireless communication.
  • the wireless communication can use any communication standard or protocol, including but not limited to Global System for Mobile Communication (GSM), General Packet Radio Service (GPRS), and Code Division Multiple Access (CDMA) , Code Division Multiple Access), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), E-mail, Short Messaging Service (SMS), etc.
  • GSM Global System for Mobile Communication
  • GPRS General Packet Radio Service
  • CDMA Code Division Multiple Access
  • CDMA Code Division Multiple Access
  • WCDMA Wideband Code Division Multiple Access
  • LTE Long Term Evolution
  • E-mail Short Messaging Service
  • the memory 602 may be used to store software programs and modules, and the processor 608 executes various functional applications and data processing by running the software programs and modules stored in the memory 602.
  • the memory 602 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, at least one application required by a function (such as a sound playback function, an image playback function, etc.), etc .; the storage data area may store data according to Use data created by the terminal (such as audio data, phone book, etc.).
  • the memory 602 may include a high-speed random access memory, and may further include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 602 may further include a memory controller to provide the processor 608 and the input unit 603 to access the memory 602.
  • the input unit 603 may be used to receive inputted numeric or character information, and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.
  • the input unit 603 may include a touch-sensitive surface and other input devices.
  • a touch-sensitive surface also known as a touch display or touchpad, collects user touch operations on or near it (such as the user using a finger, stylus or any suitable object or accessory on the touch-sensitive surface or touch-sensitive Operation near the surface), and drive the corresponding connection device according to a preset program.
  • the touch-sensitive surface may include two parts, a touch detection device and a touch controller.
  • the touch detection device detects the user's touch position, and detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives touch information from the touch detection device, converts it into contact coordinates, and sends it To the processor 608, and can receive the command sent by the processor 608 and execute it.
  • various types such as resistive, capacitive, infrared, and surface acoustic waves can be used to implement a touch-sensitive surface.
  • the input unit 603 may include other input devices. Specifically, other input devices may include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, and the like.
  • the display unit 604 may be used to display information input by the user or information provided to the user and various graphical user interfaces of the terminal. These graphical user interfaces may be composed of graphics, text, icons, videos, and any combination thereof.
  • the display unit 604 may include a display panel.
  • the display panel may be configured in the form of a liquid crystal display (LCD, Liquid Crystal Display), an organic light emitting diode (OLED, Organic Light-Emitting Diode), or the like.
  • the touch-sensitive surface may cover the display panel.
  • the touch-sensitive surface When the touch-sensitive surface detects a touch operation on or near the touch-sensitive surface, the touch-sensitive surface is transmitted to the processor 608 to determine the type of the touch event, and then the processor 608 displays the touch event according to the type of the touch event.
  • the corresponding visual output is provided on the panel.
  • the touch-sensitive surface and the display panel are implemented as two separate components to implement input and input functions, in some embodiments, the touch-sensitive surface and the display panel may be integrated to implement input and output functions.
  • the terminal may further include at least one sensor 605, such as a light sensor, a motion sensor, and other sensors.
  • the light sensor may include an ambient light sensor and a proximity sensor.
  • the ambient light sensor may adjust the brightness of the display panel according to the brightness of the ambient light.
  • the proximity sensor may turn off the display panel and / or the backlight when the terminal is moved to the ear.
  • a gravity acceleration sensor can detect the magnitude of acceleration in various directions (generally three axes). It can detect the magnitude and direction of gravity when it is stationary.
  • the terminal can be used to identify the posture of the terminal (such as horizontal and vertical screen switching, related Games, magnetometer attitude calibration), vibration recognition related functions (such as pedometer, tap), etc .; as for the terminal, other sensors such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc. To repeat.
  • the audio circuit 606, a speaker, and a microphone can provide an audio interface between the user and the terminal.
  • the audio circuit 606 may transmit the received electrical data converted electrical signals to a speaker, and the speakers convert the sound signals into sound signals for output.
  • the microphone converts the collected sound signals into electrical signals, which are converted by the audio circuit 606 after being received.
  • the audio data is processed by the audio data output processor 608 and then sent to, for example, another terminal via the RF circuit 601, or the audio data is output to the memory 602 for further processing.
  • the audio circuit 606 may further include an earphone jack to provide communication between a peripheral headset and a terminal.
  • WiFi is a short-range wireless transmission technology.
  • the terminal can help users send and receive email, browse web pages, and access streaming media through the WiFi module 607. It provides users with wireless broadband Internet access.
  • FIG. 11 shows the WiFi module 607, it can be understood that it does not belong to the necessary configuration of the terminal, and can be omitted as needed without changing the essence of the invention.
  • the processor 608 is a control center of the terminal, and uses various interfaces and lines to connect various parts of the entire terminal. By running or executing software programs and / or modules stored in the memory 602, and calling data stored in the memory 602, execution is performed. Various functions and processing data of the terminal, so as to monitor the terminal as a whole.
  • the processor 608 may include one or more processing cores; preferably, the processor 608 may integrate an application processor and a modem processor, wherein the application processor mainly processes an operating system, a user interface, and an application program, etc.
  • the modem processor mainly handles wireless communication. It can be understood that the foregoing modem processor may not be integrated into the processor 608.
  • the terminal also includes a power source 609 (such as a battery) for supplying power to various components.
  • the power source can be logically connected to the processor 608 through a power management system, so as to implement functions such as managing charging, discharging, and power consumption management through the power management system.
  • the power source 609 may further include any one or more DC or AC power sources, a recharging system, a power failure detection circuit, a power converter or inverter, and a power source status indicator.
  • the terminal may further include a camera, a Bluetooth module, and the like, and details are not described herein again.
  • the processor 608 in the terminal loads the executable files corresponding to one or more application processes into the memory 602 according to the following instructions, and the processor 608 runs the stored files in the memory. 602 applications to achieve various functions:
  • the embodiment of the present application can obtain a two-dimensional image to be fused, and construct a three-dimensional face model based on the human face in the two-dimensional image to be fused, and obtain the template image and the human face in the template image in the three-dimensional space.
  • Rotation parameters and then rotate the 3D face model based on the rotation parameters to obtain a rotated 3D face model, and then convert the rotated 3D face model into a 2D face image.
  • the 2D face image and The template image is fused, for example, the face in the two-dimensional face image is fused to the area where the face in the template image is located.
  • This solution constructs and rotates a three-dimensional face model, so that a two-dimensional face image can be obtained based on the rotated three-dimensional face model, so that the two-dimensional face image is consistent with the direction of the face in the template image, so that the two-dimensional face image
  • the face in the image is accurately fused to the area where the face in the template image is located, which improves the effect of merging the face in the image.
  • All or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware by instructions.
  • the instructions may be stored in a computer-readable storage medium and loaded and executed by a processor. .
  • an embodiment of the present application provides a storage medium in which a plurality of instructions are stored, and the instructions can be loaded by a processor to execute the steps in any one of the image fusion methods provided in the embodiments of the present application.
  • the instruction can perform the following steps:
  • the storage medium may include a read-only memory (ROM, Read Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk, or an optical disk.
  • ROM read-only memory
  • RAM random access memory
  • magnetic disk or an optical disk.
  • the instructions stored in the storage medium can execute the steps in any one of the image fusion methods provided in the embodiments of the present application, it can implement what can be achieved by any of the image fusion methods provided in the embodiments of the present application.
  • the beneficial effects refer to the foregoing embodiments for details, and details are not described herein again.

Landscapes

  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Image Processing (AREA)

Abstract

Disclosed by the embodiments of the present application are an image fusion method, an apparatus, and a storage medium; in the embodiments of the present application, a two-dimensional image to be fused is obtained, and a three-dimensional human face model is constructed according to a human face in said two-dimensional image to be fused; a template image is obtained, and rotation parameters of the human face in said template image in a three-dimensional space are obtained; on the basis of said rotation parameters, the three-dimensional human face model is rotated to obtain a rotated three-dimensional human face model; the rotated human face three-dimensional model is converted into a two-dimensional human face image; the two-dimensional human face image is fused with the template image.

Description

一种图像融合方法、装置及存储介质Image fusion method, device and storage medium
本申请要求于2018年06月04日提交的申请号为201810563880.1、发明名称为“一种图像融合方法、装置及存储介质”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。This application claims priority from a Chinese patent application filed on June 04, 2018 with an application number of 201810563880.1 and an invention name of "A method, device and storage medium for image fusion", the entire contents of which are incorporated herein by reference .
技术领域Technical field
本申请涉及图像处理技术领域，具体涉及一种图像融合方法、装置及存储介质。The present application relates to the field of image processing technologies, and in particular, to an image fusion method, device, and storage medium.
背景技术Background technique
图像融合技术是计算机视觉中的一种关键技术，有着广泛的应用场景，例如，换脸、平均脸、疯狂变脸等深受用户喜爱，其关键技术就是人脸融合技术。Image fusion technology is a key technology in computer vision and has a wide range of application scenarios. For example, face change, average face, crazy face change, etc. are popular with users. The key technology is face fusion technology.
人脸融合是图像融合技术的一种应用，其主要是将包含用户人脸的人脸图像中的脸部区域，融合到包含人脸的模板图中，以替换模板图中的人脸，例如，将人脸图像中的脸部区域的像素点迁移到模板图中对应的脸部区域。Face fusion is an application of image fusion technology, which mainly fuses the face area in the face image containing the user's face into the template image containing the face to replace the face in the template image, for example To migrate the pixels of the face area in the face image to the corresponding face area in the template image.
发明内容Summary of the Invention
本申请实施例提供一种图像融合方法、装置及存储介质，旨在提高对图像中人脸进行融合的效果。Embodiments of the present application provide an image fusion method, device, and storage medium, which are intended to improve the effect of fusing faces in an image.
为解决上述技术问题，本申请实施例提供以下技术方案：To solve the above technical problems, the embodiments of the present application provide the following technical solutions:
一种图像融合方法，包括：An image fusion method includes:
获取待融合二维图像，并根据所述待融合二维图像中的人脸构建三维人脸模型；Acquiring a two-dimensional image to be fused, and constructing a three-dimensional face model according to the human face in the two-dimensional image to be fused;
获取模板图像，以及获取所述模板图像中的人脸在三维空间中的旋转参数；Acquiring a template image, and acquiring a rotation parameter of a face in the template image in a three-dimensional space;
基于所述旋转参数对所述三维人脸模型进行旋转，得到旋转后三维人脸模型；Rotating the three-dimensional face model based on the rotation parameters to obtain a rotated three-dimensional face model;
将所述旋转后人脸三维模型转换为二维人脸图像；Converting the three-dimensional model of the rotated face into a two-dimensional face image;
将所述二维人脸图像与所述模板图像进行融合。Fusion the two-dimensional face image and the template image.
一种图像融合装置，包括：An image fusion device includes:
构建单元，用于获取待融合二维图像，并根据所述待融合二维图像中的人脸构建三维人脸模型；A construction unit, configured to obtain a two-dimensional image to be fused, and construct a three-dimensional face model according to a face in the two-dimensional image to be fused;
获取单元，用于获取模板图像，以及获取所述模板图像中的人脸在三维空间中的旋转参数；An obtaining unit, configured to obtain a template image, and obtain a rotation parameter of a face in the template image in a three-dimensional space;
旋转单元，用于基于所述旋转参数对所述三维人脸模型进行旋转，得到旋转后三维人脸模型；A rotation unit, configured to rotate the three-dimensional face model based on the rotation parameters to obtain a rotated three-dimensional face model;
转换单元，用于将所述旋转后人脸三维模型转换为二维人脸图像；A converting unit, configured to convert the three-dimensional model of the rotated face into a two-dimensional face image;
融合单元，用于将所述二维人脸图像与所述模板图像进行融合。A fusion unit is configured to fuse the two-dimensional face image and the template image.
一种存储介质，所述存储介质存储有多条指令，所述指令适于处理器进行加载，以执行本申请实施所提供的任一种图像融合方法中的步骤。A storage medium stores a plurality of instructions, and the instructions are suitable for loading by a processor to execute steps in any image fusion method provided by the implementation of the present application.
附图简要说明Brief description of the drawings
为了更清楚地说明本申请实施例中的技术方案，下面将对实施例描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本申请的一些实施例，对于本领域技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。In order to explain the technical solutions in the embodiments of the present application more clearly, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are just some embodiments of the application. For those skilled in the art, other drawings can be obtained based on these drawings without paying creative labor.
图1是本申请实施例提供的图像融合方法的场景示意图；FIG. 1 is a schematic diagram of an image fusion method according to an embodiment of the present application;
图2是本申请实施例提供的图像融合方法的流程示意图；2 is a schematic flowchart of an image fusion method according to an embodiment of the present application;
图3(a)是本申请实施例提供的待融合二维图像的示意图；3 (a) is a schematic diagram of a two-dimensional image to be fused provided by an embodiment of the present application;
图3(b)是本申请实施例提供的模板图像的示意图；3 (b) is a schematic diagram of a template image provided by an embodiment of the present application;
图4(a)是本申请实施例提供的三维人脸模型的示意图；4 (a) is a schematic diagram of a three-dimensional face model provided by an embodiment of the present application;
图4(b)是本申请实施例提供的旋转后三维人脸模型的示意图；4 (b) is a schematic diagram of a rotated three-dimensional human face model provided by an embodiment of the present application;
图5是本申请实施例提供的二维人脸图像的示意图；5 is a schematic diagram of a two-dimensional face image provided by an embodiment of the present application;
图6(a)是本申请实施例提供的人脸特征点检测得到人脸特征点的示意图；FIG. 6 (a) is a schematic diagram of obtaining a facial feature point by detecting a facial feature point according to an embodiment of the present application; FIG.
图6(b)是本申请实施例提供的三角形划分得到人脸三角形区域的示意图；FIG. 6 (b) is a schematic diagram of obtaining a triangle region of a human face by dividing a triangle according to an embodiment of the present application; FIG.
图7是本申请实施例提供的融合后图像的示意图；7 is a schematic diagram of a fused image provided by an embodiment of the present application;
图8是本申请实施例提供的图像融合方法的另一流程示意图；FIG. 8 is another schematic flowchart of an image fusion method according to an embodiment of the present application; FIG.
图9是本申请实施例提供的图像融合装置的结构示意图；9 is a schematic structural diagram of an image fusion apparatus according to an embodiment of the present application;
图10是本申请实施例提供的图像融合装置的另一结构示意图；FIG. 10 is another schematic structural diagram of an image fusion apparatus according to an embodiment of the present application; FIG.
图11是本申请实施例提供的终端的结构示意图。FIG. 11 is a schematic structural diagram of a terminal according to an embodiment of the present application.
实施本发明的方式Mode of Carrying Out the Invention
下面将结合本申请实施例中的附图，对本申请实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例仅仅是本申请一部分实施例，而不是全部的实施例。基于本申请中的实施例，本领域技术人员在没有作出创造性劳动前提下所获得的所有其他实施例，都属于本申请保护的范围。The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall into the protection scope of the present application.
在对图像融合技术的研究过程中，本申请的发明人发现，一些人脸融合方案在模板图和人脸图像中的人脸方向一致(例如都是正脸或都是侧脸)的情况下才能够得到较好的融合效果，而通常，用户拍照的人脸图像中人脸方向很难达到与模板图中的人脸方向一致。例如，如果让用户拍一张包含侧脸的人脸图像，试图保持与模板图中一致的侧脸姿势，则在拍照的过程中，用户侧脸的姿势和角度很难达到与模板图中的侧脸一致。而当模板图和人脸图像中的人脸方向不一致时，模板图与人脸图像中的人脸特征点会有较大偏差，只能强行进行融合。During the research on image fusion technology, the inventors of the present application found that some face fusion schemes are only used when the orientation of the face in the template image and the face image are the same (for example, all face faces or all side faces). A good fusion effect can be obtained, and in general, the face direction of a face image taken by a user is difficult to be consistent with the face direction in the template image. For example, if you ask the user to take a face image that includes a side face, and try to maintain the same side face pose as in the template image, it is difficult for the user's side face pose and angle to reach the Side faces are consistent. When the orientation of the face in the template image and the face image are inconsistent, the feature points of the face in the template image and the face image will deviate greatly, and the fusion can only be forced.
例如，当模板图中的人脸是侧脸，且人脸图像中的人脸是正脸时，由于模板图中的侧脸有部分区域是不可见的，比如被挡住一侧鼻子，而人脸图像中的所有正脸区域都会融合到模板图中，此时，模板图中的侧脸与人脸图像中的正脸之间的人脸特征点会有偏差，强行进行融合会造成扭曲和偏移等。For example, when the face in the template image is a side face, and the face in the face image is a positive face, because some areas of the side face in the template image are not visible, such as blocking one side of the nose, and the face All the positive face areas in the image will be fused into the template image. At this time, the facial feature points between the side face in the template image and the positive face in the face image will be biased. Forcibly performing fusion will cause distortion and bias. Shift etc.
由于当需要模板图和人脸图像中的人脸方向一致时，用户提供的人脸图像中人脸方向(例如姿势和角度)很难达到与模板图保持一致，因此得到的人脸融合效果比较差。而当模板图和人脸图像中的人脸方向不一致(例如模板图中的人脸是侧脸，且人脸图像中的人脸是正脸)时，强行进行融合会造成扭曲和偏移等误差，导致人脸融合效果非常差。Because when the template image and the face direction in the face image need to be consistent, the face direction (such as posture and angle) in the face image provided by the user is difficult to be consistent with the template image, so the obtained face fusion effect is compared. difference. When the orientation of the face in the template image and the face image are inconsistent (for example, the face in the template image is a side face and the face in the face image is a positive face), forcing the fusion will cause errors such as distortion and offset. , Resulting in very poor face fusion.
因此，本申请实施例提供一种图像融合方法、装置及存储介质，可以提高对图像中人脸进行融合的效果。Therefore, the embodiments of the present application provide an image fusion method, device, and storage medium, which can improve the effect of fusing faces in an image.
请参阅图1，图1为本申请实施例所提供的图像融合方法的场景示意图。其中，图像融合装置具体可以集成在平板电脑、手机、笔记本电脑、及台式电脑等具备储存单元并安装有微处理器而具有运算能力的计算设备中，例如终端。该终端可以用于接收用户输入的图像融合指令，基于图像融合指令获取待融合 二维图像，该待融合二维图像包含用户的人脸，例如，可以向服务器发送图像获取请求，并接收服务器基于返回的待融合二维图像；或者通过摄像头采集用户的图像，得到待融合二维图像等。此时，可以根据待融合二维图像中的人脸构建三维人脸模型，例如，可以向服务器发送模型获取请求，并接收服务器基于模型获取请求返回的预设人脸模型，根据预设人脸模型获取待融合二维图像中的人脸在三维平面内的三维坐标点，并根据三维坐标点构建三维人脸模型等，在得到三维人脸模型后，可以将三维人脸模型上传至服务器进行存储。以及，可以获取模板图像，该模板图像中包含人脸，例如可以从服务器上获取模板图像，此时，可以获取模板图像中的人脸在三维空间中的旋转参数。然后，基于旋转参数对三维人脸模型进行旋转，得到旋转后三维人脸模型，并将旋转后人脸三维模型转换为二维人脸图像。最后，可以将二维人脸图像与模板图像进行融合，例如，可以分别对二维人脸图像和模板图像进行人脸特征点检测，得到各自对应的人脸特征点，并根据人脸特征点将二维人脸图像中的人脸融合至模板图像中的人脸所在区域，得到融合后图像；等等。Please refer to FIG. 1, which is a schematic diagram of a scene of an image fusion method provided by an embodiment of the present application. The image fusion device may be specifically integrated in a computing device such as a tablet, a mobile phone, a notebook computer, and a desktop computer that has a storage unit and a microprocessor and has a computing capability, such as a terminal. The terminal may be used to receive an image fusion instruction input by a user, and obtain a two-dimensional image to be fused based on the image fusion instruction, where the two-dimensional image to be fused contains a user's face. For example, the terminal may send an image acquisition request to the server, and receive the server based on The returned two-dimensional image to be fused; or the user's image is collected through a camera to obtain the two-dimensional image to be fused, and so on. At this time, a three-dimensional face model may be constructed based on the faces in the two-dimensional image to be fused. For example, a model acquisition request may be sent to the server, and a preset face model returned by the server based on the model acquisition request may be received according to the preset face. The model obtains the 3D coordinate points of the face in the 2D image to be fused in the 3D plane, and constructs a 3D face model based on the 3D coordinate points. After the 3D face model is obtained, the 3D face model can be uploaded to the server for processing. storage. And, a template image may be obtained, and the template image includes a human face. For example, a template image may be obtained from a server. At this time, a rotation parameter of the human face in the template image in a three-dimensional space may be obtained. Then, the three-dimensional face model is rotated based on the rotation parameters to obtain a rotated three-dimensional face model, and the rotated three-dimensional face model is converted into a two-dimensional face image. Finally, the two-dimensional face image can be fused with the template image. For example, the two-dimensional face image and the template image can be detected separately for the feature points of the face, and the corresponding corresponding feature points of the face can be obtained. Fusion the face in the two-dimensional face image to the area of the face in the template image to obtain the fused image; and so on.
需要说明的是，图1所示的图像融合方法的场景示意图仅仅是一个示例，本申请实施例描述的图像融合方法的场景是为了更加清楚地说明本申请实施例的技术方案，并不构成对于本申请实施例提供的技术方案的限定。随着图像融合方法的演变和新业务场景的出现，本申请实施例提供的技术方案对于类似的技术问题，同样适用。It should be noted that the scene schematic diagram of the image fusion method shown in FIG. 1 is only an example, and the scenario of the image fusion method described in the embodiment of the present application is to more clearly illustrate the technical solution of the embodiment of the present application, and does not constitute Limitations of the technical solutions provided in the embodiments of the present application. With the evolution of image fusion methods and the emergence of new business scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.
以下分别进行详细说明。Each of them will be described in detail below.
在本申请实施例中，将从图像融合装置的角度进行描述，该图像融合装置具体可以集成在平板电脑、手机、笔记本电脑、及台式电脑等具备储存单元并安装有微处理器而具有运算能力的终端中。In the embodiment of the present application, description will be made from the perspective of an image fusion device. The image fusion device can be specifically integrated into a tablet computer, a mobile phone, a notebook computer, and a desktop computer, which have a storage unit and a microprocessor and have computing capabilities. In the terminal.
一种图像融合方法，包括：获取待融合二维图像，并根据待融合二维图像中的人脸构建三维人脸模型；获取模板图像，以及获取模板图像中的人脸在三维空间中的旋转参数；基于旋转参数对三维人脸模型进行旋转，得到旋转后三维人脸模型；将旋转后人脸三维模型转换为二维人脸图像；将二维人脸图像与模板图像进行融合。An image fusion method includes: obtaining a two-dimensional image to be fused, and constructing a three-dimensional face model based on a face in the two-dimensional image to be fused; obtaining a template image, and obtaining rotation of a human face in the template image in a three-dimensional space Parameters; rotating the 3D face model based on the rotation parameters to obtain a rotated 3D face model; converting the rotated 3D face model into a 2D face image; and fusing the 2D face image with a template image.
请参阅图2，图2是本申请一实施例提供的图像融合方法的流程示意图。该图像融合方法可以包括：Please refer to FIG. 2, which is a schematic flowchart of an image fusion method according to an embodiment of the present application. The image fusion method may include:
在步骤S101中，获取待融合二维图像，并根据待融合二维图像中的人脸构建三维人脸模型。In step S101, a two-dimensional image to be fused is acquired, and a three-dimensional human face model is constructed according to a human face in the two-dimensional image to be fused.
该待融合二维图像中可以包含一张或多张人脸，还可以包括其他的物体，其中，该人脸可以是正面的人脸(即正脸)，或者任意角度的人脸(即侧脸)等。例如，如图3(a)所示，该待融合二维图像中可以包括一张正脸。The two-dimensional image to be fused may include one or more faces, and may also include other objects, wherein the face may be a frontal face (that is, a front face), or an arbitrary angled face (that is, a side face) Face) and so on. For example, as shown in FIG. 3 (a), the two-dimensional image to be fused may include a front face.
待融合二维图像可以是图像融合装置在拍照的过程中，拍照摄像头采集到的图像；待融合二维图像也可以是图像融合装置预存的图像，待融合二维图像还可以是通过在互联网上搜索或者从图像数据库中获取到的图像等；当然，待融合二维图像也可以是通过其他方式获取到的图像，具体获取方式在此处不作限定。The two-dimensional image to be fused can be an image collected by a camera during the image fusion device's photographing process; the two-dimensional image to be fused can also be a pre-stored image of the image fusion device, and the two-dimensional image to be fused can also be downloaded on the Internet. Search or image obtained from the image database; of course, the two-dimensional image to be fused may also be an image obtained by other methods, and the specific acquisition method is not limited here.
在得到待融合二维图像后，图像融合装置可以根据待融合二维图像中的人脸构建三维人脸模型。在某些实施方式中，根据待融合二维图像中的人脸构建三维人脸模型的步骤可以包括：After obtaining the two-dimensional image to be fused, the image fusion device may construct a three-dimensional face model according to the human face in the two-dimensional image to be fused. In some embodiments, the step of constructing a three-dimensional face model based on a face in a two-dimensional image to be fused may include:
获取预设人脸模型；根据预设人脸模型，获取待融合二维图像中的人脸在三维平面内的三维坐标点，并根据三维坐标点构建三维人脸模型。Obtain a preset face model; according to the preset face model, obtain three-dimensional coordinate points of a face in a two-dimensional image to be fused in a three-dimensional plane, and construct a three-dimensional face model according to the three-dimensional coordinate points.
其中，预设人脸模型可以是三维平均人脸模型，该三维平均人脸模型可以是预先设置。可以通过改变三维平均人脸模型的三维坐标点的位置信息来生成不同的表情及不同身份的用户的三维人脸模型，也就是说所有三维人脸模型都可以通过在三维平均人脸模型的三维坐标点上增加偏移来表示。The preset face model may be a three-dimensional average face model, and the three-dimensional average face model may be preset. You can generate 3D face models of users with different expressions and different identities by changing the position information of the 3D coordinate points of the 3D average face model. That is to say, all 3D face models can An offset is added to the coordinate point to indicate.
图像融合装置可以接收用户输入的设置指令，并基于设置指令设置预设人脸模型；或者，图像融合装置可以向服务器发送模型获取请求，并接收服务器基于模型获取请求返回的预设人脸模型。在得到预设人脸模型后，图像融合装置可以根据预设人脸模型，获取待融合二维图像中的人脸在三维平面内的三维坐标点，并根据三维坐标点构建三维人脸模型。The image fusion device may receive a setting instruction input by the user and set a preset face model based on the setting instruction; or, the image fusion device may send a model acquisition request to the server and receive the preset face model returned by the server based on the model acquisition request. After obtaining the preset face model, the image fusion device can obtain the three-dimensional coordinate points of the face in the two-dimensional image to be fused in the three-dimensional plane according to the preset face model, and construct a three-dimensional face model according to the three-dimensional coordinate points.
在某些实施方式中，根据预设人脸模型，获取待融合二维图像中的人脸在三维平面内的三维坐标点，并根据三维坐标点构建三维人脸模型的步骤可以包括：In some embodiments, the step of obtaining a three-dimensional coordinate point of a face in a two-dimensional image to be fused in a three-dimensional plane according to a preset face model, and constructing a three-dimensional face model according to the three-dimensional coordinate point may include:
(1)获取待融合二维图像中的人脸投影在二维平面内的第一投影参数；(1) Acquire a first projection parameter of a face projected in a two-dimensional plane to be fused in a two-dimensional plane;
(2)获取预设人脸模型投影在二维平面内的第二投影参数；(2) acquiring a second projection parameter of the preset face model projected in a two-dimensional plane;
(3)根据第一投影参数及第二投影参数，获取待融合二维图像中的人脸在三维平面内的三维坐标点；(3) obtaining, according to the first projection parameter and the second projection parameter, a three-dimensional coordinate point of a face in a two-dimensional image to be fused in a three-dimensional plane;
(4)根据三维坐标点构建三维人脸模型。(4) Construct a three-dimensional face model based on three-dimensional coordinate points.
具体地，图像融合装置根据待融合二维图像中的人脸构建三维人脸模型的过程中，可以从待融合二维图像中提取出人脸，并获取待融合二维图像中的人脸投影在二维平面内的第一投影参数，该第一投影参数可以是人脸在二维平面内的坐标点。Specifically, in the process of the image fusion device constructing a three-dimensional face model based on the faces in the two-dimensional image to be fused, the face can be extracted from the two-dimensional image to be fused, and the face projection in the two-dimensional image to be fused is obtained. A first projection parameter in a two-dimensional plane. The first projection parameter may be a coordinate point of a human face in a two-dimensional plane.
以及，图像融合装置获取预设人脸模型投影在二维平面内的第二投影参数，该第二投影参数可以是预设人脸模型在二维平面内的坐标点。由于可以通过改变三维平均人脸模型的三维坐标点的位置信息来生成不同的表情及不同身份的用户的三维人脸模型，也就是说所有三维人脸模型都可以通过在三维平均人脸模型的三维坐标点上增加偏移来表示，因此，用户的人脸的三维人脸模型可以用如下公式(1)表示：And, the image fusion device obtains a second projection parameter of the preset face model projected in the two-dimensional plane, and the second projection parameter may be a coordinate point of the preset face model in the two-dimensional plane. Because the position information of the three-dimensional average point model of the three-dimensional average face model can be changed to generate three-dimensional face models of users with different expressions and different identities, that is, all three-dimensional face models can be obtained by The offset is added to the three-dimensional coordinate points to represent it. Therefore, the three-dimensional face model of the user's face can be expressed by the following formula (1):
Figure PCTCN2019085841-appb-000001
Figure PCTCN2019085841-appb-000001
其中，公式(1)中的M表示人脸的三维人脸模型，
Figure PCTCN2019085841-appb-000002
表示预设人脸模型(即三维平均人脸模型)，
Figure PCTCN2019085841-appb-000003
N为三维平均人脸模型包含的三维坐标点的点数；A idP id表示身份偏移项，
Figure PCTCN2019085841-appb-000004
A id为m id维身份基，P id为身份参数；A expP exp表示表情偏移项，
Figure PCTCN2019085841-appb-000005
A exp为m exp维表情基，P exp为表情参数。 Among them, M in the formula (1) represents a three-dimensional face model of a human face,
Figure PCTCN2019085841-appb-000002
Represents a preset face model (ie, a three-dimensional average face model),
Figure PCTCN2019085841-appb-000003
N is the number of points of the three-dimensional coordinate point included in the three-dimensional average face model; A id P id represents an identity offset term,
Figure PCTCN2019085841-appb-000004
A id is the identity base of m id , P id is the identity parameter; A exp P exp represents the expression offset term,
Figure PCTCN2019085841-appb-000005
A exp is the expression base of m exp dimension, P exp is the expression parameter.
在得到人脸投影在二维平面内的第一投影参数，以及预设人脸模型投影在二维平面内的第二投影参数后，图像融合装置可以根据第一投影参数及第二投影参数，获取人脸在三维平面内的三维坐标点，例如，如下公式(2)和公式(3)所示：After obtaining the first projection parameter of the human face projected in the two-dimensional plane and the preset second projection parameter of the human face model projected in the two-dimensional plane, the image fusion device may according to the first projection parameter and the second projection parameter, Obtain the three-dimensional coordinate points of the human face in the three-dimensional plane, for example, as shown in the following formula (2) and formula (3):
arg min||X 3d-X 2d||    (2) arg min || X 3d -X 2d || (2)
X 3d＝p(M keypoint)    (3) X 3d = p (M keypoint ) (3)
其中，X 3d表示预设人脸模型的三维坐标点，M keypoint表示预设人脸模型在二维平面内的第二投影参数，p为投影函数，X 2d为人脸投影在二维平面内的第一投影参数。 Among them, X 3d represents the three-dimensional coordinate point of the preset face model, and M keypoint represents the second projection parameter of the preset face model in the two-dimensional plane, p is a projection function, and X 2d is a projection of the face in the two-dimensional plane. First projection parameter.
其中，投影可以包括正交投影和透视投影等，以正交投影进行三维建模为例，可以通过如下公式(4)进行正交投影：The projection may include orthogonal projection, perspective projection, and the like. Taking orthogonal projection for 3D modeling as an example, orthogonal projection can be performed by the following formula (4):
X 3d＝S×R×M keypoint+T    (4) X 3d = S × R × M keypoint + T (4)
其中，S为放缩系数，R为旋转系数，T为二维平移分量，因此，可以通过公式(1)、公式(3)及公式(4)迭代求解公式(2)，即可得到身份参数P id 和表情参数P exp、放缩系数S、旋转系数R及二维平移分量T，根据参数[P id，P exp，S，R，T]得到三维坐标点，根据三维坐标点即可生成三维人脸模型。也就是说，如果将各个角度的三维人脸模型投影到二维平面都可以与待融合二维图像中人脸相匹配，那么这个三维人脸模型就是需要获取的三维人脸模型。例如，根据待融合二维图像中的人脸构建得到的三维人脸模型如图4(a)所示。 Among them, S is a scaling factor, R is a rotation coefficient, and T is a two-dimensional translation component. Therefore, formula (2) can be solved iteratively by formula (1), formula (3), and formula (4), and the identity parameter can be obtained P id and expression parameter P exp , scaling factor S, rotation coefficient R, and two-dimensional translation component T. According to the parameters [P id , P exp , S, R, T], a three-dimensional coordinate point is obtained, and the three-dimensional coordinate point can be generated. Three-dimensional face model. That is, if the three-dimensional face model of each angle is projected onto a two-dimensional plane and can be matched with the face in the two-dimensional image to be fused, then this three-dimensional face model is a three-dimensional face model that needs to be obtained. For example, a three-dimensional face model constructed based on a face in a two-dimensional image to be fused is shown in FIG. 4 (a).
需要说明的是，除了可以通过上述公式构建三维人脸模型之外，三维人脸模型还可以是通过其他方式获取，具体构建方式在此不做限定。It should be noted that in addition to using the above formula to construct a three-dimensional face model, the three-dimensional face model may also be obtained through other methods, and the specific construction method is not limited here.
需要说明的是，当待融合二维图像中的人脸包括多个时，图像融合装置可以从待融合二维图像中任意提取一张人脸来构建三维人脸模型，或者，图像融合装置可以输出选择项，并接收用户基于选择项从待融合二维图像中选择其中一张人脸来构建三维人脸模型，或者，图像融合装置可以从待融合二维图像中提取出所有人脸，并根据各个人脸分别构建三维人脸模型；等等。It should be noted that when there are multiple faces in the two-dimensional image to be fused, the image fusion device may arbitrarily extract a face from the two-dimensional image to be fused to construct a three-dimensional face model, or the image fusion device may Output a selection item and receive a user to select a face from the two-dimensional image to be fused based on the selection item to construct a three-dimensional face model, or the image fusion device may extract all human faces from the two-dimensional image to be fused, and Construct a three-dimensional face model based on each face; etc.
在步骤S102中，获取模板图像，以及获取模板图像中的人脸在三维空间中的旋转参数。In step S102, a template image is obtained, and a rotation parameter of a human face in the template image in a three-dimensional space is obtained.
其中，步骤S102可以在步骤S101之前执行，也可以在步骤S101之后执行，具体执行顺序可以根据实际需要进行灵活设置，在此处不作限定。Among them, step S102 may be performed before step S101, or may be performed after step S101. The specific execution order may be flexibly set according to actual needs, which is not limited here.
该模板图像中可以包含一张或多张人脸，还可以包括其他的物体，其中，该人脸可以是正面的人脸(即正脸)，或者任意角度的人脸(即侧脸)等。例如，如图3(b)所示，该模板图像中可以包含一张侧脸。模板图像可以是图像融合装置通过拍照摄像头采集到的图像，也可以是图像融合装置预先设置的图像，还可以是通过在互联网上搜索或者从图像数据库中获取到的图像等；当然，模板图像也可以是通过其他方式获取到，具体获取方式在此处不作限定。The template image may include one or more human faces, and may also include other objects, where the human face may be a front human face (that is, a front face), or a human face at any angle (that is, a side face). . For example, as shown in FIG. 3 (b), the template image may include a side face. The template image can be an image collected by the image fusion device through a camera, or an image preset by the image fusion device, or an image obtained by searching on the Internet or obtained from an image database; of course, the template image is also It can be obtained through other methods, and the specific acquisition method is not limited here.
在得到模板图像后，图像融合装置可以获取模板图像中的人脸在三维空间中的旋转参数，该旋转参数可以是人脸在三维空间中的X轴、Y轴、及Z轴方向上的转向。例如，当模板图像中的人脸为正脸时，可以设置模板图像中的正脸在三维空间中的旋转参数为0；当模板图像中的人脸为侧脸时，可以设置模板图像中的侧脸相对于正脸的旋转参数为：在X轴方向上旋转的x角度、在Y轴方向上旋转的y角度、及在Z轴方向上旋转的z角度。After the template image is obtained, the image fusion device can obtain the rotation parameters of the face in the template image in the three-dimensional space, and the rotation parameter can be the turning of the face in the X-axis, Y-axis, and Z-axis directions in the three-dimensional space. . For example, when the human face in the template image is a positive face, the rotation parameter of the normal face in the template image in three-dimensional space can be set to 0; when the human face in the template image is a side face, the The rotation parameters of the side face with respect to the front face are: an x angle rotated in the X axis direction, a y angle rotated in the Y axis direction, and a z angle rotated in the Z axis direction.
图像融合装置可以接收用户输入的旋转参数，并将该旋转参数设置为模板图像中的人脸在三维空间中的旋转参数，即该旋转参数可以由人工设置。The image fusion device may receive a rotation parameter input by a user and set the rotation parameter as a rotation parameter of a face in a template image in a three-dimensional space, that is, the rotation parameter may be manually set.
在某些实施方式中，获取模板图像中的人脸在三维空间中的旋转参数的步骤可以包括：根据模板图像中的人脸构建三维模型；基于三维模型获取模板图像中人脸在三维空间中的旋转参数。In some embodiments, the step of obtaining the rotation parameters of the human face in the template image in the three-dimensional space may include: constructing a three-dimensional model based on the human face in the template image; and acquiring the human face in the template image in the three-dimensional space based on the three-dimensional model. Rotation parameters.
在某些实施方式中，根据模板图像中的人脸构建三维模型的步骤可以包括：In some embodiments, the step of constructing a three-dimensional model based on a face in a template image may include:
获取模板图像中的人脸投影在二维平面内的第三投影参数；获取预设人脸模型，及预设人脸模型投影在二维平面内的第四投影参数；根据第三投影参数及第四投影参数，获取模板图像中的人脸在三维平面内的目标三维坐标点；根据目标三维坐标点构建三维模型。Obtain a third projection parameter of the face projection in the template image in a two-dimensional plane; obtain a preset face model, and a fourth projection parameter of the preset face model projected in the two-dimensional plane; according to the third projection parameter and The fourth projection parameter obtains the target three-dimensional coordinate points of the human face in the template image in the three-dimensional plane; and constructs a three-dimensional model according to the target three-dimensional coordinate points.
具体地，图像融合装置可以从模板图像中提取出人脸，并获取模板图像中的人脸投影在二维平面内的第三投影参数，该第三投影参数可以是人脸在二维平面内的坐标点。以及，图像融合装置可以获取预设人脸模型(该预设人脸模型与上述提及的预设人脸模型一致，即为三维平均人脸模型)投影在二维平面内的第四投影参数，该第四投影参数可以是预设人脸模型在二维平面内的坐标点。然后可以利用上述公式(1)、公式(2)、公式(3)和公式(4)根据第三投影参数及第四投影参数，计算模板图像中的人脸在三维平面内的目标三维坐标点，并根据目标三维坐标点构建三维模型。在得到三维模型后，可以基于三维模型获取模板图像中人脸在三维空间中的旋转参数，该旋转参数即为上述公式(4)中的旋转系数R。Specifically, the image fusion device may extract a human face from the template image, and obtain a third projection parameter of the human face in the template image projected in the two-dimensional plane, and the third projection parameter may be that the human face is in the two-dimensional plane. Coordinate point. And, the image fusion device can obtain a fourth projection parameter of a preset face model (the preset face model is consistent with the above-mentioned preset face model, that is, a three-dimensional average face model) in a two-dimensional plane. The fourth projection parameter may be a coordinate point of a preset face model in a two-dimensional plane. Then, the above formulas (1), (2), (3), and (4) can be used to calculate the target three-dimensional coordinate points of the human face in the template image in the three-dimensional plane according to the third projection parameter and the fourth projection parameter. , And build a 3D model based on the target 3D coordinate points. After the three-dimensional model is obtained, the rotation parameter of the face in the three-dimensional space in the template image can be obtained based on the three-dimensional model, and the rotation parameter is the rotation coefficient R in the above formula (4).
在步骤S103中，基于旋转参数对三维人脸模型进行旋转，得到旋转后三维人脸模型。In step S103, the three-dimensional face model is rotated based on the rotation parameters to obtain a rotated three-dimensional face model.
在得到三维人脸模型和旋转参数后，图像融合装置可以将旋转参数应用到三维人脸模型上，控制三维人脸模型在三维空间中进行X轴、Y轴、及Z轴等方向上的旋转，使得三维人脸模型中人脸的方向(包括姿势和角度等)与模板图中人脸方向一致。例如，基于旋转参数对三维人脸模型进行旋转，得到旋转后三维人脸模型如图4(b)所示。After obtaining the 3D face model and rotation parameters, the image fusion device can apply the rotation parameters to the 3D face model, and control the 3D face model to rotate in the X, Y, and Z directions in the 3D space. , So that the direction of the face (including pose and angle, etc.) in the three-dimensional face model is consistent with the direction of the face in the template image. For example, the 3D face model is rotated based on the rotation parameters, and the rotated 3D face model is obtained as shown in FIG. 4 (b).
例如，当三维人脸模型中的人脸为正脸，且模板图像中的人脸为侧脸时，为了使得三维人脸模型中人脸的方向与模板图中人脸方向一致，可以基于模板图中侧脸在三维空间中相对于正脸的旋转参数，对三维人脸模型中的正脸进行旋转，得到旋转后三维人脸模型中的人脸为侧脸。For example, when the face in the 3D face model is a positive face and the face in the template image is a side face, in order to make the direction of the face in the 3D face model consistent with the face direction in the template image, the template may be based on the template. In the figure, the rotation parameters of the side face relative to the front face in the three-dimensional space are used to rotate the front face in the three-dimensional face model, and the face in the three-dimensional face model after rotation is obtained as the side face.
又例如，当三维人脸模型中的人脸为侧脸，且模板图像中的人脸为正脸时，为了使得三维人脸模型中人脸的方向与模板图中人脸方向一致，可以基于模板图中正脸在三维空间中相对于侧脸的旋转参数，对三维人脸模型中的侧脸进行旋转，得到旋转后三维人脸模型中的人脸为正脸。For another example, when the face in the three-dimensional face model is a side face and the face in the template image is a positive face, in order to make the direction of the face in the three-dimensional face model consistent with the face direction in the template image, it can be based on The rotation parameters of the positive face in the template image relative to the side face in the three-dimensional space are used to rotate the side face in the three-dimensional face model to obtain that the face in the three-dimensional face model after rotation is the positive face.
需要说明的是，在得到旋转后三维人脸模型后，图像融合装置可以对该旋转后三维人脸模型进行预处理，例如通过亮度调节方式，使得旋转后三维人脸模型中人脸上的光照环境保持与模板图相当等。It should be noted that after obtaining the rotated 3D face model, the image fusion device can pre-process the rotated 3D face model, for example, by adjusting the brightness, so that the light on the face in the rotated 3D face model is illuminated. The environment remains comparable to the template map.
在步骤S104中，将旋转后人脸三维模型转换为二维人脸图像。In step S104, the three-dimensional model of the face after the rotation is converted into a two-dimensional face image.
在得到旋转后人脸三维模型后，图像融合装置可以将旋转后人脸三维模型转换为二维人脸图像，该二维人脸图像可以如图5所示，例如，可以接收截图指令，根据截图指令直接截取旋转后人脸三维模型在显示界面内显示的图像，得到二维人脸图像；或者，按照上述根据人脸构建三维人脸模型的方法进行逆映射，将旋转后人脸三维模型逆映射为二维人脸图像，当然，还可以通过其他方式将旋转后人脸三维模型转换为二维人脸图像。After obtaining the three-dimensional model of the rotated face, the image fusion device may convert the three-dimensional model of the rotated face into a two-dimensional face image. The two-dimensional face image may be as shown in FIG. 5. For example, a screenshot instruction may be received. The screenshot instruction directly captures the image displayed on the display interface of the rotated three-dimensional model of the face to obtain a two-dimensional face image; or, according to the method of constructing a three-dimensional face model based on the face described above, the inverse mapping is performed to convert the rotated three-dimensional face model The inverse mapping is a two-dimensional face image. Of course, there are other ways to convert the three-dimensional model of the rotated face into a two-dimensional face image.
在步骤S105中，将二维人脸图像与模板图像进行融合。In step S105, the two-dimensional face image and the template image are fused.
图像融合装置可以先做人脸特征点检测，然后根据人脸特征点划分人脸三角形区域，然后根据人脸三角形区域做仿射变换，最后将二维人脸图像中人脸上的每一块人脸三角形区域融合到模板图像中人脸的对应位置上，还可以做一些边缘柔等后处理操作，即可完成人脸图像融合。The image fusion device can first perform face feature point detection, then divide the face triangle area according to the face feature points, then perform affine transformation according to the face triangle area, and finally each person in the face in the two-dimensional face image The triangular area of the face is fused to the corresponding position of the face in the template image, and some post-processing operations such as edge softening can also be performed to complete the face image fusion.
需要说明的是，当二维人脸图像和模板图像中的人脸均为多个时，图像融合装置可以根据待融合二维图像中的各个人脸分别构建三维人脸模型，以及获取模板图像中的各个人脸在三维空间中的旋转参数，基于各个旋转参数分别对各个三维人脸模型进行旋转，得到旋转后三维人脸模型；然后将各个旋转后人脸三维模型转换为二维人脸图像，并将二维人脸图像中各个人脸与模板图像中各个人脸进行融合。例如，当二维人脸图像中包括第一人脸和第二人脸，模板图像中包括第三人脸和第四人脸时，可以根据第一人脸和第二人脸分别构建三维人脸模型，以及第三人脸和第四人脸在三维空间中的旋转参数，基于第三人脸对应的旋转参数对第一人脸对应的三维人脸模型进行旋转，基于第四人脸对应的旋转参数对第二人脸对应的三维人脸模型进行旋转，得到旋转后三维人脸模型；然后将各个旋转后人脸三维模型转换为二维人脸图像，最后将二维人脸 图像中的第一人脸与模板图像中的第三人脸进行融合，将二维人脸图像中的第二人脸与模板图像中的第四人脸进行融合。It should be noted that when there are multiple faces in the two-dimensional face image and the template image, the image fusion device may separately construct a three-dimensional face model according to each face in the two-dimensional image to be fused, and obtain a template image. The rotation parameters of each face in the three-dimensional space are based on each rotation parameter to rotate each three-dimensional face model to obtain a rotated three-dimensional face model; and then convert each rotated three-dimensional model into a two-dimensional face Image, and fuse each face in the two-dimensional face image with each face in the template image. For example, when a two-dimensional face image includes a first face and a second face, and a template image includes a third face and a fourth face, a three-dimensional person may be constructed based on the first face and the second face, respectively. The face model, and the rotation parameters of the third face and the fourth face in the three-dimensional space. Based on the rotation parameters corresponding to the third face, the three-dimensional face model corresponding to the first face is rotated, and the fourth face correspondence is based on Rotate the 3D face model corresponding to the second face to obtain a rotated 3D face model; then convert each rotated 3D model into a 2D face image, and finally convert the 2D face image into The first face in the image is fused with the third face in the template image, and the second face in the two-dimensional face image is fused with the fourth face in the template image.
在某些实施方式中，将二维人脸图像与模板图像进行融合的步骤可以包括：In some embodiments, the step of fusing the two-dimensional face image with the template image may include:
(1)对二维人脸图像进行人脸特征点检测，得到第一人脸特征点；(1) Perform face feature point detection on a two-dimensional face image to obtain a first face feature point;
(2)对模板图像进行人脸特征点检测，得到第二人脸特征点；(2) Perform face feature point detection on the template image to obtain a second face feature point;
(3)根据第一人脸特征点和第二人脸特征点，将二维人脸图像中的人脸融合至模板图像中的人脸所在区域。(3) According to the first face feature points and the second face feature points, the face in the two-dimensional face image is fused to the area where the face in the template image is located.
图像融合装置可以通过人脸识别技术，对二维人脸图像中人脸上的眼睛、鼻子、眉毛及嘴巴等各个人脸器官进行特征点检测，生成第一人脸特征点，该第一人脸特征点可以是各个人脸器官对应的关键点的位置坐标信息，该第一人脸特征点可以在人脸的外部轮廓和各个人脸器官的边缘或中心等，该第一人脸特征点的个数可以是106个、83个或68个等，具体个数可以根据实际需要进行灵活设置。The image fusion device can use face recognition technology to detect feature points on each face organ such as eyes, nose, eyebrows, and mouth in a two-dimensional face image to generate a first face feature point, the first person The face feature point may be position coordinate information of a key point corresponding to each face organ. The first face feature point may be an outer contour of the face and an edge or a center of each face organ. The number can be 106, 83, or 68. The specific number can be flexibly set according to actual needs.
以及，图像融合装置可以通过人脸识别技术，对模板图像中人脸上的眼睛、鼻子、眉毛及嘴巴等各个人脸器官进行特征点检测，生成第二人脸特征点，该第二人脸特征点可以是各个人脸器官对应的关键点的位置坐标信息，该第二人脸特征点可以在人脸的外部轮廓和各个人脸器官的边缘或中心等，该第二人脸特征点的个数可以是106个、83个或68个等，具体个数可以根据实际需要进行灵活设置。为了能够将二维人脸图像与模板图像进行精准融合，可以设置第一人脸特征点和第二人脸特征点的个数一致。例如，进行人脸特征点检测得到包含的人脸特征点人脸图像，如图6(a)所示。在图6(a)中，各特征点可以进行编号或以其他方式进行标识，例如，其中编号为56、62、64的特征点。In addition, the image fusion device may use face recognition technology to detect feature points on each face organ such as eyes, nose, eyebrows, and mouth in the template image to generate a second face feature point, and the second face The feature point may be position coordinate information of a key point corresponding to each face organ, and the second face feature point may be in the outer contour of the face and the edge or center of each face organ, etc. The number can be 106, 83, or 68. The specific number can be flexibly set according to actual needs. In order to be able to accurately fuse the two-dimensional face image and the template image, the number of feature points of the first face and the number of feature points of the second face can be set to be the same. For example, face feature point detection is performed to obtain a face image containing the face feature points, as shown in FIG. 6 (a). In FIG. 6 (a), each feature point may be numbered or identified in other ways, for example, feature points numbered 56, 62, and 64.
此时，图像融合装置可以根据第一人脸特征点和第二人脸特征点，将二维人脸图像中的人脸融合至模板图像中的人脸所在区域。At this time, the image fusion device may fuse the face in the two-dimensional face image to the area where the face in the template image is located according to the first face feature point and the second face feature point.
在某些实施方式中，根据第一人脸特征点和第二人脸特征点，将二维人脸图像中的人脸融合至模板图像中的人脸所在区域的步骤包括：In some embodiments, the step of fusing a face in a two-dimensional face image to a region where the face is in a template image according to the first face feature point and the second face feature point includes:
(a)根据第一人脸特征点，对二维人脸图像中的人脸进行三角形划分，得到第一人脸三角形区域；(a) Triangulate the face in the two-dimensional face image according to the first face feature point to obtain the first face triangle area;
(b)根据第二人脸特征点，对模板图像中的人脸进行三角形划分，得到第二人脸三角形区域；(b) Triangulate the face in the template image according to the second face feature point to obtain the second face triangle area;
(c)基于第一人脸三角形区域和第二人脸三角形区域进行仿射变换，得到目标人脸三角形区域；(c) performing affine transformation based on the first face triangle area and the second face triangle area to obtain the target face triangle area;
(d)将目标人脸三角形区域融合至模板图像中的人脸所在区域。(d) Fusion the triangle area of the target face into the area of the face in the template image.
具体地，图像融合装置可以根据第一人脸特征点对二维人脸图像中的人脸进行三角形划分，得到第一人脸三角形区域，例如，将各个人脸特征点按照预设算法进行连线，从而可以将人脸划分为多个三角形区域，该预设算法可以根据实际需要进行灵活设置。以及，根据第二人脸特征点对模板图像中的人脸进行三角形划分，得到第二人脸三角形区域，例如，根据人脸特征点对人脸进行三角形划分得到包含人脸三角形区域的人脸图像，如图6(b)所示。Specifically, the image fusion device may perform triangle division on the face in the two-dimensional face image according to the first face feature point to obtain the first face triangle area. For example, each face feature point is connected according to a preset algorithm. Lines, so that the face can be divided into multiple triangular regions, and the preset algorithm can be flexibly set according to actual needs. And, the face in the template image is triangle-divided according to the second face feature point to obtain a second face triangle area. For example, the face is triangle-divided according to the face feature point to obtain a face including the face triangle area. The image is shown in Figure 6 (b).
然后，图像融合装置可以基于二维人脸图像对应的第一人脸三角形区域和模板图像对应的第二人脸三角形区域进行仿射变换，得到目标人脸三角形区域。例如，可以将第一人脸三角形区域和第二人脸三角形区域中对应位置的人脸三角形区域进行比较，若第一人脸三角形区域和第二人脸三角形区域中对应位置的人脸三角形区域的形状和大小等不一致，此时可以对第一人脸三角形区域中该位置的人脸三角形区域进行缩放、旋转或平移等操作，得到目标人脸三角形区域，使得目标人脸三角形区域和第二人脸三角形区域中对应位置的人脸三角形区域形状和大小等一致。Then, the image fusion device may perform affine transformation based on the first face triangle area corresponding to the two-dimensional face image and the second face triangle area corresponding to the template image to obtain the target face triangle area. For example, the first face triangle area and the second face face triangle area at the corresponding position of the face triangle area can be compared. If the first face triangle area and the second face face triangle area correspond to the corresponding face triangle area. The shape and size of the target face are inconsistent. At this time, you can perform operations such as zooming, rotating, or panning on the face triangle area at the position in the first face triangle area to obtain the target face triangle area, making the target face triangle area and the second face area The shape and size of the face triangle area at the corresponding position in the face triangle area are the same.
此时，可以将目标人脸三角形区域融合至模板图像中的人脸所在区域，得到融合后图像，例如，该融合后图像可以如图7所示。为了使得融合效果更加自然，可以使用alpha-blend或者seamless clone等融合算法进行融合。At this time, the triangle area of the target face can be fused to the area of the face in the template image to obtain a fused image. For example, the fused image can be shown in FIG. 7. In order to make the fusion effect more natural, you can use fusion algorithms such as alpha-blend or seamless clone.
以下将进行举例说明，当待融合二维图像中的人脸为正脸，且模板图像中的人脸为侧脸时，根据待融合二维图像中的正脸构建三维人脸模型，得到的该三维人脸模型中的人脸为正脸，以及获取模板图像中的侧脸在三维空间中相对于正脸的旋转参数，该旋转参数为模板图像中侧脸在三维空间中的相对于正脸的旋转参数。The following will illustrate by example. When the human face in the two-dimensional image to be fused is a positive face and the human face in the template image is a side face, a three-dimensional human face model is constructed based on the positive face in the two-dimensional image to be fused. The face in the three-dimensional face model is a front face, and a rotation parameter of the side face in the template image relative to the front face in the three-dimensional space is obtained. The rotation parameter is the side face relative to the positive face in the three-dimensional space in the template image. Face rotation parameters.
此时，基于旋转参数对三维人脸模型进行旋转，得到旋转后三维人脸模型的步骤可以包括：基于旋转参数对三维人脸模型中正脸的进行旋转，得到旋转后三维人脸模型，旋转后三维人脸模型中的人脸为侧脸。将二维人脸图像与模板图像进行融合的步骤可以包括：将二维人脸图像中的旋转后侧脸融合至模板 图像中的侧脸所在区域。从而通过构建三维人脸模型的方式来解决侧脸融合难的问题，用户只需正常拍摄正脸图像，通过正脸图像构建三维人脸模型，再自动化旋转三维人脸模型到与模板图中侧脸方向一致，最后将旋转后三维人脸模型转换为二维人脸图像，进行二维人脸图像与模板图像之间进行侧脸融合。例如，只需要待融合二维图像中人脸为正脸，就能实现与各个侧脸的模板图进行侧脸融合，方便快捷且效果很好。At this time, the three-dimensional face model is rotated based on the rotation parameters to obtain a rotated three-dimensional face model. The method may include: rotating the positive face in the three-dimensional face model based on the rotation parameters to obtain a rotated three-dimensional face model. The face in the three-dimensional face model is a side face. The step of fusing the two-dimensional face image with the template image may include: fusing the rotated side face in the two-dimensional face image to an area where the side face in the template image is located. In order to solve the problem of difficult side-face fusion by constructing a three-dimensional face model, the user only needs to take a normal face image normally, construct a three-dimensional face model from the front face image, and then automatically rotate the three-dimensional face model to the side of the template image. The face directions are the same. Finally, the rotated 3D face model is converted into a 2D face image, and a side face fusion is performed between the 2D face image and the template image. For example, as long as the face in the two-dimensional image to be fused is a normal face, side-face fusion with the template image of each side face can be realized, which is convenient, fast, and effective.
需要说明的是，图像融合装置可以先获取待融合二维图像，并根据待融合二维图像中的人脸构建三维人脸模型；获取模板图像，根据模板图像中的人脸构建三维模型，以及基于三维模型获取模板图像中人脸在三维空间中的旋转参数；然后，基于旋转参数对三维人脸模型进行旋转，得到旋转后三维人脸模型；其次，将旋转后三维人脸模型中的人脸区域与三维模型中的人脸区域进行融合，得到融合后三维人脸模型，最后将融合后三维人脸模型转换为二维人脸图像；等等。It should be noted that the image fusion device may first obtain a two-dimensional image to be fused, and construct a three-dimensional face model based on the face in the two-dimensional image to be fused; obtain a template image, and construct a three-dimensional model based on the face in the template image, and Based on the three-dimensional model, the rotation parameters of the face in the template image in three-dimensional space are obtained; then, the three-dimensional face model is rotated based on the rotation parameter to obtain the rotated three-dimensional face model; secondly, the person in the rotated three-dimensional face model is The face area is fused with the face area in the 3D model to obtain a fused 3D face model, and finally the fused 3D face model is converted into a 2D face image; and so on.
由上可知，本申请实施例可以获取待融合二维图像，并根据待融合二维图像中的人脸构建三维人脸模型，以及，获取模板图像及模板图像中的人脸在三维空间中的旋转参数，然后基于旋转参数对三维人脸模型进行旋转，得到旋转后三维人脸模型，再将旋转后三维人脸模型转换为二维人脸图像，此时，可以将二维人脸图像与模板图像进行融合，例如将二维人脸图像中的人脸融合至模板图像中的人脸所在区域。该方案通过构建并旋转三维人脸模型，从而可以基于旋转后三维人脸模型得到二维人脸图像，使得二维人脸图像与模板图像中人脸的方向一致，以便将二维人脸图像中的人脸精准融合至模板图像中的人脸所在区域，提高了对图像中人脸进行融合的效果。It can be known from the above that the embodiment of the present application can obtain a two-dimensional image to be fused, and construct a three-dimensional face model based on the human face in the two-dimensional image to be fused, and obtain the template image and the human face in the template image in the three-dimensional space. Rotation parameters, and then rotate the 3D face model based on the rotation parameters to obtain a rotated 3D face model, and then convert the rotated 3D face model into a 2D face image. At this time, the 2D face image and The template image is fused, for example, the face in the two-dimensional face image is fused to the area where the face in the template image is located. This solution constructs and rotates a three-dimensional face model, so that a two-dimensional face image can be obtained based on the rotated three-dimensional face model, so that the two-dimensional face image is consistent with the direction of the face in the template image, so that the two-dimensional face image The face in the image is accurately fused to the area where the face in the template image is located, which improves the effect of merging the face in the image.
根据上述实施例所描述的方法，以下将举例作进一步详细说明。According to the method described in the foregoing embodiment, an example will be further described in detail below.
本实施例以图像融合装置为终端为例，以待融合二维图像中的人脸为正脸，且模板图像中的人脸为侧脸，待融合二维图像和模板图像中的人脸均为一个为例，可以理解的是，本实施例只是为了便于描述所举的例子，不应理解为是对待融合二维图像和模板图像中的人脸方向、及个数进行限定，但不管待融合二维图像和模板图像中的人脸方向是何种方向、及个数为多少，其图像融合的过程都是类似的，都可以按照该示例进行理解。In this embodiment, an image fusion device is used as a terminal. The human face in the two-dimensional image to be fused is a positive face, and the human face in the template image is a side face. As an example, it can be understood that this embodiment is only for the convenience of describing the examples given, and should not be construed as limiting the direction and number of faces in the two-dimensional image and the template image, but it does not matter The direction and the number of faces in the two-dimensional image and the template image are fused. The image fusion process is similar and can be understood according to this example.
请参阅图8，图8为本申请实施例提供的图像融合方法的流程示意图。该方法流程可以包括：Please refer to FIG. 8, which is a schematic flowchart of an image fusion method according to an embodiment of the present application. The method flow may include:
S201、终端获取待融合二维图像和模板图像。S201. The terminal obtains a two-dimensional image and a template image to be fused.
其中，该待融合二维图像中可以包括一张正脸，如图3(a)所示；该模板图像中可以包含一张侧脸，如图3(b)所示，当然，待融合二维图像和模板图像中还可以包括其他的物体。The two-dimensional image to be fused may include a front face, as shown in FIG. 3 (a); the template image may include a side face, as shown in FIG. 3 (b). Of course, the two to be fused two The dimensional image and the template image may also include other objects.
待融合二维图像和模板图像可以是终端在拍照的过程中，拍照摄像头采集到的图像，也可以是终端预先设置的图像，还可以通过在互联网上搜索或者从图像数据库中获取得到的图像等；当然，也可以是通过其他方式获取到的图像，具体获取方式在此处不作限定。例如，待融合二维图像可以是用户自拍得到包含正脸的图像，模板图像可以是预先设置的图像。The two-dimensional image to be fused and the template image can be the image collected by the camera during the photographing process of the terminal, or the image preset by the terminal, or the image obtained by searching on the Internet or obtaining from the image database, etc. ; Of course, it can also be an image obtained by other methods, and the specific acquisition method is not limited here. For example, the two-dimensional image to be fused may be an image obtained by a user taking a self-portrait, and the template image may be a preset image.
S202、终端根据预设人脸模型，获取待融合二维图像中的人脸在三维平面内的三维坐标点，并根据三维坐标点构建三维人脸模型。S202. The terminal obtains a three-dimensional coordinate point of a face in a two-dimensional image to be fused in a three-dimensional plane according to a preset face model, and constructs a three-dimensional face model according to the three-dimensional coordinate point.
其中，预设人脸模型可以是三维平均人脸模型，该三维平均人脸模型可以是预先设置，可以通过改变三维平均人脸模型的三维坐标点的位置信息来生成不同的表情及不同身份的用户的三维人脸模型，也就是说所有三维人脸模型都可以通过在三维平均人脸模型的三维坐标点上增加偏移来表示。The preset face model may be a three-dimensional average face model, and the three-dimensional average face model may be preset, and different expressions and different identities may be generated by changing the position information of the three-dimensional coordinate points of the three-dimensional average face model. The user's three-dimensional face model, that is, all three-dimensional face models can be represented by adding offsets to the three-dimensional coordinate points of the three-dimensional average face model.
终端可以根据预设人脸模型，获取待融合二维图像中的正脸在三维平面内的三维坐标点，并根据三维坐标点构建三维人脸模型。例如，终端可以按照上述公式(1)、公式(2)、公式(3)及公式(4)，获取待融合二维图像中的人脸投影在二维平面内的第一投影参数，以及获取预设人脸模型投影在二维平面内的第二投影参数，然后根据第一投影参数及第二投影参数，获取待融合二维图像中的人脸在三维平面内的三维坐标点，并根据三维坐标点构建三维人脸模型；等等，该三维人脸模型可以如图4(a)所示。The terminal can obtain the three-dimensional coordinate points of the normal face in the two-dimensional image to be fused in the three-dimensional plane according to the preset face model, and construct a three-dimensional face model according to the three-dimensional coordinate points. For example, the terminal may obtain the first projection parameter of the human face in the two-dimensional image to be fused in a two-dimensional plane according to the above formula (1), formula (2), formula (3), and formula (4), and obtain Preset the second projection parameters of the face model projected in the two-dimensional plane, and then obtain the three-dimensional coordinate points of the face in the two-dimensional image to be fused in the three-dimensional plane according to the first projection parameter and the second projection parameter. The three-dimensional coordinate points are used to construct a three-dimensional face model; and so on, the three-dimensional face model may be as shown in FIG. 4 (a).
S203、终端根据模板图像中的人脸构建三维模型，基于三维模型获取模板图像中人脸在三维空间中的旋转参数。S203. The terminal constructs a three-dimensional model according to the face in the template image, and obtains a rotation parameter of the face in the three-dimensional space based on the three-dimensional model.
其中，S203可以在步骤S202之前执行，也可以在步骤S202之后执行。Among them, S203 may be performed before step S202, or may be performed after step S202.
终端可以按照上述公式(1)、公式(2)、公式(3)及公式(4)，获取模板图像中的人脸投影在二维平面内的第三投影参数；获取预设人脸模型，及预设人脸模型投影在二维平面内的第四投影参数；然后根据第三投影参数及第四投影参数，获取模板图像中的人脸在三维平面内的目标三维坐标点，并根据目标三维坐标点构建三维模型；等等。在得到三维模型后，可以基于三维模型获取模板图像中侧脸在三维空间中相对于正脸的旋转参数，该旋转参数即为上 述公式(4)中的旋转系数R。例如，旋转参数可以包括在X轴方向上旋转的x角度、在Y轴方向上旋转的y角度、及在Z轴方向上旋转的z角度等。该旋转参数还可以是由用户设定的。The terminal may obtain the third projection parameter of the face projected in the template image in a two-dimensional plane according to the above formula (1), formula (2), formula (3), and formula (4); obtain a preset face model, And preset a fourth projection parameter in which the face model is projected in a two-dimensional plane; and then according to the third projection parameter and the fourth projection parameter, obtain the target three-dimensional coordinate point of the face in the template image in the three-dimensional plane, and according to the target 3D coordinate points to build a 3D model; etc. After the three-dimensional model is obtained, the rotation parameter of the side face in the three-dimensional space relative to the front face in the template image can be obtained based on the three-dimensional model, and the rotation parameter is the rotation coefficient R in the above formula (4). For example, the rotation parameter may include an x-angle rotated in the X-axis direction, a y-angle rotated in the Y-axis direction, a z-angle rotated in the Z-axis direction, and the like. The rotation parameter may also be set by a user.
S204、终端基于旋转参数对三维人脸模型进行旋转，得到旋转后三维人脸模型。S204. The terminal rotates the three-dimensional face model based on the rotation parameters to obtain a rotated three-dimensional face model.
终端可以将旋转参数应用到三维人脸模型上，控制三维人脸模型在三维空间中进行X轴、Y轴、及Z轴等方向上的旋转，使得三维人脸模型中人脸的方向与模板图中人脸方向一致。例如，可以基于模板图中侧脸在三维空间中相对于正脸的旋转参数，对三维人脸模型中的正脸进行旋转，得到旋转后三维人脸模型中的人脸为侧脸，旋转后三维人脸模型如图4(b)所示。The terminal can apply the rotation parameters to the 3D face model, and control the 3D face model to rotate in the X, Y, and Z directions in the 3D space, so that the direction of the face and the template in the 3D face model Face orientation is the same in the picture. For example, based on the rotation parameters of the side face relative to the front face in the three-dimensional space in the template map, the front face in the three-dimensional face model can be rotated to obtain the face in the rotated three-dimensional face model as the side face. The three-dimensional face model is shown in Figure 4 (b).
S205、终端将旋转后人脸三维模型转换为二维人脸图像。S205. The terminal converts the three-dimensional face model after rotation into a two-dimensional face image.
终端可以接收截图指令，根据截图指令直接截取旋转后人脸三维模型在显示界面内显示的图像，得到二维人脸图像；或者，按照上述根据人脸构建三维人脸模型的方法进行逆映射，将旋转后人脸三维模型逆映射为二维人脸图像，当然，还可以通过其他方式将旋转后人脸三维模型转换为二维人脸图像。例如，该二维人脸图像可以如图5所示，得到的二维人脸图像中的人脸为侧脸。The terminal may receive a screenshot instruction, and directly capture an image displayed on the display interface of the rotated three-dimensional model of the face according to the screenshot instruction to obtain a two-dimensional face image; or, perform inverse mapping in accordance with the method for constructing a three-dimensional face model based on a face, The three-dimensional model of the rotated face is inversely mapped into a two-dimensional face image. Of course, the three-dimensional model of the rotated face can also be converted into a two-dimensional face image by other methods. For example, the two-dimensional face image may be as shown in FIG. 5, and the face in the obtained two-dimensional face image is a side face.
S206、终端对二维人脸图像和模板图像分别进行人脸特征点检测，对应得到第一人脸特征点和第二人脸特征点。S206. The terminal performs face feature point detection on the two-dimensional face image and the template image, respectively, and obtains the first face feature point and the second face feature point correspondingly.
终端可以通过人脸识别技术分别对二维人脸图像中人脸上的眼睛、鼻子、眉毛及嘴巴等各个人脸器官进行特征点检测，生成第一人脸特征点，以及分别对模板图像中人脸上的眼睛、鼻子、眉毛及嘴巴等各个人脸器官进行特征点检测，生成第二人脸特征点。该第一人脸特征点和第二人脸特征点可以是各个人脸器官对应的关键点的位置坐标信息，该第一人脸特征点和第二人脸特征点可以在人脸的外部轮廓和各个人脸器官的边缘或中心等。该第一人脸特征点和第二人脸特征点的个数可以是106个、83个或68个等，具体个数可以根据实际需要进行灵活设置。例如，进行人脸特征点检测得到包含的人脸特征点人脸图像，如图6(a)所示。The terminal can use face recognition technology to detect feature points on each face organ such as eyes, nose, eyebrows, and mouth in the two-dimensional face image, generate first face feature points, and separately in the template image Face points such as eyes, nose, eyebrows, and mouth of a human face are detected to generate feature points to generate a second face feature point. The first facial feature point and the second facial feature point may be position coordinate information of a key point corresponding to each facial organ, and the first facial feature point and the second facial feature point may be on the outer contour of the human face. And the edges or centers of various facial organs. The number of the first face feature points and the second face feature points may be 106, 83, or 68, and the specific number may be flexibly set according to actual needs. For example, face feature point detection is performed to obtain a face image containing the face feature points, as shown in FIG. 6 (a).
S207、终端根据第一人脸特征点和第二人脸特征点，将二维人脸图像中的人脸融合至模板图像中的人脸所在区域。S207. The terminal fuses the face in the two-dimensional face image to the area where the face in the template image is located according to the first face feature point and the second face feature point.
终端可以根据第一人脸特征点对二维人脸图像中的人脸进行三角形划分，得到第一人脸三角形区域，以及，根据第二人脸特征点对模板图像中的人脸进 行三角形划分，得到第二人脸三角形区域，例如，根据人脸特征点对人脸进行三角形划分得到包含人脸三角形区域的人脸图像，如图6(b)所示。The terminal may perform triangle division on the face in the two-dimensional face image according to the first face feature point to obtain a triangle region of the first face, and perform triangle division on the face in the template image according to the second face feature point. To obtain a second triangular area of the face, for example, triangulating the face according to the facial feature points to obtain a face image containing the triangular area of the face, as shown in FIG. 6 (b).
然后，终端可以基于二维人脸图像对应的第一人脸三角形区域和模板图像对应的第二人脸三角形区域进行仿射变换，得到目标人脸三角形区域。例如，图6(a)的二维人脸图像中人脸特征点56、62和64之间的连线构成一个人脸三角形区域A1(如图6(b)所示)，模板图像中人脸特征点56、62和64(未示出)之间的连线构成一个人脸三角形区域A2(未示出)，将人脸三角形区域A1与人脸三角形区域A2进行比较，若人脸三角形区域A1与人脸三角形区域A2的形状和大小等不一致，此时可以对人脸三角形区域A1进行缩放、旋转或平移等操作，得到目标人脸三角形区域A，该目标人脸三角形区域A可以与人脸三角形区域A2形状和大小等一致，其他人脸三角形区域也进行同样的处理。此时，终端可以将目标人脸三角形区域融合至模板图像中的人脸所在区域，得到融合后图像，例如，该融合后图像可以如图7所示。Then, the terminal may perform affine transformation based on the first face triangle area corresponding to the two-dimensional face image and the second face triangle area corresponding to the template image to obtain the target face triangle area. For example, the connection between the face feature points 56, 62, and 64 in the two-dimensional face image of Fig. 6 (a) constitutes a triangle area A1 of the face (as shown in Fig. 6 (b)). The line between the face feature points 56, 62 and 64 (not shown) constitutes a face triangle area A2 (not shown). The face triangle area A1 is compared with the face triangle area A2. The shape and size of the area A1 and the face triangle area A2 are not consistent. At this time, the face triangle area A1 can be zoomed, rotated, or translated to obtain the target face triangle area A. The target face triangle area A can be The face triangle area A2 has the same shape and size, and other face triangle areas also perform the same processing. At this time, the terminal may fuse the triangle area of the target face to the area of the face in the template image to obtain a fused image. For example, the fused image may be as shown in FIG. 7.
本申请实施例通过待融合二维图像中的正脸构建三维人脸模型，并根据模板图像中的侧脸的旋转参数自动化旋转三维人脸模型，使得三维人脸模型中正脸与模板图中侧脸方向一致，最后将旋转后三维人脸模型转换为二维人脸图像，将二维人脸图像与模板图像进行侧脸融合。该方案通过构建三维人脸模型的方式改善二维侧脸融合效果，只需要待融合二维图像中人脸为正脸，就能实现与模板图中任意角度的侧脸进行完美融合，得到融合后图像的融合效果非常好，融合后的人脸自然美观，并且全程自动化，方便快捷。例如，可以完善疯狂变脸的应用场景，增加趣味和可玩性。The embodiment of the present application constructs a three-dimensional face model by using a front face in a two-dimensional image to be fused, and automatically rotates the three-dimensional face model according to a rotation parameter of a side face in a template image, so that the front face in the three-dimensional face model and the side of the template image Face directions are the same. Finally, the rotated 3D face model is converted into a 2D face image, and the 2D face image and the template image are subjected to side-face fusion. This solution improves the two-dimensional side-face fusion effect by constructing a three-dimensional face model. As long as the face in the two-dimensional image to be fused is a normal face, it can be perfectly fused with the side face at any angle in the template image to obtain the fusion. The fusion effect of the post-image is very good, the face after fusion is naturally beautiful, and the whole process is automated, convenient and quick. For example, it can perfect the application scene of crazy face change, increasing the fun and playability.
为便于更好的实施本申请实施例提供的图像融合方法，本申请实施例还提供一种基于上述图像融合方法的装置。其中名词的含义与上述图像融合方法中相同，具体实现细节可以参考方法实施例中的说明。In order to facilitate better implementation of the image fusion method provided in the embodiment of the present application, the embodiment of the present application further provides a device based on the above image fusion method. The meanings of the nouns are the same as in the above image fusion method. For specific implementation details, refer to the description in the method embodiment.
请参阅图9，图9为本申请实施例提供的图像融合装置的结构示意图，其中该图像融合装置可以包括构建单元301、获取单元302、旋转单元303、转换单元304及融合单元305等。Please refer to FIG. 9, which is a schematic structural diagram of an image fusion apparatus according to an embodiment of the present application. The image fusion apparatus may include a construction unit 301, an acquisition unit 302, a rotation unit 303, a conversion unit 304, and a fusion unit 305.
其中，构建单元301，用于获取待融合二维图像，并根据待融合二维图像中的人脸构建三维人脸模型。The construction unit 301 is configured to obtain a two-dimensional image to be fused, and construct a three-dimensional face model according to a face in the two-dimensional image to be fused.
该待融合二维图像中可以包含一张或多张人脸，还可以包括其他的物体，其中，该人脸可以正脸或者侧脸等。例如，如图3(a)所示，该待融合二维图像中可以包括一张正脸。The two-dimensional image to be fused may include one or more human faces, and may also include other objects, wherein the human face may be a front face or a side face. For example, as shown in FIG. 3 (a), the two-dimensional image to be fused may include a front face.
待融合二维图像可以是在拍照的过程中，拍照摄像头采集到的图像；也可以是图像融合装置预存的图像；还可以是通过在互联网上搜索或者从图像数据库中获取到的图像等；当然，待融合二维图像也可以是通过其他方式获取到的图像，具体获取方式在此处不作限定。The two-dimensional image to be fused may be an image collected by a camera during the photographing process; it may also be an image pre-stored by an image fusion device; it may also be an image obtained by searching on the Internet or obtained from an image database, etc .; of course The two-dimensional image to be fused may also be an image obtained by other methods, and the specific acquisition method is not limited here.
在某些实施方式中，构建单元301可以包括第一获取子单元和第一构建子单元等具体可以如下：In some embodiments, the construction unit 301 may include a first acquisition subunit, a first construction subunit, and the like, which may specifically be as follows:
第一获取子单元，用于获取预设人脸模型；A first acquisition subunit, configured to acquire a preset face model;
第一构建子单元，用于根据预设人脸模型，获取待融合二维图像中的人脸在三维平面内的三维坐标点，并根据三维坐标点构建三维人脸模型。The first construction subunit is configured to obtain a three-dimensional coordinate point of a face in a two-dimensional image to be fused in a three-dimensional plane according to a preset face model, and construct a three-dimensional face model according to the three-dimensional coordinate point.
其中，预设人脸模型可以是三维平均人脸模型，该三维平均人脸模型可以是预先设置。可以通过改变三维平均人脸模型的三维坐标点的位置信息来生成不同的表情及不同身份的用户的三维人脸模型，也就是说所有三维人脸模型都可以通过在三维平均人脸模型的三维坐标点上增加偏移来表示。The preset face model may be a three-dimensional average face model, and the three-dimensional average face model may be preset. You can generate 3D face models of users with different expressions and different identities by changing the position information of the 3D coordinate points of the 3D average face model. That is to say, all 3D face models can An offset is added to the coordinate point to indicate.
第一获取子单元可以接收设置指令，并基于设置指令设置预设人脸模型；或者，第一获取子单元可以向服务器发送模型获取请求，并接收服务器基于模型获取请求返回的预设人脸模型。在得到预设人脸模型后，第一构建子单元可以根据预设人脸模型，获取待融合二维图像中的人脸在三维平面内的三维坐标点，并根据三维坐标点构建三维人脸模型。The first obtaining subunit may receive a setting instruction and set a preset face model based on the setting instruction; or, the first obtaining subunit may send a model obtaining request to the server and receive the preset face model returned by the server based on the model obtaining request. . After obtaining the preset face model, the first construction subunit can obtain the three-dimensional coordinate points of the face in the two-dimensional image to be fused in the three-dimensional plane according to the preset face model, and construct a three-dimensional face based on the three-dimensional coordinate points. model.
在某些实施方式中，第一构建子单元具体可以用于：获取待融合二维图像中的人脸投影在二维平面内的第一投影参数；获取预设人脸模型投影在二维平面内的第二投影参数；根据第一投影参数及第二投影参数，获取待融合二维图像中的人脸在三维平面内的三维坐标点；根据三维坐标点构建三维人脸模型。In some embodiments, the first constructing subunit may be specifically configured to: obtain a first projection parameter of a human face projected in a two-dimensional image to be fused in a two-dimensional plane; and obtain a preset face model projected in a two-dimensional plane. A second projection parameter in the image; obtaining a three-dimensional coordinate point of a face in a two-dimensional image to be fused in a three-dimensional plane according to the first projection parameter and the second projection parameter; and constructing a three-dimensional face model according to the three-dimensional coordinate point.
具体地，第一构建子单元根据待融合二维图像中的人脸构建三维人脸模型的过程中，可以从待融合二维图像中提取出人脸，并获取待融合二维图像中的人脸投影在二维平面内的第一投影参数，该第一投影参数可以是人脸在二维平面内的坐标点。以及，第一构建子单元可以获取预设人脸模型投影在二维平面内的第二投影参数，该第二投影参数可以是预设人脸模型在二维平面内的坐标点。由于可以通过改变三维平均人脸模型的三维坐标点的位置信息来生成不同 的表情及不同身份的用户的三维人脸模型，因此，可以通过上述公式(1)、公式(2)、公式(3)和公式(4)，根据第一投影参数及第二投影参数，获取待融合二维图像中的人脸在三维平面内的三维坐标点，并根据三维坐标点构建三维人脸模型。也就是说，如果将各个角度的三维人脸模型投影到二维平面都可以与待融合二维图像中人脸相匹配，那么这个三维人脸模型就是需要获取的三维人脸模型。例如，根据待融合二维图像中的人脸构建得到的三维人脸模型如图4(a)所示。Specifically, in the process of constructing a three-dimensional face model based on the faces in the two-dimensional image to be fused by the first construction subunit, the human face can be extracted from the two-dimensional image to be fused, and the person in the two-dimensional image to be fused is obtained. A first projection parameter of the face projection in the two-dimensional plane, and the first projection parameter may be a coordinate point of the human face in the two-dimensional plane. And, the first construction subunit may obtain a second projection parameter of the preset face model projected in the two-dimensional plane, and the second projection parameter may be a coordinate point of the preset face model in the two-dimensional plane. Since the position information of the three-dimensional coordinate points of the three-dimensional average face model can be changed to generate three-dimensional face models of users with different expressions and different identities, the above formula (1), formula (2), formula (3) ) And formula (4), according to the first projection parameter and the second projection parameter, obtain the three-dimensional coordinate points of the face in the two-dimensional image to be fused in the three-dimensional plane, and construct a three-dimensional face model according to the three-dimensional coordinate points. That is, if the three-dimensional face model of each angle is projected onto a two-dimensional plane and can be matched with the face in the two-dimensional image to be fused, then this three-dimensional face model is a three-dimensional face model that needs to be obtained. For example, a three-dimensional face model constructed based on a face in a two-dimensional image to be fused is shown in FIG. 4 (a).
需要说明的是，除了可以通过上述公式构建三维人脸模型之外，三维人脸模型还可以是通过其他方式获取，具体构建方式在此不做限定。It should be noted that in addition to using the above formula to construct a three-dimensional face model, the three-dimensional face model may also be obtained through other methods, and the specific construction method is not limited here.
需要说明的是，当待融合二维图像中的人脸包括多个时，第一构建子单元可以从待融合二维图像中任意提取一张人脸来构建三维人脸模型，或者，第一构建子单元可以输出选择项，并接收用户基于选择项从待融合二维图像中选择其中一张人脸来构建三维人脸模型，或者，第一构建子单元可以从待融合二维图像中提取出所有人脸，并根据各个人脸分别构建三维人脸模型；等等。It should be noted that when there are multiple faces in the two-dimensional image to be fused, the first construction subunit may arbitrarily extract a face from the two-dimensional image to be fused to construct a three-dimensional face model, or the first The construction sub-unit may output selection items and receive a user to select a face from the two-dimensional image to be fused based on the selection items to construct a three-dimensional face model. Alternatively, the first construction sub-unit may be extracted from the two-dimensional image to be fused. Make all faces, and build 3D face models based on each face; and so on.
获取单元302，用于获取模板图像，以及获取模板图像中的人脸在三维空间中的旋转参数。The obtaining unit 302 is configured to obtain a template image, and obtain a rotation parameter of a human face in the template image in a three-dimensional space.
该模板图像中可以包含一张或多张人脸，还可以包括其他的物体，其中，该人脸可以是正脸或者任意角度的侧脸。例如，如图3(b)所示，该模板图像中可以包含一张侧脸。模板图像可以是通过拍照摄像头采集到的图像，也可以是预先设置的图像，还可以是通过在互联网上搜索或者从图像数据库中获取到的图像等；当然，模板图像也可以是通过其他方式获取到，具体获取方式在此处不作限定。The template image may include one or more human faces, and may also include other objects, where the human face may be a front face or a side face at any angle. For example, as shown in FIG. 3 (b), the template image may include a side face. The template image can be an image collected through a camera, or a preset image, or an image obtained by searching on the Internet or obtained from an image database; of course, the template image can also be obtained by other methods The specific acquisition method is not limited here.
获取单元302可以获取模板图像中的人脸在三维空间中的旋转参数，该旋转参数可以是人脸在三维空间中的X轴、Y轴、及Z轴方向上的转向，例如，当模板图像中的人脸为正脸时，可以设置模板图像中的正脸在三维空间中的旋转参数为0；当模板图像中的人脸为侧脸时，可以设置模板图像中的侧脸相对于正脸的旋转参数为：在X轴方向上旋转的x角度、在Y轴方向上旋转的y角度、及在Z轴方向上旋转的z角度。The obtaining unit 302 may obtain a rotation parameter of a face in a three-dimensional space in the template image, and the rotation parameter may be a turning direction of the face in the X-axis, Y-axis, and Z-axis directions in the three-dimensional space. For example, when the template image When the face in the template is a positive face, the rotation parameter of the normal face in the template image in the three-dimensional space can be set to 0; when the face in the template image is a side face, the side face in the template image can be set relative to the positive face. The rotation parameters of the face are: an x angle rotated in the X axis direction, a y angle rotated in the Y axis direction, and a z angle rotated in the Z axis direction.
获取单元302可以接收用户输入的旋转参数，并将该旋转参数设置为模板图像中的人脸在三维空间中的旋转参数，即该旋转参数可以由人工设置。The obtaining unit 302 may receive a rotation parameter input by a user and set the rotation parameter as a rotation parameter of a face in a template image in a three-dimensional space, that is, the rotation parameter may be manually set.
在某些实施方式中，获取单元302可以包括第二构建子单元和第二获取子单元等，具体可以如下：In some implementations, the obtaining unit 302 may include a second construction subunit, a second obtaining subunit, and the like, and may specifically be as follows:
第二构建子单元，用于根据模板图像中的人脸构建三维模型；A second construction subunit, configured to construct a three-dimensional model according to a face in the template image;
第二获取子单元，用于基于三维模型获取模板图像中人脸在三维空间中的旋转参数。The second obtaining subunit is configured to obtain a rotation parameter of the face in the template image in the three-dimensional space based on the three-dimensional model.
在某些实施方式中，第二构建子单元具体可以用于：In some embodiments, the second construction subunit may be specifically used for:
获取模板图像中的人脸投影在二维平面内的第三投影参数；Obtaining a third projection parameter of the face projection in the template image in a two-dimensional plane;
获取预设人脸模型，及预设人脸模型投影在二维平面内的第四投影参数；Obtaining a preset face model and a fourth projection parameter of the preset face model projected in a two-dimensional plane;
根据第三投影参数及第四投影参数，获取模板图像中的人脸在三维平面内的目标三维坐标点；Obtaining the target three-dimensional coordinate points of the human face in the template image in the three-dimensional plane according to the third projection parameter and the fourth projection parameter;
根据目标三维坐标点构建三维模型。Build a 3D model based on the target 3D coordinate points.
具体地，第二构建子单元可以从模板图像中提取出人脸，并获取模板图像中的人脸投影在二维平面内的第三投影参数，该第三投影参数可以是人脸在二维平面内的坐标点。以及，图像融合装置可以获取预设人脸模型投影在二维平面内的第四投影参数，该第四投影参数可以是预设人脸模型在二维平面内的坐标点。然后可以利用上述公式(1)、公式(2)、公式(3)和公式(4)根据第三投影参数及第四投影参数，计算模板图像中的人脸在三维平面内的目标三维坐标点，并根据目标三维坐标点构建三维模型。在得到三维模型后，第二获取子单元可以基于三维模型获取模板图像中人脸在三维空间中的旋转参数，该旋转参数即为上述公式(4)中的旋转系数R。Specifically, the second construction sub-unit may extract a human face from the template image, and obtain a third projection parameter of the human face in the template image projected in the two-dimensional plane, and the third projection parameter may be the human face in the two-dimensional plane. Coordinate points in the plane. And, the image fusion device may obtain a fourth projection parameter of the preset face model projected in the two-dimensional plane, and the fourth projection parameter may be a coordinate point of the preset face model in the two-dimensional plane. Then, the above formulas (1), (2), (3), and (4) can be used to calculate the target three-dimensional coordinate points of the human face in the template image in the three-dimensional plane according to the third projection parameter and the fourth projection parameter. , And build a 3D model based on the target 3D coordinate points. After obtaining the three-dimensional model, the second acquisition subunit may obtain a rotation parameter of the face in the three-dimensional space in the template image based on the three-dimensional model, and the rotation parameter is the rotation coefficient R in the above formula (4).
旋转单元303，用于基于旋转参数对三维人脸模型进行旋转，得到旋转后三维人脸模型。The rotation unit 303 is configured to rotate the three-dimensional face model based on the rotation parameters to obtain a rotated three-dimensional face model.
在得到三维人脸模型和旋转参数后，旋转单元303可以将旋转参数应用到三维人脸模型上，控制三维人脸模型在三维空间中进行X轴、Y轴、及Z轴等方向上的旋转，使得三维人脸模型中人脸的方向与模板图中人脸方向一致。例如，基于旋转参数对三维人脸模型进行旋转，得到旋转后三维人脸模型如图4(b)所示。After obtaining the three-dimensional face model and rotation parameters, the rotation unit 303 can apply the rotation parameters to the three-dimensional face model, and control the three-dimensional face model to rotate in the directions of X-axis, Y-axis, and Z-axis in the three-dimensional space. , So that the direction of the face in the three-dimensional face model is consistent with the direction of the face in the template image. For example, the 3D face model is rotated based on the rotation parameters, and the rotated 3D face model is obtained as shown in FIG. 4 (b).
例如，当三维人脸模型中的人脸为正脸，且模板图像中的人脸为侧脸时，为了使得三维人脸模型中人脸的方向与模板图中人脸方向一致，可以基于模板图中侧脸在三维空间中相对于正脸的旋转参数，对三维人脸模型中的正脸进行旋转，得到旋转后三维人脸模型中的人脸为侧脸。For example, when the face in the 3D face model is a positive face and the face in the template image is a side face, in order to make the direction of the face in the 3D face model consistent with the face direction in the template image, the template may be based on the template. In the figure, the rotation parameters of the side face relative to the front face in the three-dimensional space are used to rotate the front face in the three-dimensional face model, and the face in the three-dimensional face model after rotation is obtained as the side face.
又例如，当三维人脸模型中的人脸为侧脸，且模板图像中的人脸为正脸时，为了使得三维人脸模型中人脸的方向与模板图中人脸方向一致，可以基于模板图中正脸在三维空间中相对于侧脸的旋转参数，对三维人脸模型中的侧脸进行旋转，得到旋转后三维人脸模型中的人脸为正脸。For another example, when the face in the three-dimensional face model is a side face and the face in the template image is a positive face, in order to make the direction of the face in the three-dimensional face model consistent with the face direction in the template image, it can be based on The rotation parameters of the positive face in the template image relative to the side face in the three-dimensional space are used to rotate the side face in the three-dimensional face model to obtain that the face in the three-dimensional face model after rotation is the positive face.
转换单元304，用于将旋转后人脸三维模型转换为二维人脸图像。The conversion unit 304 is configured to convert the three-dimensional model of the face after the rotation into a two-dimensional face image.
在得到旋转后人脸三维模型后，转换单元304可以将旋转后人脸三维模型转换为二维人脸图像，该二维人脸图像可以如图5所示，例如，可以接收截图指令，根据截图指令直接截取旋转后人脸三维模型在显示界面内显示的图像，得到二维人脸图像；或者，按照上述根据人脸构建三维人脸模型的方法进行逆映射，将旋转后人脸三维模型逆映射为二维人脸图像，当然，还可以通过其他方式将旋转后人脸三维模型转换为二维人脸图像。After the three-dimensional model of the rotated face is obtained, the conversion unit 304 may convert the three-dimensional model of the rotated face into a two-dimensional face image. The two-dimensional face image may be as shown in FIG. 5. For example, a screenshot instruction may be received. The screenshot instruction directly captures the image displayed on the display interface of the rotated three-dimensional model of the face to obtain a two-dimensional face image; or, according to the method of constructing a three-dimensional face model based on the face described above, the inverse mapping is performed to convert the rotated three-dimensional face model The inverse mapping is a two-dimensional face image. Of course, there are other ways to convert the three-dimensional model of the rotated face into a two-dimensional face image.
融合单元305，用于将二维人脸图像与模板图像进行融合。A fusion unit 305 is configured to fuse a two-dimensional face image and a template image.
融合单元305可以先做人脸特征点检测，然后根据人脸特征点划分人脸三角形区域，然后根据人脸三角形区域做仿射变换，最后将二维人脸图像中人脸上的每一块人脸三角形区域融合到模板图像中人脸的对应位置上，还可以做一些边缘柔等后处理操作，即可完成人脸图像融合。The fusion unit 305 may first perform face feature point detection, then divide the face triangle area according to the face feature point, and then perform affine transformation based on the face triangle area, and finally each person in the face in the two-dimensional face image The triangular area of the face is fused to the corresponding position of the face in the template image, and some post-processing operations such as edge softening can also be performed to complete the face image fusion.
需要说明的是，当二维人脸图像和模板图像中的人脸均为多个时，融合单元305可以根据待融合二维图像中的各个人脸分别构建三维人脸模型，以及获取模板图像中的各个人脸在三维空间中的旋转参数，基于各个旋转参数分别对各个三维人脸模型进行旋转，得到旋转后三维人脸模型；然后将各个旋转后人脸三维模型转换为二维人脸图像，并将二维人脸图像中各个人脸与模板图像中各个人脸进行融合。例如，当二维人脸图像中包括第一人脸和第二人脸，模板图像中包括第三人脸和第四人脸时，可以根据第一人脸和第二人脸分别构建三维人脸模型，以及第三人脸和第四人脸在三维空间中的旋转参数，基于第三人脸对应的旋转参数对第一人脸对应的三维人脸模型进行旋转，基于第四人脸对应的旋转参数对第二人脸对应的三维人脸模型进行旋转，得到旋转后三维人脸模型；然后将各个旋转后人脸三维模型转换为二维人脸图像，最后将二维人脸图像中的第一人脸与模板图像中的第三人脸进行融合，将二维人脸图像中的第二人脸与模板图像中的第四人脸进行融合。It should be noted that when there are multiple faces in the two-dimensional face image and the template image, the fusion unit 305 may separately construct a three-dimensional face model according to each face in the two-dimensional image to be fused, and obtain a template image. The rotation parameters of each face in the three-dimensional space are based on each rotation parameter to rotate each three-dimensional face model to obtain a rotated three-dimensional face model; and then convert each rotated three-dimensional model into a two-dimensional face Image, and fuse each face in the two-dimensional face image with each face in the template image. For example, when a two-dimensional face image includes a first face and a second face, and a template image includes a third face and a fourth face, a three-dimensional person may be constructed based on the first face and the second face, respectively. The face model, and the rotation parameters of the third face and the fourth face in the three-dimensional space. Based on the rotation parameters corresponding to the third face, the three-dimensional face model corresponding to the first face is rotated, and the fourth face correspondence is based on Rotate the 3D face model corresponding to the second face to obtain a rotated 3D face model; then convert each rotated 3D model into a 2D face image, and finally convert the 2D face image into The first face in the image is fused with the third face in the template image, and the second face in the two-dimensional face image is fused with the fourth face in the template image.
在某些实施方式中，如图10所示，融合单元305可以包括第一检测子单元3051、第二检测子单元3052、及融合子单元3053等，具体可以如下：In some implementations, as shown in FIG. 10, the fusion unit 305 may include a first detection sub-unit 3051, a second detection sub-unit 3052, and a fusion sub-unit 3053, etc., which may specifically be as follows:
第一检测子单元3051，用于对二维人脸图像进行人脸特征点检测，得到第一人脸特征点；A first detection sub-unit 3051, configured to perform face feature point detection on a two-dimensional face image to obtain a first face feature point;
第二检测子单元3052，用于对模板图像进行人脸特征点检测，得到第二人脸特征点；A second detection subunit 3052, configured to detect a facial feature point on the template image to obtain a second facial feature point;
融合子单元3053，用于根据第一人脸特征点和第二人脸特征点，将二维人脸图像中的人脸融合至模板图像中的人脸所在区域。A fusion subunit 3053 is configured to fuse the face in the two-dimensional face image to the region where the face is in the template image according to the first face feature point and the second face feature point.
第一检测子单元3051可以通过人脸识别技术，对二维人脸图像中人脸上的眼睛、鼻子、眉毛及嘴巴等各个人脸器官进行特征点检测，生成第一人脸特征点，该第一人脸特征点可以是各个人脸器官对应的关键点的位置坐标信息，该第一人脸特征点可以在人脸的外部轮廓和各个人脸器官的边缘或中心等，该第一人脸特征点的个数可以是106个、83个或68个等，具体个数可以根据实际需要进行灵活设置。The first detection sub-unit 3051 may use face recognition technology to detect feature points on each face organ such as eyes, nose, eyebrows, and mouth in a two-dimensional face image to generate a first face feature point. The first face feature point may be position coordinate information of a key point corresponding to each face organ, and the first face feature point may be in the outer contour of the face and the edge or center of each face organ. The number of facial feature points can be 106, 83, or 68. The specific number can be flexibly set according to actual needs.
第二检测子单元3052可以通过人脸识别技术，对模板图像中人脸上的眼睛、鼻子、眉毛及嘴巴等各个人脸器官进行特征点检测，生成第二人脸特征点，该第二人脸特征点可以是各个人脸器官对应的关键点的位置坐标信息，该第二人脸特征点可以在人脸的外部轮廓和各个人脸器官的边缘或中心等，该第二人脸特征点的个数可以是106个、83个或68个等，具体个数可以根据实际需要进行灵活设置。为了能够将二维人脸图像与模板图像进行精准融合，可以设置第一人脸特征点和第二人脸特征点的个数一致。例如，进行人脸特征点检测得到包含的人脸特征点人脸图像，如图6(a)所示。The second detection sub-unit 3052 can use face recognition technology to detect feature points on each face organ such as eyes, nose, eyebrows, and mouth in the template image to generate a second face feature point. The second person The face feature point may be position coordinate information of a key point corresponding to each face organ, and the second face feature point may be in the outer contour of the face and the edge or center of each face organ, etc. The second face feature point The number can be 106, 83, or 68. The specific number can be flexibly set according to actual needs. In order to be able to accurately fuse the two-dimensional face image and the template image, the number of feature points of the first face and the number of feature points of the second face can be set to be the same. For example, face feature point detection is performed to obtain a face image containing the face feature points, as shown in FIG. 6 (a).
此时，融合子单元3053可以根据第一人脸特征点和第二人脸特征点，将二维人脸图像中的人脸融合至模板图像中的人脸所在区域。At this time, the fusion subunit 3053 can fuse the face in the two-dimensional face image to the area where the face in the template image is located according to the first face feature point and the second face feature point.
在某些实施方式中，融合子单元3053具体可以用于：In some embodiments, the fusion subunit 3053 may be specifically configured to:
根据第一人脸特征点，对二维人脸图像中的人脸进行三角形划分，得到第一人脸三角形区域；根据第二人脸特征点，对模板图像中的人脸进行三角形划分，得到第二人脸三角形区域；基于第一人脸三角形区域和第二人脸三角形区域进行仿射变换，得到目标人脸三角形区域；将目标人脸三角形区域融合至模板图像中的人脸所在区域。According to the first face feature point, the face in the two-dimensional face image is triangle-divided to obtain the first face triangle area; according to the second face feature point, the face in the template image is triangle-divided to obtain The second face triangle area; the affine transformation is performed based on the first face triangle area and the second face triangle area to obtain the target face triangle area; and the target face triangle area is fused to the face area in the template image.
具体地，融合子单元3053可以根据第一人脸特征点对二维人脸图像中的人脸进行三角形划分，得到第一人脸三角形区域，例如，将各个人脸特征点按照预设算法进行连线，从而可以将人脸划分为多个三角形区域，该预设算法可以 根据实际需要进行灵活设置。以及，根据第二人脸特征点对模板图像中的人脸进行三角形划分，得到第二人脸三角形区域，例如，根据人脸特征点对人脸进行三角形划分得到包含人脸三角形区域的人脸图像，如图6(b)所示。Specifically, the fusion sub-unit 3053 may triangulate the face in the two-dimensional face image according to the first face feature point to obtain a first face triangle area. For example, each face feature point is performed according to a preset algorithm. Connect the lines so that the face can be divided into multiple triangular areas. The preset algorithm can be flexibly set according to actual needs. And, the face in the template image is triangle-divided according to the second face feature point to obtain a second face triangle area. For example, the face is triangle-divided according to the face feature point to obtain a face including the face triangle area. The image is shown in Figure 6 (b).
然后，融合子单元3053可以基于二维人脸图像对应的第一人脸三角形区域和模板图像对应的第二人脸三角形区域进行仿射变换，得到目标人脸三角形区域。例如，可以将第一人脸三角形区域和第二人脸三角形区域中对应位置的人脸三角形区域进行比较，若第一人脸三角形区域和第二人脸三角形区域中对应位置的人脸三角形区域的形状和大小等不一致，此时可以对第一人脸三角形区域中该位置的人脸三角形区域进行缩放、旋转或平移等操作，得到目标人脸三角形区域，使得目标人脸三角形区域和第二人脸三角形区域中对应位置的人脸三角形区域形状和大小等一致。此时，可以将目标人脸三角形区域融合至模板图像中的人脸所在区域，得到融合后图像，例如该融合后图像可以如图7所示。Then, the fusion subunit 3053 may perform affine transformation based on the first face triangle area corresponding to the two-dimensional face image and the second face triangle area corresponding to the template image to obtain the target face triangle area. For example, the first face triangle area and the second face face triangle area at the corresponding position of the face triangle area can be compared. If the first face triangle area and the second face face triangle area correspond to the corresponding face triangle area. The shape and size of the target face are inconsistent. At this time, you can perform operations such as zooming, rotating, or panning on the face triangle area at the position in the first face triangle area to obtain the target face triangle area, making the target face triangle area and the second face area The shape and size of the face triangle area at the corresponding position in the face triangle area are the same. At this time, the triangle area of the target face can be fused to the area of the face in the template image to obtain a fused image. For example, the fused image can be shown in FIG. 7.
以下将进行举例说明，当待融合二维图像中的人脸为正脸，且模板图像中的人脸为侧脸时，根据待融合二维图像中的正脸构建三维人脸模型，得到的该三维人脸模型中的人脸为正脸，以及获取模板图像中的侧脸在三维空间中相对于正脸的旋转参数，该旋转参数为模板图像中侧脸在三维空间中的相对于正脸的旋转参数。The following will illustrate by example. When the human face in the two-dimensional image to be fused is a positive face and the human face in the template image is a side face, a three-dimensional human face model is constructed based on the positive face in the two-dimensional image to be fused. The face in the three-dimensional face model is a front face, and a rotation parameter of the side face in the template image relative to the front face in the three-dimensional space is obtained. The rotation parameter is the side face relative to the positive face in the three-dimensional space in the template image. Face rotation parameters.
此时，旋转单元303具体可以用于：基于旋转参数对三维人脸模型中正脸的进行旋转，得到旋转后三维人脸模型，旋转后三维人脸模型中的人脸为侧脸。At this time, the rotation unit 303 may be specifically configured to rotate the positive face in the three-dimensional face model based on the rotation parameters to obtain a rotated three-dimensional face model, and the face in the rotated three-dimensional face model is a side face.
融合单元305具体可以用于：将二维人脸图像中的旋转后侧脸融合至模板图像中的侧脸所在区域。从而通过构建三维人脸模型的方式来解决侧脸融合难的问题，用户只需正常拍摄正脸图像，通过正脸图像构建三维人脸模型，再自动化旋转三维人脸模型到与模板图中侧脸方向一致，最后将旋转后三维人脸模型转换为二维人脸图像，进行二维人脸图像与模板图像之间进行侧脸融合。例如，只需要待融合二维图像中人脸为正脸，就能实现与各个侧脸的模板图进行侧脸融合，方便快捷且效果很好。The fusion unit 305 may be specifically configured to fuse the rotated side face in the two-dimensional face image to the region where the side face is located in the template image. In order to solve the problem of difficult side-face fusion by constructing a three-dimensional face model, the user only needs to take a normal face image normally, construct a three-dimensional face model from the front face image, and then automatically rotate the three-dimensional face model to the side of the template image. The face directions are the same. Finally, the rotated 3D face model is converted into a 2D face image, and a side face fusion is performed between the 2D face image and the template image. For example, as long as the face in the two-dimensional image to be fused is a normal face, side-face fusion with the template image of each side face can be realized, which is convenient, fast, and effective.
由上可知，本申请实施例可以由构建单元301获取待融合二维图像，并根据待融合二维图像中的人脸构建三维人脸模型，以及，由获取单元302获取模板图像及模板图像中的人脸在三维空间中的旋转参数，然后由旋转单元303基 于旋转参数对三维人脸模型进行旋转，得到旋转后三维人脸模型，再由转换单元304将旋转后三维人脸模型转换为二维人脸图像，此时，融合单元305可以将二维人脸图像与模板图像进行融合，例如将二维人脸图像中的人脸融合至模板图像中的人脸所在区域。该方案通过构建并旋转三维人脸模型，从而可以基于旋转后三维人脸模型得到二维人脸图像，使得二维人脸图像与模板图像中人脸的方向一致，以便将二维人脸图像中的人脸精准融合至模板图像中的人脸所在区域，提高了对图像中人脸进行融合的效果。As can be seen from the above, in the embodiment of the present application, the construction unit 301 can obtain the two-dimensional image to be fused, and the three-dimensional face model is constructed based on the face in the two-dimensional image to be fused, and the template image and the template image are obtained by the acquisition unit 302. The rotation parameters of the human face in the three-dimensional space, and then the rotation unit 303 rotates the three-dimensional face model based on the rotation parameters to obtain a rotated three-dimensional face model, and the conversion unit 304 converts the rotated three-dimensional face model into two Dimension face image, at this time, the fusion unit 305 may fuse the two-dimensional face image with the template image, for example, fuse the face in the two-dimensional face image to the area where the face in the template image is located. This solution constructs and rotates a three-dimensional face model, so that a two-dimensional face image can be obtained based on the rotated three-dimensional face model, so that the two-dimensional face image is consistent with the direction of the face in the template image, so that the two-dimensional face image The face in the image is accurately fused to the area where the face in the template image is located, which improves the effect of merging the face in the image.
相应的，本申请实施例还提供一种终端，如图11所示，该终端可以包括射频(RF，Radio Frequency)电路601、包括有一个或一个以上计算机可读存储介质的存储器602、输入单元603、显示单元604、传感器605、音频电路606、无线保真(WiFi，Wireless Fidelity)模块607、包括有一个或者一个以上处理核心的处理器608、以及电源609等部件。本领域技术人员可以理解，图11中示出的终端结构并不构成对终端的限定，可以包括比图示更多或更少的部件，或者组合某些部件，或者不同的部件布置。其中：Correspondingly, an embodiment of the present application further provides a terminal. As shown in FIG. 11, the terminal may include a radio frequency (RF) circuit 601, a memory 602 including one or more computer-readable storage media, and an input unit. 603, a display unit 604, a sensor 605, an audio circuit 606, a wireless fidelity (WiFi) module 607, a processor 608 including one or more processing cores, a power supply 609, and other components. Those skilled in the art may understand that the terminal structure shown in FIG. 11 does not constitute a limitation on the terminal, and may include more or fewer components than shown in the figure, or some components may be combined, or different component arrangements. among them:
RF电路601可用于收发信息或通话过程中，信号的接收和发送，特别地，将基站的下行信息接收后，交由一个或者一个以上处理器608处理；另外，将涉及上行的数据发送给基站。通常，RF电路601包括但不限于天线、至少一个放大器、调谐器、一个或多个振荡器、用户身份模块(SIM，Subscriber Identity Module)卡、收发信机、耦合器、低噪声放大器(LNA，Low Noise Amplifier)、双工器等。此外，RF电路601还可以通过无线通信与网络和其他设备通信。所述无线通信可以使用任一通信标准或协议，包括但不限于全球移动通讯***(GSM，Global System of Mobile communication)、通用分组无线服务(GPRS，General Packet Radio Service)、码分多址(CDMA，Code Division Multiple Access)、宽带码分多址(WCDMA，Wideband Code Division Multiple Access)、长期演进(LTE，Long Term Evolution)、电子邮件、短消息服务(SMS，Short Messaging Service)等。The RF circuit 601 can be used to receive and send signals during the transmission and reception of information or during a call. In particular, after receiving the downlink information of the base station, it is processed by one or more processors 608. In addition, the uplink-related data is sent to the base station. . Generally, the RF circuit 601 includes, but is not limited to, an antenna, at least one amplifier, a tuner, one or more oscillators, a subscriber identity module (SIM, Subscriber Identity Module) card, a transceiver, a coupler, and a low noise amplifier (LNA, Low Noise Amplifier), duplexer, etc. In addition, the RF circuit 601 can also communicate with a network and other devices through wireless communication. The wireless communication can use any communication standard or protocol, including but not limited to Global System for Mobile Communication (GSM), General Packet Radio Service (GPRS), and Code Division Multiple Access (CDMA) , Code Division Multiple Access), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), E-mail, Short Messaging Service (SMS), etc.
存储器602可用于存储软件程序以及模块，处理器608通过运行存储在存储器602的软件程序以及模块，从而执行各种功能应用以及数据处理。存储器602可主要包括存储程序区和存储数据区，其中，存储程序区可存储操作***、至少一个功能所需的应用程序(比如声音播放功能、图像播放功能等)等；存储数据区可存储根据终端的使用所创建的数据(比如音频数据、电话本 等)等。此外，存储器602可以包括高速随机存取存储器，还可以包括非易失性存储器，例如至少一个磁盘存储器件、闪存器件、或其他易失性固态存储器件。相应地，存储器602还可以包括存储器控制器，以提供处理器608和输入单元603对存储器602的访问。The memory 602 may be used to store software programs and modules, and the processor 608 executes various functional applications and data processing by running the software programs and modules stored in the memory 602. The memory 602 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, at least one application required by a function (such as a sound playback function, an image playback function, etc.), etc .; the storage data area may store data according to Use data created by the terminal (such as audio data, phone book, etc.). In addition, the memory 602 may include a high-speed random access memory, and may further include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 602 may further include a memory controller to provide the processor 608 and the input unit 603 to access the memory 602.
输入单元603可用于接收输入的数字或字符信息，以及产生与用户设置以及功能控制有关的键盘、鼠标、操作杆、光学或者轨迹球信号输入。具体地，在一个具体的实施例中，输入单元603可包括触敏表面以及其他输入设备。触敏表面，也称为触摸显示屏或者触控板，可收集用户在其上或附近的触摸操作(比如用户使用手指、触笔等任何适合的物体或附件在触敏表面上或在触敏表面附近的操作)，并根据预先设定的程式驱动相应的连接装置。可选的，触敏表面可包括触摸检测装置和触摸控制器两个部分。其中，触摸检测装置检测用户的触摸方位，并检测触摸操作带来的信号，将信号传送给触摸控制器；触摸控制器从触摸检测装置上接收触摸信息，并将它转换成触点坐标，再送给处理器608，并能接收处理器608发来的命令并加以执行。此外，可以采用电阻式、电容式、红外线以及表面声波等多种类型实现触敏表面。除了触敏表面，输入单元603还可以包括其他输入设备。具体地，其他输入设备可以包括但不限于物理键盘、功能键(比如音量控制按键、开关按键等)、轨迹球、鼠标、操作杆等中的一种或多种。The input unit 603 may be used to receive inputted numeric or character information, and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control. Specifically, in a specific embodiment, the input unit 603 may include a touch-sensitive surface and other input devices. A touch-sensitive surface, also known as a touch display or touchpad, collects user touch operations on or near it (such as the user using a finger, stylus or any suitable object or accessory on the touch-sensitive surface or touch-sensitive Operation near the surface), and drive the corresponding connection device according to a preset program. Optionally, the touch-sensitive surface may include two parts, a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch position, and detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives touch information from the touch detection device, converts it into contact coordinates, and sends it To the processor 608, and can receive the command sent by the processor 608 and execute it. In addition, various types such as resistive, capacitive, infrared, and surface acoustic waves can be used to implement a touch-sensitive surface. In addition to the touch-sensitive surface, the input unit 603 may include other input devices. Specifically, other input devices may include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, and the like.
显示单元604可用于显示由用户输入的信息或提供给用户的信息以及终端的各种图形用户接口，这些图形用户接口可以由图形、文本、图标、视频和其任意组合来构成。显示单元604可包括显示面板，可选的，可以采用液晶显示器(LCD，Liquid Crystal Display)、有机发光二极管(OLED，Organic Light-Emitting Diode)等形式来配置显示面板。进一步的，触敏表面可覆盖显示面板，当触敏表面检测到在其上或附近的触摸操作后，传送给处理器608以确定触摸事件的类型，随后处理器608根据触摸事件的类型在显示面板上提供相应的视觉输出。虽然在图11中，触敏表面与显示面板是作为两个独立的部件来实现输入和输入功能，但是在某些实施例中，可以将触敏表面与显示面板集成而实现输入和输出功能。The display unit 604 may be used to display information input by the user or information provided to the user and various graphical user interfaces of the terminal. These graphical user interfaces may be composed of graphics, text, icons, videos, and any combination thereof. The display unit 604 may include a display panel. Optionally, the display panel may be configured in the form of a liquid crystal display (LCD, Liquid Crystal Display), an organic light emitting diode (OLED, Organic Light-Emitting Diode), or the like. Further, the touch-sensitive surface may cover the display panel. When the touch-sensitive surface detects a touch operation on or near the touch-sensitive surface, the touch-sensitive surface is transmitted to the processor 608 to determine the type of the touch event, and then the processor 608 displays the touch event according to the type of the touch event. The corresponding visual output is provided on the panel. Although in FIG. 11, the touch-sensitive surface and the display panel are implemented as two separate components to implement input and input functions, in some embodiments, the touch-sensitive surface and the display panel may be integrated to implement input and output functions.
终端还可包括至少一种传感器605，比如光传感器、运动传感器以及其他传感器。具体地，光传感器可包括环境光传感器及接近传感器，其中，环境光传感器可根据环境光线的明暗来调节显示面板的亮度，接近传感器可在终端移 动到耳边时，关闭显示面板和/或背光。作为运动传感器的一种，重力加速度传感器可检测各个方向上(一般为三轴)加速度的大小，静止时可检测出重力的大小及方向，可用于识别终端姿态的应用(比如横竖屏切换、相关游戏、磁力计姿态校准)、振动识别相关功能(比如计步器、敲击)等；至于终端还可配置的陀螺仪、气压计、湿度计、温度计、红外线传感器等其他传感器，在此不再赘述。The terminal may further include at least one sensor 605, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor. The ambient light sensor may adjust the brightness of the display panel according to the brightness of the ambient light. The proximity sensor may turn off the display panel and / or the backlight when the terminal is moved to the ear. . As a type of motion sensor, a gravity acceleration sensor can detect the magnitude of acceleration in various directions (generally three axes). It can detect the magnitude and direction of gravity when it is stationary. It can be used to identify the posture of the terminal (such as horizontal and vertical screen switching, related Games, magnetometer attitude calibration), vibration recognition related functions (such as pedometer, tap), etc .; as for the terminal, other sensors such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc. To repeat.
音频电路606、扬声器，传声器可提供用户与终端之间的音频接口。音频电路606可将接收到的音频数据转换后的电信号，传输到扬声器，由扬声器转换为声音信号输出；另一方面，传声器将收集的声音信号转换为电信号，由音频电路606接收后转换为音频数据，再将音频数据输出处理器608处理后，经RF电路601以发送给比如另一终端，或者将音频数据输出至存储器602以便进一步处理。音频电路606还可能包括耳塞插孔，以提供外设耳机与终端的通信。The audio circuit 606, a speaker, and a microphone can provide an audio interface between the user and the terminal. The audio circuit 606 may transmit the received electrical data converted electrical signals to a speaker, and the speakers convert the sound signals into sound signals for output. On the other hand, the microphone converts the collected sound signals into electrical signals, which are converted by the audio circuit 606 after being received. The audio data is processed by the audio data output processor 608 and then sent to, for example, another terminal via the RF circuit 601, or the audio data is output to the memory 602 for further processing. The audio circuit 606 may further include an earphone jack to provide communication between a peripheral headset and a terminal.
WiFi属于短距离无线传输技术，终端通过WiFi模块607可以帮助用户收发电子邮件、浏览网页和访问流式媒体等，它为用户提供了无线的宽带互联网访问。虽然图11示出了WiFi模块607，但是可以理解的是，其并不属于终端的必须构成，完全可以根据需要在不改变发明的本质的范围内而省略。WiFi is a short-range wireless transmission technology. The terminal can help users send and receive email, browse web pages, and access streaming media through the WiFi module 607. It provides users with wireless broadband Internet access. Although FIG. 11 shows the WiFi module 607, it can be understood that it does not belong to the necessary configuration of the terminal, and can be omitted as needed without changing the essence of the invention.
处理器608是终端的控制中心，利用各种接口和线路连接整个终端的各个部分，通过运行或执行存储在存储器602内的软件程序和/或模块，以及调用存储在存储器602内的数据，执行终端的各种功能和处理数据，从而对终端进行整体监控。可选的，处理器608可包括一个或多个处理核心；优选的，处理器608可集成应用处理器和调制解调处理器，其中，应用处理器主要处理操作***、用户界面和应用程序等，调制解调处理器主要处理无线通信。可以理解的是，上述调制解调处理器也可以不集成到处理器608中。The processor 608 is a control center of the terminal, and uses various interfaces and lines to connect various parts of the entire terminal. By running or executing software programs and / or modules stored in the memory 602, and calling data stored in the memory 602, execution is performed. Various functions and processing data of the terminal, so as to monitor the terminal as a whole. Optionally, the processor 608 may include one or more processing cores; preferably, the processor 608 may integrate an application processor and a modem processor, wherein the application processor mainly processes an operating system, a user interface, and an application program, etc. The modem processor mainly handles wireless communication. It can be understood that the foregoing modem processor may not be integrated into the processor 608.
终端还包括给各个部件供电的电源609(比如电池)，优选的，电源可以通过电源管理***与处理器608逻辑相连，从而通过电源管理***实现管理充电、放电、以及功耗管理等功能。电源609还可以包括一个或一个以上的直流或交流电源、再充电***、电源故障检测电路、电源转换器或者逆变器、电源状态指示器等任意组件。The terminal also includes a power source 609 (such as a battery) for supplying power to various components. Preferably, the power source can be logically connected to the processor 608 through a power management system, so as to implement functions such as managing charging, discharging, and power consumption management through the power management system. The power source 609 may further include any one or more DC or AC power sources, a recharging system, a power failure detection circuit, a power converter or inverter, and a power source status indicator.
尽管未示出，终端还可以包括摄像头、蓝牙模块等，在此不再赘述。具体在本实施例中，终端中的处理器608会按照如下的指令，将一个或一个以上的 应用程序的进程对应的可执行文件加载到存储器602中，并由处理器608来运行存储在存储器602中的应用程序，从而实现各种功能：Although not shown, the terminal may further include a camera, a Bluetooth module, and the like, and details are not described herein again. Specifically, in this embodiment, the processor 608 in the terminal loads the executable files corresponding to one or more application processes into the memory 602 according to the following instructions, and the processor 608 runs the stored files in the memory. 602 applications to achieve various functions:
获取待融合二维图像，并根据待融合二维图像中的人脸构建三维人脸模型；获取模板图像，以及获取模板图像中的人脸在三维空间中的旋转参数；基于旋转参数对三维人脸模型进行旋转，得到旋转后三维人脸模型；将旋转后人脸三维模型转换为二维人脸图像；将二维人脸图像与模板图像进行融合。Obtain the two-dimensional image to be fused, and construct a three-dimensional face model based on the faces in the two-dimensional image to be fused; obtain the template image, and obtain the rotation parameters of the face in the template image in the three-dimensional space; The face model is rotated to obtain a rotated 3D face model; the rotated 3D model is converted into a 2D face image; and the 2D face image is fused with a template image.
在上述实施例中，对各个实施例的描述都各有侧重，某个实施例中没有详述的部分，可以参见上文针对图像融合方法的详细描述，此处不再赘述。In the above embodiments, the description of each embodiment has its own emphasis. For the parts that are not described in detail in an embodiment, refer to the detailed description of the image fusion method above, which will not be repeated here.
由上可知，本申请实施例可以获取待融合二维图像，并根据待融合二维图像中的人脸构建三维人脸模型，以及，获取模板图像及模板图像中的人脸在三维空间中的旋转参数，然后基于旋转参数对三维人脸模型进行旋转，得到旋转后三维人脸模型，再将旋转后三维人脸模型转换为二维人脸图像，此时，可以将二维人脸图像与模板图像进行融合，例如将二维人脸图像中的人脸融合至模板图像中的人脸所在区域。该方案通过构建并旋转三维人脸模型，从而可以基于旋转后三维人脸模型得到二维人脸图像，使得二维人脸图像与模板图像中人脸的方向一致，以便将二维人脸图像中的人脸精准融合至模板图像中的人脸所在区域，提高了对图像中人脸进行融合的效果。It can be known from the above that the embodiment of the present application can obtain a two-dimensional image to be fused, and construct a three-dimensional face model based on the human face in the two-dimensional image to be fused, and obtain the template image and the human face in the template image in the three-dimensional space. Rotation parameters, and then rotate the 3D face model based on the rotation parameters to obtain a rotated 3D face model, and then convert the rotated 3D face model into a 2D face image. At this time, the 2D face image and The template image is fused, for example, the face in the two-dimensional face image is fused to the area where the face in the template image is located. This solution constructs and rotates a three-dimensional face model, so that a two-dimensional face image can be obtained based on the rotated three-dimensional face model, so that the two-dimensional face image is consistent with the direction of the face in the template image, so that the two-dimensional face image The face in the image is accurately fused to the area where the face in the template image is located, which improves the effect of merging the face in the image.
上述实施例的各种方法中的全部或部分步骤可以通过指令来完成，或通过指令控制相关的硬件来完成，该指令可以存储于一计算机可读存储介质中，并由处理器进行加载和执行。All or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware by instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor. .
为此，本申请实施例提供一种存储介质，其中存储有多条指令，该指令能够被处理器进行加载，以执行本申请实施例所提供的任一种图像融合方法中的步骤。例如，该指令可以执行如下步骤：To this end, an embodiment of the present application provides a storage medium in which a plurality of instructions are stored, and the instructions can be loaded by a processor to execute the steps in any one of the image fusion methods provided in the embodiments of the present application. For example, the instruction can perform the following steps:
获取待融合二维图像，并根据待融合二维图像中的人脸构建三维人脸模型；获取模板图像，以及获取模板图像中的人脸在三维空间中的旋转参数；基于旋转参数对三维人脸模型进行旋转，得到旋转后三维人脸模型；将旋转后人脸三维模型转换为二维人脸图像；将二维人脸图像与模板图像进行融合。Obtain the two-dimensional image to be fused, and construct a three-dimensional face model based on the faces in the two-dimensional image to be fused; obtain the template image, and obtain the rotation parameters of the face in the template image in the three-dimensional space; The face model is rotated to obtain a rotated 3D face model; the rotated 3D model is converted into a 2D face image; and the 2D face image is fused with a template image.
以上各个操作的具体实施可参见前面的实施例，在此不再赘述。For specific implementation of the foregoing operations, refer to the foregoing embodiments, and details are not described herein again.
其中，该存储介质可以包括：只读存储器(ROM，Read Only Memory)、随机存取记忆体(RAM，Random Access Memory)、磁盘或光盘等。The storage medium may include a read-only memory (ROM, Read Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk, or an optical disk.
由于该存储介质中所存储的指令，可以执行本申请实施例所提供的任一种图像融合方法中的步骤，因此，可以实现本申请实施例所提供的任一种图像融合方法所能实现的有益效果，详见前面的实施例，在此不再赘述。Since the instructions stored in the storage medium can execute the steps in any one of the image fusion methods provided in the embodiments of the present application, it can implement what can be achieved by any of the image fusion methods provided in the embodiments of the present application. For the beneficial effects, refer to the foregoing embodiments for details, and details are not described herein again.
以上对本申请实施例所提供的一种图像融合方法、装置及存储介质进行了详细介绍，本文中应用了具体个例对本申请的原理及实施方式进行了阐述，以上实施例的说明只是用于帮助理解本申请的方法及其核心思想；同时，对于本领域的技术人员，依据本申请的思想，在具体实施方式及应用范围上均会有改变之处，综上所述，本说明书内容不应理解为对本申请的限制。The image fusion method, device, and storage medium provided in the embodiments of the present application have been described in detail above. Specific examples have been used in this document to explain the principles and implementation of the present application. The descriptions of the above embodiments are only used to help Understand the methods and core ideas of this application; at the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation and application scope. In summary, the contents of this description should not be Understood as a limitation on this application.

Claims (15)

  1. 一种图像融合方法，由计算设备执行，其特征在于，包括：An image fusion method, executed by a computing device, is characterized in that it includes:
    获取待融合二维图像，并根据所述待融合二维图像中的人脸构建三维人脸模型；Acquiring a two-dimensional image to be fused, and constructing a three-dimensional face model according to the human face in the two-dimensional image to be fused;
    获取模板图像，以及获取所述模板图像中的人脸在三维空间中的旋转参数；Acquiring a template image, and acquiring a rotation parameter of a face in the template image in a three-dimensional space;
    基于所述旋转参数对所述三维人脸模型进行旋转，得到旋转后三维人脸模型；Rotating the three-dimensional face model based on the rotation parameters to obtain a rotated three-dimensional face model;
    将所述旋转后人脸三维模型转换为二维人脸图像；Converting the three-dimensional model of the rotated face into a two-dimensional face image;
    将所述二维人脸图像与所述模板图像进行融合。Fusion the two-dimensional face image and the template image.
  2. 根据权利要求1所述的图像融合方法，其特征在于，所述根据所述待融合二维图像中的人脸构建三维人脸模型的步骤包括：The image fusion method according to claim 1, wherein the step of constructing a three-dimensional face model based on the faces in the two-dimensional image to be fused comprises:
    获取预设人脸模型；Obtaining a preset face model;
    根据所述预设人脸模型，获取所述待融合二维图像中的人脸在三维平面内的三维坐标点，并根据所述三维坐标点构建三维人脸模型。According to the preset face model, a three-dimensional coordinate point of a face in the two-dimensional image to be fused in a three-dimensional plane is obtained, and a three-dimensional face model is constructed according to the three-dimensional coordinate point.
  3. 根据权利要求2所述的图像融合方法，其特征在于，所述根据所述预设人脸模型，获取所述待融合二维图像中的人脸在三维平面内的三维坐标点，并根据所述三维坐标点构建三维人脸模型的步骤包括：The image fusion method according to claim 2, wherein the three-dimensional coordinate points of the face in the two-dimensional image to be fused in a three-dimensional plane are obtained according to the preset face model, and The steps of constructing a three-dimensional face model by describing the three-dimensional coordinate points include:
    获取所述待融合二维图像中的人脸投影在二维平面内的第一投影参数；Acquiring a first projection parameter of the face projection in the two-dimensional image to be fused in a two-dimensional plane;
    获取所述预设人脸模型投影在二维平面内的第二投影参数；Acquiring a second projection parameter of the preset face model projected in a two-dimensional plane;
    根据所述第一投影参数及第二投影参数，获取所述待融合二维图像中的人脸在三维平面内的三维坐标点；Obtaining, according to the first projection parameter and the second projection parameter, a three-dimensional coordinate point of a human face in a two-dimensional image to be fused in a three-dimensional plane;
    根据所述三维坐标点构建三维人脸模型。A three-dimensional human face model is constructed according to the three-dimensional coordinate points.
  4. 根据权利要求1所述的图像融合方法，其特征在于，所述获取所述模板图像中的人脸在三维空间中的旋转参数的步骤包括：The image fusion method according to claim 1, wherein the step of obtaining a rotation parameter of a face in the template image in a three-dimensional space comprises:
    根据所述模板图像中的人脸构建三维模型；Constructing a three-dimensional model according to a human face in the template image;
    基于所述三维模型获取所述模板图像中人脸在三维空间中的旋转参数。A rotation parameter of a face in the template image in a three-dimensional space is obtained based on the three-dimensional model.
  5. 根据权利要求4所述的图像融合方法，其特征在于，所述根据所述模板图像中的人脸构建三维模型的步骤包括：The image fusion method according to claim 4, wherein the step of constructing a three-dimensional model based on a face in the template image comprises:
    获取所述模板图像中的人脸投影在二维平面内的第三投影参数；Acquiring a third projection parameter of the face projection in the template image in a two-dimensional plane;
    获取预设人脸模型，及所述预设人脸模型投影在二维平面内的第四投影参数；Obtaining a preset face model and a fourth projection parameter of the preset face model projected in a two-dimensional plane;
    根据所述第三投影参数及第四投影参数，获取模板图像中的人脸在三维平面内的目标三维坐标点；Obtaining, according to the third projection parameter and the fourth projection parameter, a target three-dimensional coordinate point of a human face in a template image in a three-dimensional plane;
    根据所述目标三维坐标点构建三维模型。A three-dimensional model is constructed according to the target three-dimensional coordinate points.
  6. 根据权利要求1至5任一项所述的图像融合方法，其特征在于，所述将所述二维人脸图像与所述模板图像进行融合的步骤包括：The image fusion method according to any one of claims 1 to 5, wherein the step of fusing the two-dimensional face image with the template image comprises:
    对所述二维人脸图像进行人脸特征点检测，得到第一人脸特征点；Performing face feature point detection on the two-dimensional face image to obtain a first face feature point;
    对所述模板图像进行人脸特征点检测，得到第二人脸特征点；Performing face feature point detection on the template image to obtain a second face feature point;
    根据所述第一人脸特征点和第二人脸特征点，将所述二维人脸图像中的人脸融合至所述模板图像中的人脸所在区域。According to the first facial feature point and the second facial feature point, the face in the two-dimensional face image is fused to a region where the face in the template image is located.
  7. 根据权利要求6所述的图像融合方法，其特征在于，所述根据所述第一人脸特征点和第二人脸特征点，将所述二维人脸图像中的人脸融合至所述模板图像中的人脸所在区域的步骤包括：The image fusion method according to claim 6, wherein, according to the first face feature points and the second face feature points, a face in the two-dimensional face image is fused to the face The steps for the area where the face is in the template image include:
    根据所述第一人脸特征点，对所述二维人脸图像中的人脸进行三角形划分，得到第一人脸三角形区域；Triangulate a face in the two-dimensional face image according to the first face feature point to obtain a first face triangle area;
    根据所述第二人脸特征点，对所述模板图像中的人脸进行三角形划分，得到第二人脸三角形区域；Triangulate the face in the template image according to the second face feature point to obtain a second face triangle area;
    基于所述第一人脸三角形区域和第二人脸三角形区域进行仿射变换，得到目标人脸三角形区域；Performing an affine transformation based on the first face triangle area and the second face triangle area to obtain a target face triangle area;
    将所述目标人脸三角形区域融合至所述模板图像中的人脸所在区域。The triangle region of the target face is fused to the region of the face in the template image.
  8. 根据权利要求1至5任一项所述的图像融合方法，其特征在于，当所述待融合二维图像中的人脸为正脸，且所述模板图像中的人脸为侧脸时，所述旋转参数为所述模板图像中侧脸在三维空间中的相对于正脸的旋转参数，所述基于所述旋转参数对所述三维人脸模型进行旋转，得到旋转后三维人脸模型的步骤包括：The image fusion method according to any one of claims 1 to 5, wherein when a human face in the two-dimensional image to be fused is a front face, and a human face in the template image is a side face, The rotation parameter is a rotation parameter of the side face relative to the front face in the three-dimensional space in the template image, and the three-dimensional face model is rotated based on the rotation parameter to obtain a rotated three-dimensional face model. The steps include:
    基于所述旋转参数对所述三维人脸模型中正脸的进行旋转，得到旋转后三维人脸模型，所述旋转后三维人脸模型中的人脸为侧脸；Rotating the positive face in the three-dimensional face model based on the rotation parameter to obtain a rotated three-dimensional face model, and the face in the rotated three-dimensional face model is a side face;
    所述将所述二维人脸图像与所述模板图像进行融合的步骤包括：The step of fusing the two-dimensional face image with the template image includes:
    将所述二维人脸图像中的旋转后侧脸融合至所述模板图像中的侧脸所在区域。Fusion the rotated side face in the two-dimensional face image to the area where the side face is located in the template image.
  9. 一种图像融合装置，其特征在于，包括：An image fusion device, comprising:
    构建单元，用于获取待融合二维图像，并根据所述待融合二维图像中的人脸构建三维人脸模型；A construction unit, configured to obtain a two-dimensional image to be fused, and construct a three-dimensional face model according to a face in the two-dimensional image to be fused;
    获取单元，用于获取模板图像，以及获取所述模板图像中的人脸在三维空间中的旋转参数；An obtaining unit, configured to obtain a template image, and obtain a rotation parameter of a face in the template image in a three-dimensional space;
    旋转单元，用于基于所述旋转参数对所述三维人脸模型进行旋转，得到旋转后三维人脸模型；A rotation unit, configured to rotate the three-dimensional face model based on the rotation parameters to obtain a rotated three-dimensional face model;
    转换单元，用于将所述旋转后人脸三维模型转换为二维人脸图像；A converting unit, configured to convert the three-dimensional model of the rotated face into a two-dimensional face image;
    融合单元，用于将所述二维人脸图像与所述模板图像进行融合。A fusion unit is configured to fuse the two-dimensional face image and the template image.
  10. 根据权利要求9所述的图像融合装置，其特征在于，所述构建单元包括：The image fusion device according to claim 9, wherein the construction unit comprises:
    第一获取子单元，用于获取预设人脸模型；A first acquisition subunit, configured to acquire a preset face model;
    第一构建子单元，用于根据所述预设人脸模型，获取所述待融合二维图像中的人脸在三维平面内的三维坐标点，并根据所述三维坐标点构建三维人脸模型。A first construction subunit, configured to obtain a three-dimensional coordinate point of a face in a two-dimensional image to be fused in a three-dimensional plane according to the preset face model, and construct a three-dimensional face model according to the three-dimensional coordinate point .
  11. 根据权利要求10所述的图像融合装置，其特征在于，所述第一构建子单元具体用于：The image fusion apparatus according to claim 10, wherein the first construction subunit is specifically configured to:
    获取所述待融合二维图像中的人脸投影在二维平面内的第一投影参数；Acquiring a first projection parameter of a face projection in the two-dimensional image to be fused in a two-dimensional plane;
    获取所述预设人脸模型投影在二维平面内的第二投影参数；Acquiring a second projection parameter of the preset face model projected in a two-dimensional plane;
    根据所述第一投影参数及第二投影参数，获取所述待融合二维图像中的人脸在三维平面内的三维坐标点；Obtaining, according to the first projection parameter and the second projection parameter, a three-dimensional coordinate point of a human face in a two-dimensional image to be fused in a three-dimensional plane;
    根据所述三维坐标点构建三维人脸模型。A three-dimensional human face model is constructed according to the three-dimensional coordinate points.
  12. 根据权利要求9所述的图像融合装置，其特征在于，所述获取单元包括：The image fusion apparatus according to claim 9, wherein the obtaining unit comprises:
    第二构建子单元，用于根据所述模板图像中的人脸构建三维模型；A second construction subunit, configured to construct a three-dimensional model according to a face in the template image;
    第二获取子单元，用于基于所述三维模型获取所述模板图像中人脸在三维空间中的旋转参数。A second acquisition subunit is configured to acquire a rotation parameter of a face in the template image in a three-dimensional space based on the three-dimensional model.
  13. 根据权利要求9至12任一项所述的图像融合装置，其特征在于，所述融合单元包括：The image fusion device according to any one of claims 9 to 12, wherein the fusion unit includes:
    第一检测子单元，用于对所述二维人脸图像进行人脸特征点检测，得到第一人脸特征点；A first detection subunit, configured to perform face feature point detection on the two-dimensional face image to obtain a first face feature point;
    第二检测子单元，用于对所述模板图像进行人脸特征点检测，得到第二人脸特征点；A second detection subunit, configured to detect facial feature points on the template image to obtain a second facial feature point;
    融合子单元，用于根据所述第一人脸特征点和第二人脸特征点，将所述二维人脸图像中的人脸融合至所述模板图像中的人脸所在区域。A fusion subunit is configured to fuse a face in the two-dimensional face image to a region where a face in the template image is located according to the first face feature point and the second face feature point.
  14. 根据权利要求13所述的图像融合装置，其特征在于，所述融合子单元具体用于：The image fusion device according to claim 13, wherein the fusion subunit is specifically configured to:
    根据所述第一人脸特征点，对所述二维人脸图像中的人脸进行三角形划分，得到第一人脸三角形区域；Triangulate a face in the two-dimensional face image according to the first face feature point to obtain a first face triangle area;
    根据所述第二人脸特征点，对所述模板图像中的人脸进行三角形划分，得到第二人脸三角形区域；Triangulate the face in the template image according to the second face feature point to obtain a second face triangle area;
    基于所述第一人脸三角形区域和第二人脸三角形区域进行仿射变换，得到目标人脸三角形区域；Performing an affine transformation based on the first face triangle area and the second face triangle area to obtain a target face triangle area;
    将所述目标人脸三角形区域融合至所述模板图像中的人脸所在区域。The triangle region of the target face is fused to the region of the face in the template image.
  15. 一种存储介质，其特征在于，所述存储介质存储有多条指令，所述指令适于处理器进行加载，以执行权利要求1至8任一项所述的图像融合方法中的步骤。A storage medium, wherein the storage medium stores a plurality of instructions, and the instructions are adapted to be loaded by a processor to execute the steps in the image fusion method according to any one of claims 1 to 8.
PCT/CN2019/085841 2018-06-04 2019-05-07 Image fusion method, apparatus, and storage medium WO2019233229A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201810563880.1A CN108921795A (en) 2018-06-04 2018-06-04 A kind of image interfusion method, device and storage medium
CN201810563880.1 2018-06-04

Publications (1)

Publication Number Publication Date
WO2019233229A1 true WO2019233229A1 (en) 2019-12-12

Family

ID=64418246

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/085841 WO2019233229A1 (en) 2018-06-04 2019-05-07 Image fusion method, apparatus, and storage medium

Country Status (2)

Country Link
CN (1) CN108921795A (en)
WO (1) WO2019233229A1 (en)

Families Citing this family (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108921795A (en) * 2018-06-04 2018-11-30 腾讯科技（深圳）有限公司 A kind of image interfusion method, device and storage medium
CN109816628B (en) * 2018-12-20 2021-09-14 深圳云天励飞技术有限公司 Face evaluation method and related product
CN109767487A (en) * 2019-01-04 2019-05-17 北京达佳互联信息技术有限公司 Face three-dimensional rebuilding method, device, electronic equipment and storage medium
CN109948526B (en) * 2019-03-18 2021-10-29 北京市商汤科技开发有限公司 Image processing method and device, detection equipment and storage medium
CN110189248B (en) * 2019-05-16 2023-05-02 腾讯科技（深圳）有限公司 Image fusion method and device, storage medium and electronic equipment
CN110232730B (en) * 2019-06-03 2024-01-19 深圳市三维人工智能科技有限公司 Three-dimensional face model mapping fusion method and computer processing equipment
CN111008935B (en) * 2019-11-01 2023-12-12 北京迈格威科技有限公司 Face image enhancement method, device, system and storage medium
CN113301242A (en) * 2020-02-21 2021-08-24 中兴通讯股份有限公司 Photographing method, terminal, and computer-readable storage medium
CN113810626B (en) * 2020-06-15 2024-04-05 浙江宇视科技有限公司 Video fusion method, device, equipment and storage medium based on three-dimensional map
CN112257657B (en) * 2020-11-11 2024-02-27 网易（杭州）网络有限公司 Face image fusion method and device, storage medium and electronic equipment
CN112887796B (en) * 2021-02-10 2022-07-22 北京字跳网络技术有限公司 Video generation method, device, equipment and medium
CN115205169A (en) * 2021-04-08 2022-10-18 北京字跳网络技术有限公司 Image processing method, image processing device, electronic equipment and storage medium
CN113223137B (en) * 2021-05-13 2023-03-24 广州虎牙科技有限公司 Generation method and device of perspective projection human face point cloud image and electronic equipment
CN114677572B (en) * 2022-04-08 2023-04-18 北京百度网讯科技有限公司 Object description parameter generation method and deep learning model training method
CN115239576B (en) * 2022-06-15 2023-08-04 荣耀终端有限公司 Photo optimization method, electronic equipment and storage medium

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106023063A (en) * 2016-05-09 2016-10-12 西安北升信息科技有限公司 Video transplantation face changing method
CN106067190A (en) * 2016-05-27 2016-11-02 俞怡斐 A kind of fast face threedimensional model based on single image generates and alternative approach
US20160335774A1 (en) * 2015-02-06 2016-11-17 Ming Chuan University Method for automatic video face replacement by using a 2d face image to estimate a 3d vector angle of the face image
CN106599817A (en) * 2016-12-07 2017-04-26 腾讯科技（深圳）有限公司 Face replacement method and device
CN107146199A (en) * 2017-05-02 2017-09-08 厦门美图之家科技有限公司 A kind of fusion method of facial image, device and computing device
CN107316020A (en) * 2017-06-26 2017-11-03 司马大大（北京）智能***有限公司 Face replacement method, device and electronic equipment
CN108921795A (en) * 2018-06-04 2018-11-30 腾讯科技（深圳）有限公司 A kind of image interfusion method, device and storage medium

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104657974A (en) * 2013-11-25 2015-05-27 腾讯科技（上海）有限公司 Image processing method and device
CN104484858B (en) * 2014-12-31 2018-05-08 小米科技有限责任公司 Character image processing method and processing device
CN105069746B (en) * 2015-08-23 2018-02-16 杭州欣禾圣世科技有限公司 Video real-time face replacement method and its system based on local affine invariant and color transfer technology

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160335774A1 (en) * 2015-02-06 2016-11-17 Ming Chuan University Method for automatic video face replacement by using a 2d face image to estimate a 3d vector angle of the face image
CN106023063A (en) * 2016-05-09 2016-10-12 西安北升信息科技有限公司 Video transplantation face changing method
CN106067190A (en) * 2016-05-27 2016-11-02 俞怡斐 A kind of fast face threedimensional model based on single image generates and alternative approach
CN106599817A (en) * 2016-12-07 2017-04-26 腾讯科技（深圳）有限公司 Face replacement method and device
CN107146199A (en) * 2017-05-02 2017-09-08 厦门美图之家科技有限公司 A kind of fusion method of facial image, device and computing device
CN107316020A (en) * 2017-06-26 2017-11-03 司马大大（北京）智能***有限公司 Face replacement method, device and electronic equipment
CN108921795A (en) * 2018-06-04 2018-11-30 腾讯科技（深圳）有限公司 A kind of image interfusion method, device and storage medium

Also Published As

Publication number Publication date
CN108921795A (en) 2018-11-30

Similar Documents

Publication Publication Date Title
WO2019233229A1 (en) Image fusion method, apparatus, and storage medium
TWI785258B (en) Method, device, and storage medium for processing human face image
US10055879B2 (en) 3D human face reconstruction method, apparatus and server
WO2019184889A1 (en) Method and apparatus for adjusting augmented reality model, storage medium, and electronic device
TWI672629B (en) Emoticon display method, apparatus and computer-readable storage medium
WO2018113639A1 (en) Interaction method between user terminals, terminal, server, system and storage medium
TW201837783A (en) Method and related device of determining camera posture information
JP2021524957A (en) Image processing methods and their devices, terminals and computer programs
CN110427110B (en) Live broadcast method and device and live broadcast server
CN108985220B (en) Face image processing method and device and storage medium
US9811611B2 (en) Method and apparatus for creating curved surface model
US9760998B2 (en) Video processing method and apparatus
CN108346175B (en) Face image restoration method, device and storage medium
CN111290948B (en) Test data acquisition method and device, computer equipment and readable storage medium
JP7487293B2 (en) Method and device for controlling virtual camera movement, and computer device and program
US10675541B2 (en) Control method of scene sound effect and related products
CN110599593B (en) Data synthesis method, device, equipment and storage medium
US9883018B2 (en) Apparatus for recording conversation and method thereof
CN114092655A (en) Map construction method, device, equipment and storage medium
CN110717964B (en) Scene modeling method, terminal and readable storage medium
CN105635553B (en) Image shooting method and device
CN110070617B (en) Data synchronization method, device and hardware device
CN109618055B (en) Position sharing method and mobile terminal
CN111192072A (en) User grouping method and device and storage medium
CN111556248B (en) Shooting method, shooting device, storage medium and mobile terminal

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19815829

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19815829

Country of ref document: EP

Kind code of ref document: A1