CN109472820A - Monocular RGB-D camera real-time face method for reconstructing and device - Google Patents
Monocular RGB-D camera real-time face method for reconstructing and device Download PDFInfo
- Publication number
- CN109472820A CN109472820A CN201811222294.7A CN201811222294A CN109472820A CN 109472820 A CN109472820 A CN 109472820A CN 201811222294 A CN201811222294 A CN 201811222294A CN 109472820 A CN109472820 A CN 109472820A
- Authority
- CN
- China
- Prior art keywords
- characteristic point
- face
- dimensional coordinate
- rigid motion
- key frame
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/50—Depth or shape recovery
- G06T7/55—Depth or shape recovery from multiple images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T15/00—3D [Three Dimensional] image rendering
- G06T15/005—General purpose rendering architectures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10024—Color image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30196—Human being; Person
- G06T2207/30201—Face
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Computer Graphics (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a kind of monocular RGB-D camera real-time face method for reconstructing and devices, wherein method includes: to detect the position of human face characteristic point on the face RGB image of input by advanced facial feature points detection algorithm；The three-dimensional coordinate of each characteristic point of present frame is obtained according to the position of human face characteristic point；Obtain the current three-dimensional coordinate of each human face characteristic point on key frame；It obtains key frame according to three-dimensional coordinate and current three-dimensional coordinate to move to the global rigid of each frame, to obtain rigid motion result；Use rigid motion result as the initialization of ICP, to finely tune face rigid motion；Rigid motion result is acted on into key frame model, is indicated with the TSDF of more new model.This method has effectively removed the depth in non-face region, removes the influence of non-rigid motion, and can use the accuracy that human face characteristic point improves rigid motion estimation.
Description
Technical field
The present invention relates to three-dimensional reconstruction field, in particular to a kind of monocular RGB-D camera real-time face method for reconstructing
And device.
Background technique
In the related technology, three-dimensional reconstruction is the research hotspot of computer vision and field of Computer Graphics, is empty
One of core technologies in fields such as quasi- reality/augmented reality, automatic Pilot, robot, have a wide range of applications.Occur in recent years
Many work using consumption levels depth camera (such as Microsoft Kinect, Intel RealSense) to general scene with
Object carries out real-time three-dimensional reconstruction.
This kind of work is mostly based on ICP algorithm and carries out rigidity to reconstructed good geometric part and present frame input point cloud
Registration estimates present frame with respect to the rigid motion of key frame (global rotation and translation).This method quickly move in camera or
There is biggish limitation when person's reconstructed object fast moves, often occur causing because of rigid motion estimation inaccuracy
Reconstruction failure.
Summary of the invention
The application is to be made based on inventor to the understanding of following problems and discovery:
The reconstruction of monocular RGB-D camera real-time three-dimensional is the research hotspot of computer graphics and computer vision field, such as
What quickly and accurately rebuilds the geometry, reflectivity and ambient lighting etc. of general object according to the input data of monocular RGB-D camera
Information is an important research topic.Advanced reconstruction technique is mostly used in geometrical registration link based on repeatedly in recent years
For closest approach (ICP) algorithm, but such methods can only generally cope with the movement of slower camera or object.
The present invention is directed to solve at least some of the technical problems in related technologies.
For this purpose, an object of the present invention is to provide a kind of monocular RGB-D camera real-time face method for reconstructing, this method
The depth for having effectively removed non-face region, removes the influence of non-rigid motion, and can use human face characteristic point and improve rigidity
The accuracy of estimation.
It is another object of the present invention to propose a kind of monocular RGB-D camera real-time face method for reconstructing.
In order to achieve the above objectives, one aspect of the present invention embodiment proposes a kind of monocular RGB-D camera real-time face reconstruction
Method, comprising the following steps: step S1: it is examined on the face RGB image of input by advanced facial feature points detection algorithm
Survey the position of human face characteristic point；Step S2: the three-dimensional of each characteristic point of present frame is obtained according to the position of the human face characteristic point
Coordinate；Step S3: the current three-dimensional coordinate of each human face characteristic point on key frame is obtained；Step S4: according to the three-dimensional coordinate
It obtains the key frame with the current three-dimensional coordinate to move to the global rigid of each frame, to obtain rigid motion result；Step
Rapid S5: initialization of the rigid motion result as ICP is used, to finely tune face rigid motion；Step S6: by the rigidity
Motion result acts on key frame model, is indicated with the TSDF of more new model.
The monocular RGB-D camera real-time face method for reconstructing of the embodiment of the present invention, it is contemplated that the particularity of human face structure, benefit
The accuracy that monocular RGB-D camera real-time reconstruction face is improved with advanced features of human face images detection technique, for face
The method of the new estimation global rigid movement of one kind that this kind of special objective proposes, it is real to can handle face when face quickly moves
When three-dimensional reconstruction to effectively remove the depth in non-face region remove the influence of non-rigid motion, and can use face
Characteristic point improves the accuracy of rigid motion estimation.
In addition, monocular RGB-D camera real-time face method for reconstructing according to the above embodiment of the present invention can also have with
Under additional technical characteristic:
Further, in one embodiment of the invention, the step S1 further comprises: by the feature of face outer ring
Point is divided into left characteristic point and right characteristic point；The left characteristic point and the right characteristic point are used into Thermal conduction respectively,
And after fitting, the depth data in reservation while the region being located above two curves；By the depth other than the region
Value is set to zero.
Further, in one embodiment of the invention, the step S2 further comprises: according to remaining internal special
Sign point finds each characteristic point corresponding position on depth image, and obtains institute by the internal reference matrix back projection of depth camera
State the three-dimensional coordinate of each characteristic point of present frame.
Further, in one embodiment of the invention, the step S3 further comprises: the model that will currently rebuild
Its corresponding depth map is rendered, and uses the current three-dimensional coordinate for obtaining characteristic point on key frame model.
Further, in one embodiment of the invention, the step S4 further comprises: global rigid movement is built
Mould is an optimization problem, the target of optimization are as follows:
Wherein, R and t respectively indicates rigid rotating and translation to be optimized, and n is characterized quantity a little,Indicate current
The three-dimensional coordinate of input frame ith feature point,Indicate the three-dimensional coordinate of key frame ith feature point.
In order to achieve the above objectives, another aspect of the present invention embodiment proposes a kind of monocular RGB-D camera real-time face weight
Build device, comprising the following steps: detection module, for by advanced facial feature points detection algorithm input face RGB
The position of human face characteristic point is detected on image；First processing module, for being obtained currently according to the position of the human face characteristic point
The three-dimensional coordinate of each characteristic point of frame；Module is obtained, for obtaining the current three-dimensional coordinate of each human face characteristic point on key frame；
Second processing module, for obtaining the key frame to the complete of each frame according to the three-dimensional coordinate and the current three-dimensional coordinate
Office's rigid motion, to obtain rigid motion result；Initialization module, for using the rigid motion result as the first of ICP
Beginningization, to finely tune face rigid motion；Update module, for the rigid motion result to be acted on key frame model, with more
The TSDF of new model is indicated.
The monocular RGB-D camera real-time face reconstructing device of the embodiment of the present invention, it is contemplated that the particularity of human face structure, benefit
The accuracy that monocular RGB-D camera real-time reconstruction face is improved with advanced features of human face images detection technique, for face
The method of the new estimation global rigid movement of one kind that this kind of special objective proposes, it is real to can handle face when face quickly moves
When three-dimensional reconstruction to effectively remove the depth in non-face region remove the influence of non-rigid motion, and can use face
Characteristic point improves the accuracy of rigid motion estimation.
In addition, monocular RGB-D camera real-time face reconstructing device according to the above embodiment of the present invention can also have with
Under additional technical characteristic:
Further, in one embodiment of the invention, the detection module is further used for the spy of face outer ring
Sign point is divided into left characteristic point and right characteristic point, and the left characteristic point and the right characteristic point are intended with exponential function curve respectively
It closes, and after fitting, the depth data in reservation while the region being located above two curves, by the depth other than the region
Angle value is set to zero.
Further, in one embodiment of the invention, the first processing module is further used for according to remaining
Inter characteristic points find each characteristic point corresponding position on depth image, and pass through the internal reference matrix back projection of depth camera
Obtain the three-dimensional coordinate of each characteristic point of the present frame.
Further, in one embodiment of the invention, the acquisition module is further used for the mould that will currently rebuild
Type renders its corresponding depth map, and uses the current three-dimensional coordinate for obtaining characteristic point on key frame model.
Further, in one embodiment of the invention, the Second processing module is further used for global rigid
Motion modeling is an optimization problem, the target of optimization are as follows:
Wherein, R and t respectively indicates rigid rotating and translation to be optimized, and n is characterized quantity a little,Indicate current
The three-dimensional coordinate of input frame ith feature point,Indicate the three-dimensional coordinate of key frame ith feature point.
The additional aspect of the present invention and advantage will be set forth in part in the description, and will partially become from the following description
Obviously, or practice through the invention is recognized.
Detailed description of the invention
Above-mentioned and/or additional aspect and advantage of the invention will become from the following description of the accompanying drawings of embodiments
Obviously and it is readily appreciated that, in which:
Fig. 1 is the flow chart according to the monocular RGB-D camera real-time face method for reconstructing of one embodiment of the invention；
Fig. 2 is the flow chart according to the monocular RGB-D camera real-time face method for reconstructing of a specific embodiment of the invention；
Fig. 3 is to use facial feature estimation rigid motion and ICP method estimation effect pair according to one embodiment of the invention
Than figure；
Fig. 4 is the structural schematic diagram according to the monocular RGB-D camera real-time face reconstructing device of one embodiment of the invention.
Specific embodiment
The embodiment of the present invention is described below in detail, examples of the embodiments are shown in the accompanying drawings, wherein from beginning to end
Same or similar label indicates same or similar element or element with the same or similar functions.Below with reference to attached
The embodiment of figure description is exemplary, it is intended to is used to explain the present invention, and is not considered as limiting the invention.
Describe with reference to the accompanying drawings the monocular RGB-D camera real-time face method for reconstructing proposed according to embodiments of the present invention and
Device describes the monocular RGB-D camera real-time face method for reconstructing proposed according to embodiments of the present invention with reference to the accompanying drawings first.
Fig. 1 is the flow chart of the monocular RGB-D camera real-time face method for reconstructing of one embodiment of the invention.
As shown in Figure 1, monocular RGB-D camera real-time face method for reconstructing the following steps are included:
Step S1: face characteristic is detected on the face RGB image of input by advanced facial feature points detection algorithm
The position of point.
Further, in one embodiment of the invention, step S1 further comprises: by the characteristic point of face outer ring point
For left characteristic point and right characteristic point；Left characteristic point and right characteristic point are used into Thermal conduction respectively, and in fitting
Afterwards, the depth data in reservation while the region being located above two curves；Depth value other than region is set to zero.
It is understood that as shown in Fig. 2, using advanced facial feature points detection algorithm in input face RGB image
The position of upper detection human face characteristic point, this step only use the characteristic point of face outer ring；The embodiment of the present invention is face outer ring
Characteristic point is divided into left and right two halves, and each Thermal conduction of using by oneself after fitting, only retains while being located at two curves
The depth data in the region of top, and the part other than this region is thought to be not belonging to face part, and on these positions
Depth value be set to zero.
It should be noted that the embodiment of the present invention uses the RGB image that resolution ratio is 640 × 480 and has identical resolution
The depth image of rate, RGB image and depth image were aligned in advance, so that the pixel of same position has correspondence on two images
Relationship, herein only as an example, being not specifically limited.
Specifically, the embodiment of the present invention removes non-face regional depth data, specifically include:
Input depth image generally comprises the depth data in non-face region, such as shoulder, background, since face is turning
Its movement and the movement in non-face region are not consistent during dynamic, produce non-rigid motion on the whole, utilize face periphery
The curve that characteristic point surrounds, the depth data in the non-face region of automatic rejection.
The characteristic point of face outer ring is divided into left and right two halves by the embodiment of the present invention, and each personal exponential function curve is quasi-
It closes, after fitting, the region for retaining while being located above two curves, and the part other than this region is not belonging to face
Point, therefore, the depth value on these positions is set to zero.
Step S2: the three-dimensional coordinate of each characteristic point of present frame is obtained according to the position of human face characteristic point.
Further, in one embodiment of the invention, step S2 further comprises: according to remaining inter characteristic points
Each characteristic point corresponding position on depth image is found, and present frame is obtained by the internal reference matrix back projection of depth camera
The three-dimensional coordinate of each characteristic point.
It is understood that as shown in Fig. 2, the embodiment of the present invention uses face characteristic on the RGB image inside step S1
The pixel coordinate of point, this step and step S1 on the contrary, do not use the characteristic point of outer ring, and use remaining inter characteristic points, and
Each characteristic point corresponding position on depth image is found, here because RGB image and depth image were aligned, the two
Pixel coordinate having the same.Finally the three-dimensional of each characteristic point of present frame is obtained using the internal reference matrix back projection of depth camera
Coordinate p_i^live | p_i^live ∈ R^3, i=1 ..., n }.
Step S3: the current three-dimensional coordinate of each human face characteristic point on key frame is obtained.
Further, in one embodiment of the invention, step S3 further comprises: the model rendering that will currently rebuild
Its corresponding depth map, and use the current three-dimensional coordinate for obtaining characteristic point on key frame model.
It is understood that as shown in Fig. 2, calculating the three-dimensional coordinate of each human face characteristic point at this time on key frame, this hair
Bright embodiment needs its corresponding depth map of the model rendering currently rebuild then to calculate using the method similar with step S2
The three-dimensional coordinate of characteristic point on key frame model
Step S4: obtaining key frame according to three-dimensional coordinate and current three-dimensional coordinate and move to the global rigid of each frame, with
Obtain rigid motion result.
It is understood that as shown in Fig. 2, calculating key frame to each frame according to the three-dimensional coordinate of this two groups of characteristic points
Global rigid move R and t, we are modeled as an optimization problem.
Wherein, in one embodiment of the invention, step S4 further comprises: being one by global rigid motion modeling
Optimization problem, the target of optimization are as follows:
Wherein, R and t respectively indicates rigid rotating and translation to be optimized, and n is characterized quantity a little,Indicate current
The three-dimensional coordinate of input frame ith feature point,Indicate the three-dimensional coordinate of key frame ith feature point.
Step S5: use rigid motion result as the initialization of ICP, to finely tune face rigid motion.
It is understood that as shown in Fig. 2, the embodiment of the present invention uses this estimated result as the initialization of ICP, into
One step finely tunes face rigid motion.
Step S6: rigid motion result is acted on into key frame model, is indicated with the TSDF of more new model.
It is understood that as shown in Fig. 2, the embodiment of the present invention is according to the rigid motion currently estimated as a result, acting on
The TSDF of key frame model, more new model is indicated.Also, use facial feature estimation rigid motion and ICP method estimation effect pair
It is more as shown in Figure 3 than scheming.
Specifically, the embodiment of the present invention accurately estimates that global rigid moves using characteristic point, tool according to step S2-S6
Body includes:
In each frame, calculate the three-dimensional coordinate of two groups of characteristic points, one group be present incoming frame characteristic point three-dimensional coordinate, separately
One group be the updated characteristic point of key frame three-dimensional coordinate.
The three-dimensional coordinate of the human face characteristic point of present frame input point cloud, can be by the two dimension that detects on the frame RGB image
The pixel coordinate of characteristic point and the internal reference matrix of depth camera are calculated: after detecting characteristic point on RGB image,
Find each characteristic point corresponding pixel coordinate on depth map, by depth camera internal reference matrix, available each feature
Three-dimensional coordinate of the point under depth camera coordinate systemWherein, n be using human face characteristic point
Quantity, here do not use outer ring characteristic point because semantic locations of the outer ring characteristic point on face can be sent out in different positions
Changing.
For the three-dimensional coordinate of human face characteristic point on key frame, since the faceform that each frame is rebuild is updating,
Surface is increasingly more complete, and noise is also constantly reducing, thus each frame we need the model rendering currently rebuild, its is right
Then the depth map answered uses the method similar with the characteristic point three-dimensional coordinate calculated on input point cloud to calculate on key frame model
The three-dimensional coordinate of characteristic point
According to the three-dimensional coordinate of this two groups of characteristic points, the global rigid for calculating key frame to each frame moves R and t, we
It is modeled as an optimization problem, the target of optimization is:
The embodiment of the present invention uses this estimated result as the initialization of ICP, further finely tunes the rigid motion of face,
Since compared under full spread position, when being more than 45 degree such as side face angle, some characteristic points can be blocked, and lead to Partial Feature point three
It ties up coordinate and calculates inaccuracy, the present invention still manages micro- to the estimation progress of rigid motion again using the original method based on ICP
It adjusts.
Finally, the TSDF of more new model is indicated according to the rigid motion currently estimated as a result, acting on key frame model.
The monocular RGB-D camera real-time face method for reconstructing proposed according to embodiments of the present invention, it is contemplated that human face structure
Particularity improves the accuracy of monocular RGB-D camera real-time reconstruction face using advanced features of human face images detection technique,
For a kind of method for new estimation global rigid movement that this kind of special objective of face proposes, it can handle face and quickly move
When face real-time three-dimensional rebuild, to effectively remove the depth in non-face region, remove the influence of non-rigid motion, and can be with
The accuracy of rigid motion estimation is improved using human face characteristic point.
The monocular RGB-D camera real-time face reconstructing device proposed according to embodiments of the present invention is described referring next to attached drawing.
Fig. 4 is the structural schematic diagram of the monocular RGB-D camera real-time face reconstructing device of one embodiment of the invention.
As shown in figure 4, monocular RGB-D camera real-time face reconstructing device 10 includes: that detection module 100, first is handled
Module 200 obtains module 300, Second processing module 400, initialization module 500 and update module 600.
Wherein, detection module 100 is used for the face RGB image by advanced facial feature points detection algorithm in input
Detect the position of human face characteristic point.First processing module 200 is used to obtain each spy of present frame according to the position of human face characteristic point
Levy the three-dimensional coordinate of point.Obtain the current three-dimensional coordinate that module 300 is used to obtain each human face characteristic point on key frame.At second
Reason module 400 is used to obtain key frame according to three-dimensional coordinate and current three-dimensional coordinate and moves to the global rigid of each frame, with
To rigid motion result.Initialization module 500 is rigid to finely tune face for using rigid motion result as the initialization of ICP
Property movement.Update module 600 is used to rigid motion result acting on key frame model, is indicated with the TSDF of more new model.This
The device 10 of inventive embodiments has effectively removed the depth in non-face region, removes the influence of non-rigid motion, and can use
Human face characteristic point improves the accuracy of rigid motion estimation.
Further, in one embodiment of the invention, detection module 100 is further used for the feature of face outer ring
Point is divided into left characteristic point and right characteristic point, left characteristic point and right characteristic point is used Thermal conduction respectively, and quasi-
After conjunction, the depth value other than region is set to zero by the depth data in reservation while the region being located above two curves.
Further, in one embodiment of the invention, first processing module 200 is further used for according to remaining interior
Portion's characteristic point finds each characteristic point corresponding position on depth image, and is obtained by the internal reference matrix back projection of depth camera
Take the three-dimensional coordinate of each characteristic point of present frame.
Further, in one embodiment of the invention, it obtains module 300 and is further used for the model that will currently rebuild
Its corresponding depth map is rendered, and uses the current three-dimensional coordinate for obtaining characteristic point on key frame model.
Further, in one embodiment of the invention, Second processing module 400 is further used for transporting global rigid
It is dynamic to be modeled as an optimization problem, the target of optimization are as follows:
Wherein, R and t respectively indicates rigid rotating and translation to be optimized, and n is characterized quantity a little,Indicate current
The three-dimensional coordinate of input frame ith feature point,Indicate the three-dimensional coordinate of key frame ith feature point.
It should be noted that the aforementioned explanation to monocular RGB-D camera real-time face method for reconstructing embodiment is also fitted
For the monocular RGB-D camera real-time face reconstructing device of the embodiment, details are not described herein again.
The monocular RGB-D camera real-time face reconstructing device proposed according to embodiments of the present invention, it is contemplated that human face structure
Particularity improves the accuracy of monocular RGB-D camera real-time reconstruction face using advanced features of human face images detection technique,
For a kind of method for new estimation global rigid movement that this kind of special objective of face proposes, it can handle face and quickly move
When face real-time three-dimensional rebuild, to effectively remove the depth in non-face region, remove the influence of non-rigid motion, and can be with
The accuracy of rigid motion estimation is improved using human face characteristic point.
In the description of the present invention, it is to be understood that, term " center ", " longitudinal direction ", " transverse direction ", " length ", " width ",
" thickness ", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom" "inner", "outside", " up time
The orientation or positional relationship of the instructions such as needle ", " counterclockwise ", " axial direction ", " radial direction ", " circumferential direction " be orientation based on the figure or
Positional relationship is merely for convenience of description of the present invention and simplification of the description, rather than the device or element of indication or suggestion meaning must
There must be specific orientation, be constructed and operated in a specific orientation, therefore be not considered as limiting the invention.
In addition, term " first ", " second " are used for descriptive purposes only and cannot be understood as indicating or suggesting relative importance
Or implicitly indicate the quantity of indicated technical characteristic.Define " first " as a result, the feature of " second " can be expressed or
Implicitly include at least one this feature.In the description of the present invention, the meaning of " plurality " is at least two, such as two, three
It is a etc., unless otherwise specifically defined.
In the present invention unless specifically defined or limited otherwise, term " installation ", " connected ", " connection ", " fixation " etc.
Term shall be understood in a broad sense, for example, it may be being fixedly connected, may be a detachable connection, or integral；It can be mechanical connect
It connects, is also possible to be electrically connected；It can be directly connected, can also can be in two elements indirectly connected through an intermediary
The interaction relationship of the connection in portion or two elements, unless otherwise restricted clearly.For those of ordinary skill in the art
For, the specific meanings of the above terms in the present invention can be understood according to specific conditions.
In the present invention unless specifically defined or limited otherwise, fisrt feature in the second feature " on " or " down " can be with
It is that the first and second features directly contact or the first and second features pass through intermediary mediate contact.Moreover, fisrt feature exists
Second feature " on ", " top " and " above " but fisrt feature be directly above or diagonally above the second feature, or be merely representative of
First feature horizontal height is higher than second feature.Fisrt feature can be under the second feature " below ", " below " and " below "
One feature is directly under or diagonally below the second feature, or is merely representative of first feature horizontal height less than second feature.
In the description of this specification, reference term " one embodiment ", " some embodiments ", " example ", " specifically show
The description of example " or " some examples " etc. means specific features, structure, material or spy described in conjunction with this embodiment or example
Point is included at least one embodiment or example of the invention.In the present specification, schematic expression of the above terms are not
It must be directed to identical embodiment or example.Moreover, particular features, structures, materials, or characteristics described can be in office
It can be combined in any suitable manner in one or more embodiment or examples.In addition, without conflicting with each other, the skill of this field
Art personnel can tie the feature of different embodiments or examples described in this specification and different embodiments or examples
It closes and combines.
Although the embodiments of the present invention has been shown and described above, it is to be understood that above-described embodiment is example
Property, it is not considered as limiting the invention, those skilled in the art within the scope of the invention can be to above-mentioned
Embodiment is changed, modifies, replacement and variant.
Claims (10)
1. a kind of monocular RGB-D camera real-time face method for reconstructing, which comprises the following steps:
Step S1: human face characteristic point is detected on the face RGB image of input by advanced facial feature points detection algorithm
Position；
Step S2: the three-dimensional coordinate of each characteristic point of present frame is obtained according to the position of the human face characteristic point；
Step S3: the current three-dimensional coordinate of each human face characteristic point on key frame is obtained；
Step S4: according to the three-dimensional coordinate and the current three-dimensional coordinate obtain the key frame to each frame global rigid
Movement, to obtain rigid motion result；
Step S5: initialization of the rigid motion result as ICP is used, to finely tune face rigid motion；And
Step S6: the rigid motion result is acted on into key frame model, is indicated with the TSDF of more new model.
2. monocular RGB-D camera real-time face method for reconstructing according to claim 1, which is characterized in that the step S1
Further comprise:
The characteristic point of face outer ring is divided into left characteristic point and right characteristic point；
The left characteristic point and the right characteristic point are used into Thermal conduction, and after fitting respectively, retained same
When region above two curves depth data；
Depth value other than the region is set to zero.
3. monocular RGB-D camera real-time face method for reconstructing according to claim 1, which is characterized in that the step S2
Further comprise:
Each characteristic point corresponding position on depth image is found according to remaining inter characteristic points, and passes through depth camera
Internal reference matrix back projection obtains the three-dimensional coordinate of each characteristic point of the present frame.
4. monocular RGB-D camera real-time face method for reconstructing according to claim 1, which is characterized in that the step S3
Further comprise:
By its corresponding depth map of the model rendering currently rebuild, and use the current three-dimensional for obtaining characteristic point on key frame model
Coordinate.
5. monocular RGB-D camera real-time face method for reconstructing according to claim 1, which is characterized in that the step S4
Further comprise:
It is an optimization problem, the target of optimization by global rigid motion modeling are as follows:
Wherein, R and t respectively indicates rigid rotating and translation to be optimized, and n is characterized quantity a little,Indicate current input
The three-dimensional coordinate of frame ith feature point,Indicate the three-dimensional coordinate of key frame ith feature point.
6. a kind of monocular RGB-D camera real-time face reconstructing device, which comprises the following steps:
Detection module, for detecting face spy on the face RGB image of input by advanced facial feature points detection algorithm
Levy the position of point；
First processing module, for obtaining the three-dimensional coordinate of each characteristic point of present frame according to the position of the human face characteristic point；
Module is obtained, for obtaining the current three-dimensional coordinate of each human face characteristic point on key frame；
Second processing module, for obtaining the key frame to each frame according to the three-dimensional coordinate and the current three-dimensional coordinate
Global rigid movement, to obtain rigid motion result；
Initialization module, for using initialization of the rigid motion result as ICP, to finely tune face rigid motion；With
And
Update module is indicated for the rigid motion result to be acted on key frame model with the TSDF of more new model.
7. monocular RGB-D camera real-time face reconstructing device according to claim 6, which is characterized in that the detection mould
Block is further used for the characteristic point of face outer ring being divided into left characteristic point and right characteristic point, by the left characteristic point and the right spy
Sign point uses Thermal conduction respectively, and after fitting, the depth in reservation while the region being located above two curves
Depth value other than the region is set to zero by degree evidence.
8. monocular RGB-D camera real-time face reconstructing device according to claim 6, which is characterized in that at described first
Reason module is further used for finding each characteristic point corresponding position on depth image according to remaining inter characteristic points, and leads to
The internal reference matrix back projection for crossing depth camera obtains the three-dimensional coordinate of each characteristic point of the present frame.
9. monocular RGB-D camera real-time face reconstructing device according to claim 6, which is characterized in that the acquisition mould
Block is further used for its corresponding depth map of the model rendering that will currently rebuild, and uses characteristic point on acquisition key frame model
Current three-dimensional coordinate.
10. monocular RGB-D camera real-time face reconstructing device according to claim 6, which is characterized in that at described second
Reason module is further used for global rigid motion modeling being an optimization problem, the target of optimization are as follows:
Wherein, R and t respectively indicates rigid rotating and translation to be optimized, and n is characterized quantity a little,Indicate current input
The three-dimensional coordinate of frame ith feature point,Indicate the three-dimensional coordinate of key frame ith feature point.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811222294.7A CN109472820B (en) | 2018-10-19 | 2018-10-19 | Monocular RGB-D camera real-time face reconstruction method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811222294.7A CN109472820B (en) | 2018-10-19 | 2018-10-19 | Monocular RGB-D camera real-time face reconstruction method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109472820A true CN109472820A (en) | 2019-03-15 |
| CN109472820B CN109472820B (en) | 2021-03-16 |
Family
ID=65665744
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811222294.7A Active CN109472820B (en) | 2018-10-19 | 2018-10-19 | Monocular RGB-D camera real-time face reconstruction method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109472820B (en) |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109949412A (en) * | 2019-03-26 | 2019-06-28 | 腾讯科技（深圳）有限公司 | A kind of three dimensional object method for reconstructing and device |
| CN110363858A (en) * | 2019-06-18 | 2019-10-22 | 新拓三维技术(深圳)有限公司 | A kind of three-dimensional facial reconstruction method and system |
| CN110533773A (en) * | 2019-09-02 | 2019-12-03 | 北京华捷艾米科技有限公司 | A kind of three-dimensional facial reconstruction method, device and relevant device |
| CN110689625A (en) * | 2019-09-06 | 2020-01-14 | 清华大学 | Automatic generation method and device for customized face mixed expression model |
| CN110910452A (en) * | 2019-11-26 | 2020-03-24 | 上海交通大学 | Low-texture industrial part pose estimation method based on deep learning |
| CN113221600A (en) * | 2020-01-21 | 2021-08-06 | 初速度（苏州）科技有限公司 | Method and device for calibrating image feature points |
| CN113674161A (en) * | 2021-07-01 | 2021-11-19 | 清华大学 | Face deformity scanning completion method and device based on deep learning |
| CN113902847A (en) * | 2021-10-11 | 2022-01-07 | 岱悟智能科技(上海)有限公司 | Monocular depth image pose optimization method based on three-dimensional feature constraint |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR101556992B1 (en) * | 2014-03-13 | 2015-10-05 | 손우람 | 3d scanning system using facial plastic surgery simulation |
| US9213882B2 (en) * | 2010-03-11 | 2015-12-15 | Ramot At Tel-Aviv University Ltd. | Devices and methods of reading monochromatic patterns |
| CN106289181A (en) * | 2015-05-22 | 2017-01-04 | 北京雷动云合智能技术有限公司 | A kind of real-time SLAM method that view-based access control model is measured |
| CN106446815A (en) * | 2016-09-14 | 2017-02-22 | 浙江大学 | Simultaneous positioning and map building method |
| CN106910242A (en) * | 2017-01-23 | 2017-06-30 | 中国科学院自动化研究所 | The method and system of indoor full scene three-dimensional reconstruction are carried out based on depth camera |
| CN106934827A (en) * | 2015-12-31 | 2017-07-07 | 杭州华为数字技术有限公司 | The method for reconstructing and device of three-dimensional scenic |
| CN108549873A (en) * | 2018-04-19 | 2018-09-18 | 北京华捷艾米科技有限公司 | Three-dimensional face identification method and three-dimensional face recognition system |
-
2018
- 2018-10-19 CN CN201811222294.7A patent/CN109472820B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9213882B2 (en) * | 2010-03-11 | 2015-12-15 | Ramot At Tel-Aviv University Ltd. | Devices and methods of reading monochromatic patterns |
| KR101556992B1 (en) * | 2014-03-13 | 2015-10-05 | 손우람 | 3d scanning system using facial plastic surgery simulation |
| CN106289181A (en) * | 2015-05-22 | 2017-01-04 | 北京雷动云合智能技术有限公司 | A kind of real-time SLAM method that view-based access control model is measured |
| CN106934827A (en) * | 2015-12-31 | 2017-07-07 | 杭州华为数字技术有限公司 | The method for reconstructing and device of three-dimensional scenic |
| CN106446815A (en) * | 2016-09-14 | 2017-02-22 | 浙江大学 | Simultaneous positioning and map building method |
| CN106910242A (en) * | 2017-01-23 | 2017-06-30 | 中国科学院自动化研究所 | The method and system of indoor full scene three-dimensional reconstruction are carried out based on depth camera |
| CN108549873A (en) * | 2018-04-19 | 2018-09-18 | 北京华捷艾米科技有限公司 | Three-dimensional face identification method and three-dimensional face recognition system |
Cited By (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109949412A (en) * | 2019-03-26 | 2019-06-28 | 腾讯科技（深圳）有限公司 | A kind of three dimensional object method for reconstructing and device |
| US11715224B2 (en) | 2019-03-26 | 2023-08-01 | Tencent Technology (Shenzhen) Company Limited | Three-dimensional object reconstruction method and apparatus |
| CN109949412B (en) * | 2019-03-26 | 2021-03-02 | 腾讯科技（深圳）有限公司 | Three-dimensional object reconstruction method and device |
| CN110363858A (en) * | 2019-06-18 | 2019-10-22 | 新拓三维技术(深圳)有限公司 | A kind of three-dimensional facial reconstruction method and system |
| CN110363858B (en) * | 2019-06-18 | 2022-07-01 | 新拓三维技术(深圳)有限公司 | Three-dimensional face reconstruction method and system |
| CN110533773A (en) * | 2019-09-02 | 2019-12-03 | 北京华捷艾米科技有限公司 | A kind of three-dimensional facial reconstruction method, device and relevant device |
| CN110689625B (en) * | 2019-09-06 | 2021-07-16 | 清华大学 | Automatic generation method and device for customized face mixed expression model |
| CN110689625A (en) * | 2019-09-06 | 2020-01-14 | 清华大学 | Automatic generation method and device for customized face mixed expression model |
| CN110910452A (en) * | 2019-11-26 | 2020-03-24 | 上海交通大学 | Low-texture industrial part pose estimation method based on deep learning |
| CN110910452B (en) * | 2019-11-26 | 2023-08-25 | 上海交通大学 | Low-texture industrial part pose estimation method based on deep learning |
| CN113221600A (en) * | 2020-01-21 | 2021-08-06 | 初速度（苏州）科技有限公司 | Method and device for calibrating image feature points |
| CN113221600B (en) * | 2020-01-21 | 2022-06-21 | 魔门塔(苏州)科技有限公司 | Method and device for calibrating image feature points |
| CN113674161A (en) * | 2021-07-01 | 2021-11-19 | 清华大学 | Face deformity scanning completion method and device based on deep learning |
| CN113902847A (en) * | 2021-10-11 | 2022-01-07 | 岱悟智能科技(上海)有限公司 | Monocular depth image pose optimization method based on three-dimensional feature constraint |
| CN113902847B (en) * | 2021-10-11 | 2024-04-16 | 岱悟智能科技(上海)有限公司 | Monocular depth image pose optimization method based on three-dimensional feature constraint |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109472820B (en) | 2021-03-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109472820A (en) | Monocular RGB-D camera real-time face method for reconstructing and device | |
| US10521085B2 (en) | Image processing device, image processing method, and program for displaying an image in accordance with a selection from a displayed menu and based on a detection by a sensor | |
| CN108090958B (en) | Robot synchronous positioning and map building method and system | |
| US20200213578A1 (en) | Drone based capture of multi-view interactive digital media | |
| US20180211399A1 (en) | Modeling method and apparatus using three-dimensional (3d) point cloud | |
| CN108596974A (en) | Dynamic scene robot localization builds drawing system and method | |
| Wen et al. | Hybrid semi-dense 3D semantic-topological mapping from stereo visual-inertial odometry SLAM with loop closure detection | |
| CN113888639B (en) | Visual odometer positioning method and system based on event camera and depth camera | |
| CN113393503A (en) | Classification-driven shape prior deformation category-level object 6D pose estimation method | |
| CN113614735A (en) | Dense 6-DoF gesture object detector | |
| CN110245199A (en) | A kind of fusion method of high inclination-angle video and 2D map | |
| CN110517284A (en) | A kind of target tracking method based on laser radar and Pan/Tilt/Zoom camera | |
| CN112598735A (en) | Single-image object pose estimation method fusing three-dimensional model information | |
| CN107909643B (en) | Mixed scene reconstruction method and device based on model segmentation | |
| Kundu et al. | Realtime moving object detection from a freely moving monocular camera | |
| Gählert et al. | Single-shot 3d detection of vehicles from monocular rgb images via geometrically constrained keypoints in real-time | |
| Zhang et al. | A robust visual odometry based on RGB-D camera in dynamic indoor environments | |
| Gählert et al. | Single-shot 3d detection of vehicles from monocular rgb images via geometry constrained keypoints in real-time | |
| US20230076331A1 (en) | Low motion to photon latency rapid target acquisition | |
| Ren et al. | Application of stereo vision technology in 3D reconstruction of traffic objects | |
| US11900621B2 (en) | Smooth and jump-free rapid target acquisition | |
| CN109903309A (en) | A kind of robot motion's information estimating method based on angle optical flow method | |
| Xu et al. | DOS-SLAM: A real-time dynamic object segmentation visual SLAM system | |
| Kurka et al. | Automatic estimation of camera parameters from a solid calibration box | |
| CN109325962B (en) | Information processing method, device, equipment and computer readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |