WO2022208442A1 - Automatic blending of human facial expression and full-body poses for dynamic digital human model creation using integrated photo-video volumetric capture system and mesh-tracking - Google Patents
Automatic blending of human facial expression and full-body poses for dynamic digital human model creation using integrated photo-video volumetric capture system and mesh-tracking Download PDFInfo
- Publication number
- WO2022208442A1 WO2022208442A1 PCT/IB2022/053036 IB2022053036W WO2022208442A1 WO 2022208442 A1 WO2022208442 A1 WO 2022208442A1 IB 2022053036 W IB2022053036 W IB 2022053036W WO 2022208442 A1 WO2022208442 A1 WO 2022208442A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- scanning
- mesh
- tracking
- poses
- capture system
- Prior art date
Links
- 238000002156 mixing Methods 0.000 title claims abstract description 52
- 230000008921 facial expression Effects 0.000 title description 5
- 210000003205 muscle Anatomy 0.000 claims description 85
- 238000000034 method Methods 0.000 claims description 37
- 238000010801 machine learning Methods 0.000 claims description 15
- 230000002123 temporal effect Effects 0.000 claims description 13
- 230000008685 targeting Effects 0.000 claims description 10
- 238000012545 processing Methods 0.000 claims description 4
- 238000010586 diagram Methods 0.000 description 14
- 230000009471 action Effects 0.000 description 6
- 210000003484 anatomy Anatomy 0.000 description 5
- 238000005452 bending Methods 0.000 description 5
- 230000001815 facial effect Effects 0.000 description 5
- 210000000988 bone and bone Anatomy 0.000 description 4
- 230000007935 neutral effect Effects 0.000 description 4
- 238000013459 approach Methods 0.000 description 3
- 238000000605 extraction Methods 0.000 description 3
- 210000003127 knee Anatomy 0.000 description 3
- 238000010422 painting Methods 0.000 description 3
- 210000003423 ankle Anatomy 0.000 description 2
- 238000000354 decomposition reaction Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 210000004247 hand Anatomy 0.000 description 2
- 238000010426 hand crafting Methods 0.000 description 2
- 210000001624 hip Anatomy 0.000 description 2
- 210000002414 leg Anatomy 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 230000007704 transition Effects 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 230000003190 augmentative effect Effects 0.000 description 1
- 230000015572 biosynthetic process Effects 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 210000001145 finger joint Anatomy 0.000 description 1
- 210000002683 foot Anatomy 0.000 description 1
- 230000014509 gene expression Effects 0.000 description 1
- 230000001795 light effect Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 239000003550 marker Substances 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 210000000337 motor cortex Anatomy 0.000 description 1
- 230000035479 physiological effects, processes and functions Effects 0.000 description 1
- 238000003786 synthesis reaction Methods 0.000 description 1
- 210000001226 toe joint Anatomy 0.000 description 1
- 210000005010 torso Anatomy 0.000 description 1
- 238000012549 training Methods 0.000 description 1
- 210000000707 wrist Anatomy 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
- G06T17/20—Finite element generation, e.g. wire-frame surface description, tesselation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2200/00—Indexing scheme for image data processing or generation, in general
- G06T2200/04—Indexing scheme for image data processing or generation, in general involving 3D image data
Definitions
- the present invention relates to three dimensional computer vision and graphics for the entertainment industry. More specifically, the present invention relates to acquiring and processing three dimensional computer vision and graphics for film, TV, music and game content creation.
- Virtual human creation is highly manual, time-consuming and expensive. Rather than hand-crafting Computer Graphics (CG) artwork from scratch, the recent trend is to efficiently create realistic digital human model by multi-view camera 3D/4D scanners.
- CG Computer Graphics
- Various 3D scanner studios (3Lateral, Avatta, TEN24, Pixel Light Effect, Eisko) and 4D scanner studio (4DViews, Microsoft, 8i, DGene) exist world-wide for camera captured based human digitization.
- a photo-based 3D scanner studio includes multiple array of high resolution photography cameras.
- the prior art of 3D scan typically is used to create rigged modeling and requires hand- crafting for animation as it does not capture deformation.
- An integrated photo-video volumetric capture system for 3D/4D scanning acquires 3D scans and 4D scans by acquiring images and videos simultaneously.
- the volumetric capture system for high-quality 4D scanning and mesh-tracking is used to establish topology correspondences across a 4D scanned mesh sequence for generating corrective shapes which will be used in shape interpolation and skeleton driven deformation.
- the volumetric capture system aids mesh-tracking for maintaining mesh registration (topology consistency) along with ease of extreme pose modeling.
- Major upper body and lower body joints are able to be identified that are important for generating deformation and capturing the same using a wide range of motion for all movement types across all joint categories.
- the volumetric capture system and mesh tracking the topology changes are tracked. Each pose captured will have the same topology which makes blending between multiple poses easier and more accurate.
- a method programmed in a non-transitory of a device comprises using a volumetric capture system configured for 3D scanning and 4D scanning including capturing photos and video simultaneously, wherein the 3D scanning and 4D scanning includes detecting muscle deformation of an actor and implementing mesh generation based on the 3D scanning and 4D scanning.
- the 3D scanning and 4D scanning include: 3D scans to be used to generate automatic high-fidelity extreme poses and 4D scans which include high temporal resolution which enables mesh tracking to automatically register extreme pose meshes for blending.
- Generating automatic high-fidelity extreme poses includes using 3D scans of the actor and muscle deformation of the actor to generate the automatic high-fidelity extreme poses.
- 4D scanning and mesh-tracking are used to establish topology correspondences across a 4D scanned mesh sequence for generating corrective shapes for shape interpolation and skeleton driven deformation.
- the method further comprises identifying and targeting joints and muscles of the actor by the volumetric capture system for 3D scanning and 4D scanning.
- Mesh generation includes muscle estimation or projection based on the 3D scanning and 4D scanning and machine learning.
- Implementing mesh generation includes using the 3D scanning and 4D scanning to generate meshes in extreme poses including muscle deformation.
- the method further comprises implementing mesh tracking for tracking topology changes to enable each pose captured to have a same topology for blending between poses.
- an apparatus comprises a non-transitory memory for storing an application, the application for: using a volumetric capture system configured for 3D scanning and 4D scanning including capturing photos and video simultaneously, wherein the 3D scanning and 4D scanning includes detecting muscle deformation of an actor and implementing mesh generation based on the 3D scanning and 4D scanning and a processor coupled to the memory, the processor configured for processing the application.
- the 3D scanning and 4D scanning include: 3D scans to be used to generate automatic high-fidelity extreme poses and 4D scans which include high temporal resolution which enables mesh tracking to automatically register extreme pose meshes for blending.
- Generating automatic high-fidelity extreme poses includes using 3D scans of the actor and muscle deformation of the actor to generate the automatic high- fidelity extreme poses.
- 4D scanning and mesh-tracking are used to establish topology correspondences across a 4D scanned mesh sequence for generating corrective shapes for shape interpolation and skeleton driven deformation.
- the application is further configured for identifying and targeting joints and muscles of the actor by the volumetric capture system for 3D scanning and 4D scanning.
- Mesh generation includes muscle estimation or projection based on the 3D scanning and 4D scanning and machine learning.
- Implementing mesh generation includes using the 3D scanning and 4D scanning to generate meshes in extreme poses including muscle deformation.
- the application is further configured for implementing mesh tracking for tracking topology changes to enable each pose captured to have a same topology for blending between poses.
- a system comprises a volumetric capture system for 3D and 4D scanning including capturing photos and video simultaneously, wherein the 3D scanning and 4D scanning includes detecting muscle deformation of an actor and a computing device configured for: receiving the captured photos and video from the volumetric capture system and implementing mesh generation based on the 3D scanning and 4D scanning.
- the 3D scanning and 4D scanning include: 3D scans to be used to generate automatic high-fidelity extreme poses and 4D scans which include high temporal resolution which enables mesh tracking to automatically register extreme pose meshes for blending.
- Generating automatic high-fidelity extreme poses includes using 3D scans of the actor and muscle deformation of the actor to generate the automatic high-fidelity extreme poses.
- 4D scanning and mesh-tracking are used to establish topology correspondences across a 4D scanned mesh sequence for generating corrective shapes for shape interpolation and skeleton driven deformation.
- the volumetric capture system is further configured for identifying and targeting joints and muscles of the actor by the volumetric capture system for 3D scanning and 4D scanning.
- Mesh generation includes muscle estimation or projection based on the 3D scanning and 4D scanning and machine learning.
- Implementing mesh generation includes using the 3D scanning and 4D scanning to generate meshes in extreme poses including muscle deformation.
- the volumetric capture system is further configured for implementing mesh tracking for tracking topology changes to enable each pose captured to have a same topology for blending between poses.
- Figure 1 illustrates a flowchart of a method of animating a subject using a photo-video volumetric capture system according to some embodiments.
- Figure 2 illustrates a diagram of a mesh generated by combining a neutral pose and extreme poses according to some embodiments.
- Figure 3 illustrates a diagram of the correlation between human anatomy verus computer graphics according to some embodiments.
- Figures 4A-B illustrate diagrams of muscle movements according to some embodiments.
- Figure 5 illustrates examples of major muscle groups according to some embodiments.
- Figure 6 illustrates a diagram of move types based on joints for mesh capture according to some embodiments.
- Figure 7 illustrates a diagram of move types based on joints for mesh capture according to some embodiments.
- Figure 8 illustrates examples of extreme poses according to some embodiments.
- Figure 9 illustrates a diagram of automatic blendshape extraction according to some embodiments.
- Figure 10 illustrates a flowchart of implementing mesh generation according to some embodiments.
- Figure 11 illustrates a block diagram of an exemplary computing device configured to implement the automatic blending method according to some embodiments.
- An automatic blending system utilizes an integrated photo-video volumetric capture system for 3D/4D scanning acquires 3D scans and 4D scans by acquiring images and videos simultaneously.
- the 3D scans are able to be used to generate auto high-fidelity extreme poses
- the 4D scans include high temporal resolution which enables mesh tracking to automatically register extreme pose meshes for blending.
- a volumetric capture system for high-quality 4D scanning and mesh tracking is able to be used to establish topology correspondences across a 4D scanned mesh sequence for generating corrective shapes which will be used in shape interpolation and skeleton driven deformation.
- the photo-video system aids mesh-tracking for maintaining mesh registration (topology consistency) along with ease of extreme pose modeling unlike hand-crafted shape modeling which aids registration but has manual shape generation and the 3D scanning- based approach which aids shape generation but not registration.
- Photo-video based capture is described in PCT Patent Application PCT/US2019/068151, filed December 20, 2019 titled, PHOTO-VIDEO BASED SPATIAL- TEMPORAL VOLUMETRIC CAPTURE SYSTEM FOR DYNAMIC 4D HUMAN FACE AND BODY DIGITIZATION, which is hereby incorporated by reference in its entirety for all purposes.
- the photo-video capture system is able to capture high fidelity texture in sparse time, and between the photo captures, video is captured, and the video is able to be used to establish the correspondence (e.g., transition) between the sparse photos.
- the correspondence information is able to be used to implement mesh tracking.
- Major upper body and lower body joints are able to be identified that are important for generating deformation and capturing the same using a wide range of motion for all movement types across all joint categories.
- the joints are able to be used in muscle deformation. For example, by knowing how a joint moves and how a muscle near a joint deforms, the skeleton/joint information is able to be used for muscle deformation which is able to be used for mesh generation. Furthering the example, the images and videos acquired are also able to be used by having a video of muscle deformation, the mesh of the muscle deformation is able to be more accurately generated.
- each pose captured will have the same topology which makes blending between multiple poses easier and more accurate.
- FIG. 1 illustrates a flowchart of a method of animating a subject using a photo-video volumetric capture system according to some embodiments.
- mesh creation/generation is implemented using the integrated volumetric photo-video system.
- the mesh generation includes extreme pose modeling and registration for blending.
- the integrated photo-video volumetric capture system for 3D/4D scan acquires 3D scans and 4D scans by acquiring images and videos of a subject/actor simultaneously.
- the 3D scans are able to be used to generate auto high-fidelity extreme poses, and the 4D scans include high temporal resolution which enables mesh tracking to automatically register extreme pose meshes for blending.
- skeleton fitting is implemented.
- Skeleton fitting is able to be implemented in any manner such as based on relative marker trajectories.
- skin weight painting is performed. Skin weight painting is able to be implemented in any manner such as determining the weight of each segment of skin and painting accordingly.
- animation is performed. Animation is able to be performed in any manner.
- each of the steps is able to be performed manually, semi-automatically or automatically. In some embodiments, fewer or additional steps are implemented. In some embodiments, the order of the steps is modified.
- Figure 2 illustrates a diagram of a mesh generated by combining a neutral pose and extreme poses according to some embodiments.
- a neutral pose is able to be any standard pose such as standing with arms down, arms up or arms out to the side.
- Extreme poses are the poses between standard poses such as when a subject moves between standard poses. Extreme poses are captured by targeting specific parts of the human muscle, which enables generation of the extreme shape for the game development pipeline.
- the photo-video system and mesh tracking are able to be used to target all muscle groups of the human body to capture and solve the problem of maintaining a mesh registration in the graphics game development pipeline.
- a model is captured for the game.
- An actor typically comes in to a studio one time to be recorded performing specified movements and/or actions.
- the studio comprehensively captures all of the actor’s muscle deformations using the photo-video volumetric capture system.
- a corresponding mesh is able to have similar deformations.
- a system is able to deform the model to be similar to human movements/deformations.
- the kinesiology movements, deformations and/or other knowledge and data are able to be used in training the system.
- Figure 3 illustrates a diagram of the correlation between human anatomy verus computer graphics according to some embodiments.
- musculoskeletal actuation involves receiving a signal from a person’s motor cortex. Then, muscle deformation occurs which enables joint / bone movement by the muscle pulling on the bone. Additionally, there is skin/fat movement.
- a motion driver triggers movement in an animated character, specifically by performing joint/bone movement.
- Mesh deformation Small Subspace Deformation (SSD)
- SSD Surface Subspace Deformation
- PSD Pose Space Deformation
- Figures 4A-B illustrate diagrams of muscle movements according to some embodiments.
- Human body parts bend at joints as shown such as the head bending at the neck, hands bending at the wrist, fingers bending at knuckles, legs bending at the knee, and feet bending at the ankle.
- all joint movements are able to be fit into 12 categories.
- the correct muscle deformation is able to be generated based on the classified movement. For example, when a character bends at the knee, specific muscles deform in the leg, and using machine learning, the correct muscles are able to be deformed at the appropriate time.
- the muscle movements are the types of movements the actor will perform including the range of motion.
- the muscle movements are targeted for capture.
- Figure 5 illustrates examples of major muscle groups according to some embodiments.
- the upper body and the lower body each have 4 joints (excluding finger/toe joints).
- the joints in the upper body include: the shoulder, elbow, neck and hands, and the joints in the lower body include: the torso, hips, knees and ankles.
- Each of the joints have corresponding muscle groups. As described, these corresponding muscle groups deform when the character is in motion.
- the lower body and upper body muscles are the main targets for capture when an actor is moving.
- Figure 6 illustrates a diagram of move types based on joints for mesh capture according to some embodiments.
- move types There are many different move types with varying angular ranges of motion (0 to 180 degrees) for each of the main upper and lower joints.
- the desired muscles are able to be captured and then later utilized when generating the mesh.
- Figure 7 illustrates a diagram of move types based on joints for mesh capture according to some embodiments. Two of the 12 move types are shown (flexion/extension and pronation/supination). In some embodiments, the angular range of motion is selectable from 0,
- finer tuning of the angular range of motion is possible to a specific number of degrees or even fractions of degrees.
- Figure 8 illustrates examples of extreme poses according to some embodiments.
- Image 800 shows six movement types such as lifting arms up to the side, raising arms from down at the hips to over head and putting arms out front.
- Image 802 shows the four joints and the target muscles.
- FIG. 9 illustrates a diagram of automatic blendshape extraction according to some embodiments.
- Pose parameters 900 combined with facial action units 902 result in the 4D tracked meshes 904.
- An automatic blendshape extraction method uses 4D scans of a moving face which expedites the character making process and reduces the production cost.
- a 4D face scan method such as U. S. Patent Application No. 17/411,432, filed August 25, 2021, titled, “PRESERVING GEOMETRY DETAILS IN A SEQUENCE OF TRACKED MESHES,” which is hereby incorporated by reference in its entirety for all purposes, is able to be used.
- Facial action units are of interest. With 4D tracked meshes including a variety of different expressions available, a set of character-specific facial action units are able to be automatically generated. It can be regarded as a decomposition of 4D meshes into dynamic pose parameters and static action units, where only the action units are unknown. Machine learning techniques for the decomposition problem are able to be used.
- FIG. 10 illustrates a flowchart of implementing mesh generation according to some embodiments.
- a volumetric capture system is utilized for high-quality 3D/4D scanning.
- the volumetric capture system is able to acquire photos and videos simultaneously for high-quality 3D/4D scanning.
- the high-quality 3D/4D scanning includes denser camera views for high-quality modeling.
- another system for acquiring 3D content and time information is utilized instead of utilizing the volumetric capture system. For example, at least two separate 3D scans are acquired. Furthering the example, the separate 3D scans are able to be captured and/or downloaded.
- joint and muscle movement and deformation are acquired.
- the specific muscles and the specific deformation of the muscles over time are captured.
- Specific joints and the corresponding muscles to the joints of the actor are able to be targeted during capture time.
- the target subject/actor is able to be requested to move, and the muscles will deform.
- the deformation of the muscles is able to be captured statically and in motion.
- the information acquired from the movement and deformation is able to be used to train a system so that the system is able to use the joint and muscle information to perform any movement of the character. For a very complex situation, this is very difficult for an animator to do. Any complex muscle deformation is learned during the modeling stage. This enables synthesis in the animation stage.
- mesh generation is implemented. Once high quality information is captured for the scanning, mesh generation is implemented including extreme pose modeling and registration for blending.
- the 3D scan information is able to be used to generate auto high- fidelity extreme poses.
- the frames between key frames are able to be properly generated using the 4D scan information which includes frame information between key frames.
- the high temporal resolution of the 4D scan information enables mesh tracking to auto register extreme pose meshes for blending.
- the 4D scan enables mesh generation of a muscle deforming over time.
- a mesh including muscle deformation information is able to be generated where the movement was not acquired by the capture system.
- mesh generation includes muscle estimation or projection based on the 3D scanning and 4D scanning and machine learning.
- Major upper body and lower body joints are able to be identified that are important for generating deformation and capturing deformation using a wide range of motion for all movement types across all joint categories.
- the topology changes are able to be tracked.
- each pose captured will have the same topology which makes blending between multiple poses easier and more accurate.
- the targeted joints and muscles are able to be utilized when generating the mesh.
- mesh generation includes generating a static mesh based on the 3D scan information, and the mesh is able to be modified/animated using the 4D scan information. For example, as the mesh moves in time, additional mesh information is able to be established/generated from the video content of the 4D scan information and/or machine learning information. As described, the transitions between each frame of the animated mesh are able to maintain topology, such that the mesh tracking and blending is smooth. In other words, topology correspondences are established across a 4D scanned mesh sequence for generating corrective shapes which will be used in shape interpolation and skeleton driven deformation.
- fewer or additional steps are implemented. In some embodiments, the order of the steps is modified.
- Figure 11 illustrates a block diagram of an exemplary computing device configured to implement the automatic blending method according to some embodiments.
- the computing device 1100 is able to be used to acquire, store, compute, process, communicate and/or display information such as images and videos.
- the computing device 1100 is able to implement any of the automatic blending aspects.
- a hardware structure suitable for implementing the computing device 1100 includes a network interface 1102, a memory 1104, a processor 1106, FO device(s) 1108, a bus 1110 and a storage device 1112.
- the choice of processor is not critical as long as a suitable processor with sufficient speed is chosen.
- the memory 1104 is able to be any conventional computer memory known in the art.
- the storage device 1112 is able to include a hard drive, CDROM, CDRW, DVD, DVDRW, High Definition disc/drive, ultra-HD drive, flash memory card or any other storage device.
- the computing device 1100 is able to include one or more network interfaces 1102. An example of a network interface includes a network card connected to an Ethernet or other type of LAN.
- the FO device(s) 1108 are able to include one or more of the following: keyboard, mouse, monitor, screen, printer, modem, touchscreen, button interface and other devices.
- Automatic blending application(s) 1130 used to implement the automatic blending method are likely to be stored in the storage device 1112 and memory 1104 and processed as applications are typically processed. More or fewer components shown in Figure 11 are able to be included in the computing device 1100.
- automatic blending hardware 1120 is included.
- the computing device 1100 in Figure 11 includes applications 1130 and hardware 1120 for the automatic blending method, the automatic blending method is able to be implemented on a computing device in hardware, firmware, software or any combination thereof.
- the automatic blending applications 1130 are programmed in a memory and executed using a processor.
- the automatic blending hardware 1120 is programmed hardware logic including gates specifically designed to implement the automatic blending method.
- the automatic blending application(s) 1130 include several applications and/or modules.
- modules include one or more sub-modules as well. In some embodiments, fewer or additional modules are able to be included.
- suitable computing devices include a personal computer, a laptop computer, a computer workstation, a server, a mainframe computer, a handheld computer, a personal digital assistant, a cellular/mobile telephone, a smart appliance, a gaming console, a digital camera, a digital camcorder, a camera phone, a smart phone, a portable music player, a tablet computer, a mobile device, a video player, a video disc writer/player (e.g., DVD writer/player, high definition disc writer/player, ultra high definition disc writer/player), a television, a home entertainment system, an augmented reality device, a virtual reality device, smart jewelry (e.g., smart watch), a vehicle (e.g., a self-driving vehicle) or any other suitable computing device.
- a personal computer e.g., a laptop computer, a computer workstation, a server, a mainframe computer, a handheld computer, a personal digital assistant, a cellular/mobile telephone, a smart appliance, a gaming console
- the automatic blending method is able to be implemented with user assistance or automatically without user involvement to perform automatic blending.
- the automatic blending method provides a more accurate and efficient automatic blending and animation method.
- the automatic blending method utilizes a photo video system which aids mesh-tracking for maintaining mesh registration (topology consistency) along with ease of extreme pose modeling unlike hand-crafted shape modeling which aids registration but has manual shape generation and the 3D scanning-based approach which aids shape generation but not registration.
- the topology changes are able to be tracked.
- each pose captured will have the same topology which makes blending between multiple poses easier and more accurate.
- a method programmed in a non-transitory of a device comprising: using a volumetric capture system configured for 3D scanning and 4D scanning including capturing photos and video simultaneously, wherein the 3D scanning and 4D scanning includes detecting muscle deformation of an actor; and implementing mesh generation based on the 3D scanning and 4D scanning.
- An apparatus comprising: a non-transitory memory for storing an application, the application for: using a volumetric capture system configured for 3D scanning and 4D scanning including capturing photos and video simultaneously, wherein the 3D scanning and 4D scanning includes detecting muscle deformation of an actor; and implementing mesh generation based on the 3D scanning and 4D scanning; and a processor coupled to the memory, the processor configured for processing the application.
- generating automatic high-fidelity extreme poses includes using 3D scans of the actor and muscle deformation of the actor to generate the automatic high-fidelity extreme poses.
- mesh generation includes muscle estimation or projection based on the 3D scanning and 4D scanning and machine learning.
- a system comprising: a volumetric capture system for 3D and 4D scanning including capturing photos and video simultaneously, wherein the 3D scanning and 4D scanning includes detecting muscle deformation of an actor; and a computing device configured for: receiving the captured photos and video from the volumetric capture system; and implementing mesh generation based on the 3D scanning and 4D scanning.
- volumetric capture system is further configured for identifying and targeting joints and muscles of the actor by the volumetric capture system for 3D scanning and 4D scanning.
- implementing mesh generation includes using the 3D scanning and 4D scanning to generate meshes in extreme poses including muscle deformation.
- the volumetric capture system is further configured for implementing mesh tracking for tracking topology changes to enable each pose captured to have a same topology for blending between poses.
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Multimedia (AREA)
- Computer Graphics (AREA)
- Geometry (AREA)
- Software Systems (AREA)
- Human Computer Interaction (AREA)
- Processing Or Creating Images (AREA)
- Image Generation (AREA)
Abstract
Description
Claims
Priority Applications (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
EP22715391.3A EP4292052A1 (en) | 2021-04-01 | 2022-03-31 | Automatic blending of human facial expression and full-body poses for dynamic digital human model creation using integrated photo-video volumetric capture system and mesh-tracking |
JP2023560157A JP2024513024A (en) | 2021-04-01 | 2022-03-31 | Automatic blending of facial expressions and full-body poses for dynamic digital human model creation using an integrated photo/video volumetric capture system and mesh tracking |
CN202280007210.6A CN116529766A (en) | 2021-04-01 | 2022-03-31 | Automatic mixing of human facial expressions and whole-body gestures for dynamic digital mannequin creation using integrated photo-video volume capture system and mesh tracking |
KR1020237022480A KR20230116902A (en) | 2021-04-01 | 2022-03-31 | Automatic blending of human facial expressions and full body poses to create a dynamic digital human model using integrated photo-video volumetric capture system and mesh-tracking |
Applications Claiming Priority (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US202163169323P | 2021-04-01 | 2021-04-01 | |
US63/169,323 | 2021-04-01 | ||
US17/706,996 US12033281B2 (en) | 2022-03-29 | Automatic blending of human facial expression and full-body poses for dynamic digital human model creation using integrated photo-video volumetric capture system and mesh-tracking | |
US17/706,996 | 2022-03-29 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2022208442A1 true WO2022208442A1 (en) | 2022-10-06 |
Family
ID=81326170
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/IB2022/053036 WO2022208442A1 (en) | 2021-04-01 | 2022-03-31 | Automatic blending of human facial expression and full-body poses for dynamic digital human model creation using integrated photo-video volumetric capture system and mesh-tracking |
Country Status (4)
Country | Link |
---|---|
EP (1) | EP4292052A1 (en) |
JP (1) | JP2024513024A (en) |
KR (1) | KR20230116902A (en) |
WO (1) | WO2022208442A1 (en) |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20080180448A1 (en) * | 2006-07-25 | 2008-07-31 | Dragomir Anguelov | Shape completion, animation and marker-less motion capture of people, animals or characters |
WO2019017985A1 (en) * | 2017-07-18 | 2019-01-24 | Sony Corporation | Robust mesh tracking and fusion by using part-based key frames and priori model |
WO2020132631A1 (en) * | 2018-12-20 | 2020-06-25 | Sony Corporation | Photo-video based spatial-temporal volumetric capture system |
US20210304478A1 (en) * | 2020-03-31 | 2021-09-30 | Sony Corporation | Volumetric capture and mesh-tracking based machine learning 4d face/body deformation training |
-
2022
- 2022-03-31 KR KR1020237022480A patent/KR20230116902A/en unknown
- 2022-03-31 EP EP22715391.3A patent/EP4292052A1/en active Pending
- 2022-03-31 WO PCT/IB2022/053036 patent/WO2022208442A1/en active Application Filing
- 2022-03-31 JP JP2023560157A patent/JP2024513024A/en active Pending
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20080180448A1 (en) * | 2006-07-25 | 2008-07-31 | Dragomir Anguelov | Shape completion, animation and marker-less motion capture of people, animals or characters |
WO2019017985A1 (en) * | 2017-07-18 | 2019-01-24 | Sony Corporation | Robust mesh tracking and fusion by using part-based key frames and priori model |
WO2020132631A1 (en) * | 2018-12-20 | 2020-06-25 | Sony Corporation | Photo-video based spatial-temporal volumetric capture system |
US20210304478A1 (en) * | 2020-03-31 | 2021-09-30 | Sony Corporation | Volumetric capture and mesh-tracking based machine learning 4d face/body deformation training |
Non-Patent Citations (3)
Title |
---|
DRAGOMIR ANGUELOV ET AL: "SCAPE", ACM TRANSACTIONS ON GRAPHICS, ACM, NY, US, vol. 24, no. 3, 1 July 2005 (2005-07-01), pages 408 - 416, XP058365613, ISSN: 0730-0301, DOI: 10.1145/1073204.1073207 * |
NEUMANN T ET AL: "Capture and Statistical Modeling of Arm-Muscle Deformations", COMPUTER GRAPHICS FORUM : JOURNAL OF THE EUROPEAN ASSOCIATION FOR COMPUTER GRAPHICS, WILEY-BLACKWELL, OXFORD, vol. 32, 6 May 2013 (2013-05-06), pages 285 - 294, XP071488014, ISSN: 0167-7055, DOI: 10.1111/CGF.12048 * |
THOMAS NEUMANN ET AL: "Sparse localized deformation components", ACM TRANSACTIONS ON GRAPHICS, ACM, NY, US, vol. 32, no. 6, 1 November 2013 (2013-11-01), pages 1 - 10, XP058033911, ISSN: 0730-0301, DOI: 10.1145/2508363.2508417 * |
Also Published As
Publication number | Publication date |
---|---|
KR20230116902A (en) | 2023-08-04 |
EP4292052A1 (en) | 2023-12-20 |
JP2024513024A (en) | 2024-03-21 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN109636831B (en) | Method for estimating three-dimensional human body posture and hand information | |
US10417818B2 (en) | Method for providing a three dimensional body model | |
Achenbach et al. | Fast generation of realistic virtual humans | |
Wei et al. | Videomocap: Modeling physically realistic human motion from monocular video sequences | |
CN110147737B (en) | Method, apparatus, device and storage medium for generating video | |
KR102577966B1 (en) | Photo-video based spatial-temporal volumetric capture system | |
US20230230304A1 (en) | Volumetric capture and mesh-tracking based machine learning 4d face/body deformation training | |
CN112734632A (en) | Image processing method, image processing device, electronic equipment and readable storage medium | |
US12020374B2 (en) | Body shape and pose estimation via volumetric regressor for raw three dimensional scan models | |
JP6555755B2 (en) | Image processing apparatus, image processing method, and image processing program | |
US20230419583A1 (en) | Methods and systems for markerless facial motion capture | |
Liu et al. | Animatable 3D Gaussian: Fast and High-Quality Reconstruction of Multiple Human Avatars | |
US12033281B2 (en) | Automatic blending of human facial expression and full-body poses for dynamic digital human model creation using integrated photo-video volumetric capture system and mesh-tracking | |
US20220319114A1 (en) | Automatic blending of human facial expression and full-body poses for dynamic digital human model creation using integrated photo-video volumetric capture system and mesh-tracking | |
Kang et al. | Real-time animation and motion retargeting of virtual characters based on single rgb-d camera | |
WO2022208442A1 (en) | Automatic blending of human facial expression and full-body poses for dynamic digital human model creation using integrated photo-video volumetric capture system and mesh-tracking | |
CN116529766A (en) | Automatic mixing of human facial expressions and whole-body gestures for dynamic digital mannequin creation using integrated photo-video volume capture system and mesh tracking | |
Wang et al. | Markerless body motion capturing for 3d character animation based on multi-view cameras | |
Robertini et al. | Capture of arm-muscle deformations using a depth-camera | |
Liu et al. | GEA: Reconstructing Expressive 3D Gaussian Avatar from Monocular Video | |
Ferrari et al. | 3DMM for accurate reconstruction of depth data | |
JP7134260B2 (en) | Information processing equipment | |
US20240169670A1 (en) | Three-dimensional mesh generator based on two-dimensional image | |
US20240127539A1 (en) | Mechanical weight index maps for mesh rigging | |
Zheng et al. | OHTA: One-shot Hand Avatar via Data-driven Implicit Priors |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 22715391 Country of ref document: EP Kind code of ref document: A1 |
|
WWE | Wipo information: entry into national phase |
Ref document number: 202280007210.6 Country of ref document: CN |
|
ENP | Entry into the national phase |
Ref document number: 20237022480 Country of ref document: KR Kind code of ref document: A |
|
WWE | Wipo information: entry into national phase |
Ref document number: 2022715391 Country of ref document: EP |
|
WWE | Wipo information: entry into national phase |
Ref document number: 2023560157 Country of ref document: JP |
|
ENP | Entry into the national phase |
Ref document number: 2022715391 Country of ref document: EP Effective date: 20230914 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |