CN117115398A - Virtual-real fusion digital twin fluid phenomenon simulation method - Google Patents
Virtual-real fusion digital twin fluid phenomenon simulation method Download PDFInfo
- Publication number
- CN117115398A CN117115398A CN202311014639.0A CN202311014639A CN117115398A CN 117115398 A CN117115398 A CN 117115398A CN 202311014639 A CN202311014639 A CN 202311014639A CN 117115398 A CN117115398 A CN 117115398A
- Authority
- CN
- China
- Prior art keywords
- scene
- fluid
- real
- dimensional
- indoor scene
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 239000012530 fluid Substances 0.000 title claims abstract description 64
- 238000000034 method Methods 0.000 title claims abstract description 36
- 238000004088 simulation Methods 0.000 title claims abstract description 28
- 230000004927 fusion Effects 0.000 title claims abstract description 21
- 210000000988 bone and bone Anatomy 0.000 claims abstract description 20
- 230000008447 perception Effects 0.000 claims abstract description 8
- 238000012545 processing Methods 0.000 claims abstract description 8
- 230000000295 complement effect Effects 0.000 claims abstract description 7
- 210000002411 hand bone Anatomy 0.000 claims abstract description 6
- 238000010276 construction Methods 0.000 claims abstract description 4
- 230000008569 process Effects 0.000 claims abstract description 4
- 239000002245 particle Substances 0.000 claims description 20
- 230000006870 function Effects 0.000 claims description 8
- 239000011159 matrix material Substances 0.000 claims description 8
- 238000013519 translation Methods 0.000 claims description 8
- 238000013527 convolutional neural network Methods 0.000 claims description 6
- 238000004364 calculation method Methods 0.000 claims description 5
- 238000005457 optimization Methods 0.000 claims description 5
- 230000009471 action Effects 0.000 claims description 4
- 238000004422 calculation algorithm Methods 0.000 claims description 4
- 230000014509 gene expression Effects 0.000 claims description 4
- 238000005286 illumination Methods 0.000 claims description 4
- 230000008859 change Effects 0.000 claims description 2
- 238000006073 displacement reaction Methods 0.000 claims description 2
- 238000001914 filtration Methods 0.000 claims description 2
- 238000003062 neural network model Methods 0.000 claims description 2
- 230000005855 radiation Effects 0.000 claims description 2
- 239000007787 solid Substances 0.000 claims description 2
- 238000012549 training Methods 0.000 claims description 2
- 238000011160 research Methods 0.000 abstract description 7
- 238000007654 immersion Methods 0.000 abstract description 3
- 230000003993 interaction Effects 0.000 description 13
- 230000002452 interceptive effect Effects 0.000 description 4
- 238000009877 rendering Methods 0.000 description 3
- 238000010586 diagram Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
- G06T19/006—Mixed reality
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T15/00—3D [Three Dimensional] image rendering
- G06T15/10—Geometric effects
- G06T15/20—Perspective computation
- G06T15/205—Image-based rendering
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T15/00—3D [Three Dimensional] image rendering
- G06T15/50—Lighting effects
- G06T15/506—Illumination models
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/50—Depth or shape recovery
- G06T7/55—Depth or shape recovery from multiple images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
- G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
- G06T7/75—Determining position or orientation of objects or cameras using feature-based methods involving models
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
- G06V10/443—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components by matching or filtering
- G06V10/449—Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters
- G06V10/451—Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters with interaction between the filter responses, e.g. cortical complex cells
- G06V10/454—Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/46—Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features
- G06V10/462—Salient features, e.g. scale invariant feature transforms [SIFT]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/20—Scenes; Scene-specific elements in augmented reality scenes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/35—Categorising the entire scene, e.g. birthday party or wedding scene
- G06V20/36—Indoor scenes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/60—Type of objects
- G06V20/64—Three-dimensional objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10024—Color image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10028—Range image; Depth image; 3D point clouds
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30244—Camera pose
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2210/00—Indexing scheme for image generation or computer graphics
- G06T2210/24—Fluid dynamics
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2210/00—Indexing scheme for image generation or computer graphics
- G06T2210/56—Particle system, point based geometry or rendering
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Computer Graphics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Human Computer Interaction (AREA)
- Software Systems (AREA)
- Geometry (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- Health & Medical Sciences (AREA)
- Computing Systems (AREA)
- Biodiversity & Conservation Biology (AREA)
- Biomedical Technology (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computer Hardware Design (AREA)
- Processing Or Creating Images (AREA)
Abstract
The embodiment of the disclosure discloses a digital twin fluid phenomenon simulation method for virtual-real fusion. One embodiment of the method comprises the following steps: collecting an indoor scene depth map and an indoor scene RGB map, and carrying out complement processing on the indoor scene depth map according to the indoor scene RGB map; reconstructing the three-dimensional scene point cloud and determining the point cloud semantics of the three-dimensional scene point cloud; based on physical perception, carrying out reverse twin construction on the three-dimensional fluid scene to generate a twin fluid scene; continuously and updated calculating a tracking frame data set according to the human body trunk bone motion and the hand bone motion; the method comprises the steps of controlling a sensor to measure a real scene, obtaining real environment color information, and establishing a plurality of virtual light sources in the twin fluid scene to simulate the real light field information so as to display the digital twin fluid scene. The implementation method ensures the standardization of the research process and the accessibility of the expected research targets, and improves the realism and immersion of the mixed scene.
Description
Technical Field
The embodiment of the disclosure relates to the technical field of fluid simulation and interaction based on mixed reality, in particular to a digital twin fluid phenomenon simulation method based on virtual-real fusion.
Background
Traditional fluid simulation and interactive application based on computer graphics are often completely carried out in a virtual simulation environment, and do not have the ability of sensing and fusing a real scene; meanwhile, the simulation application mostly adopts an off-line calculation mode to render output sequence images and generate animations, which are limited by the solving efficiency and rendering precision of a physical system, and has weaker interactive feedback and control capability for users.
Compared with a fine geometric structure of a re-carved scene, the fluid interaction scene twin reconstruction is more focused on the real-time performance and robustness of the system, the simulation boundary geometric features need to be extracted in a complex scene in real time rapidly, and the stable operation of the tracking system is maintained under the conditions of low texture, variable illumination and rapid movement of a sensor, so that the problems need to be solved; in addition, how to quickly and accurately track and extract interactive plane structures and even semantic information in reconstruction is an important task for supporting more realistic and intelligent fluid simulation application. Reproducing fluids consistent with real scene behavior and enabling physics-based modeling and evolution in a mixed reality environment still faces a series of problems and challenges. Firstly, capturing fluid data of a real scene by adopting any means and acquiring data, wherein the data is required to be determined according to the actual application scene; secondly, the limitation of high time consumption and low precision of the speed field inferred from the continuous surface geometry or the continuous density field by the traditional iterative optimization method is broken, namely, the real-time recovery of all flow field data from the acquired data is a challenging problem; finally, how to quickly obtain the physical attribute affecting the fluid behavior from the observed data, and further support the simulation of the dynamic evolution of the fluid based on the physical, so as to meet the requirements of the virtual fluid on the perception and fusion of the real environment, and further research is needed. In terms of real-time virtual-real fusion interaction based on physical reality, how to solve close-range real-time interaction between a user and a scene boundary and a virtual object and how to feed back interaction force of the virtual scene to the user is also an important problem to further increase immersion and reality of user experience. The real-time rendering problem and the challenge of the large-scale fluid in the virtual-real fusion environment are simultaneously reflected in the scene information acquisition of the environment and the speed and the quality of the fluid rendering. Although there is a great deal of research currently being done to determine the light field in a real scene, a complete global illumination distribution requires the establishment of a close association of the light field and objects in the scene, and how to combine digital twinning of a three-dimensional scene and light field information acquisition in a better way is a key issue.
The development of virtual-real fusion technology has higher requirements on the space-time consistency of a virtual scene and the natural interaction of a virtual object, so that the physical simulation application facing the mixed reality scene has the capabilities of fast sensing and reconstructing the boundary of a simulation environment, accurately acquiring the attribute of real fluid, generating dynamic fluid, interacting and timely feeding back the interactive and evolvable virtual fluid and the real user behavior. The three-dimensional scene rapid reconstruction method based on the depth image is used for achieving three-dimensional scene rapid reconstruction based on the virtual-real fusion fluid simulation, fluid parameter acquisition and evolution simulation based on physical perception, virtual-real fusion man-machine interaction and control three-dimensional research contents, and a typical demonstration application scene is constructed based on the three-dimensional scene rapid reconstruction.
Disclosure of Invention
The disclosure is in part intended to introduce concepts in a simplified form that are further described below in the detailed description. The disclosure is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.
Some embodiments of the present disclosure propose digital twin fluid phenomenon simulation methods of virtual-real fusion to solve one or more of the technical problems mentioned in the background section above.
The technical problem that this disclosure solved is: three-dimensional scene based on depth image is quickly reconstructed to obtain real space information, scene boundaries are provided for virtual fluid simulation and interaction, and virtual fluid scene modeling based on real space features is realized; the reality and the efficiency of the simulation are improved by acquiring and evolving simulation based on the physical perception fluid parameters; the virtual-real fusion man-machine interaction control realizes the interaction between the virtual fluid and the real user by capturing the action information of the user and feeding back the action information to the virtual fluid, constructs virtual-real through user-scene-physical phenomenon virtual-real fusion interaction, and improves the experience of the virtual-real interaction.
Some embodiments of the present disclosure provide a digital twin fluid phenomenon simulation method of virtual-real fusion, including: collecting an indoor scene depth map and an indoor scene RGB map, and carrying out complement processing on the indoor scene depth map according to the indoor scene RGB map so as to generate a complement indoor scene depth map; reconstructing the three-dimensional scene point cloud according to the indoor scene RGB image, and determining the point cloud semantics of the three-dimensional scene point cloud; based on physical perception, carrying out reverse twin construction on the three-dimensional fluid scene to generate a twin fluid scene; continuously and updated calculating a tracking frame data set according to the human body trunk bone motion and the hand bone motion; the method comprises the steps of controlling a sensor to measure a real scene, obtaining real environment color information, and establishing a plurality of virtual light sources in the twin fluid scene to simulate the real light field information so as to display the digital twin fluid scene.
The above embodiments of the present disclosure have the following advantages: through the digital twin fluid phenomenon simulation method of virtual-real fusion of some embodiments of the present disclosure, standardability of a research process and accessibility of an expected research target are guaranteed to the greatest extent, realism and immersion of a mixed scene are improved, and man-machine interaction with abundant scenes, vivid details, real-time and high efficiency is realized.
Drawings
The above and other features, advantages, and aspects of embodiments of the present disclosure will become more apparent by reference to the following detailed description when taken in conjunction with the accompanying drawings. The same or similar reference numbers will be used throughout the drawings to refer to the same or like elements. It should be understood that the figures are schematic and that elements and components are not necessarily drawn to scale.
Fig. 1 is a flow chart of some embodiments of a digital twinning fluid phenomenon simulation method of virtual-actual fusion according to the present disclosure.
Fig. 2 is a flow chart of fluid parameter acquisition and evolution simulation of some embodiments of a virtual-real fusion digital twin fluid phenomenon simulation method according to the present disclosure.
Detailed Description
Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While certain embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete. It should be understood that the drawings and embodiments of the present disclosure are for illustration purposes only and are not intended to limit the scope of the present disclosure.
It should be noted that, for convenience of description, only the portions related to the present invention are shown in the drawings. Embodiments of the present disclosure and features of embodiments may be combined with each other without conflict.
It should be noted that the terms "first," "second," and the like in this disclosure are merely used to distinguish between different devices, modules, or units and are not used to define an order or interdependence of functions performed by the devices, modules, or units.
It should be noted that references to "one", "a plurality" and "a plurality" in this disclosure are intended to be illustrative rather than limiting, and those of ordinary skill in the art will appreciate that "one or more" is intended to be understood as "one or more" unless the context clearly indicates otherwise.
The names of messages or information interacted between the various devices in the embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.
The present disclosure will be described in detail below with reference to the accompanying drawings in conjunction with embodiments.
Fig. 1 illustrates a flow 100 of some embodiments of a digital twin fluid phenomenon simulation method of virtual-actual fusion according to the present disclosure. The digital twin fluid phenomenon simulation method based on virtual-real fusion comprises the following steps:
step 101, collecting an indoor scene depth map and an indoor scene RGB map, and performing complement processing on the indoor scene depth map according to the indoor scene RGB map to generate a complement indoor scene depth map.
Continuously acquiring an indoor scene depth map and an indoor scene RGB map of an indoor scene, taking effective depth information on the indoor scene depth map as priori, utilizing geometric range constraint provided by color information displayed by the indoor scene RGB map, and fusing features into an image reconstruction process of a self-encoder through a convolutional neural network to generate a complete and complemented indoor scene depth map so as to improve the depth information quality.
And 102, reconstructing the three-dimensional scene point cloud according to the indoor scene RGB diagram, and determining the point cloud semantics of the three-dimensional scene point cloud.
Extracting a neighborhood range of ORB (Oriented FAST and Rotated BRIEF) features and ORB features which are reasonably distributed on an indoor scene RGB image by utilizing parallel calculation, and extracting high-dimensional features of multi-level RGB-D image pairs through a two-dimensional convolutional neural network;
acquiring depth information of feature points and neighborhood according to the depth map of the indoor scene after completion, carrying out matching association on adjacent frame features through the feature and neighborhood geometric distribution information, and then respectively calculating camera pose under the condition of 2D-3D and 3D-3D space feature matching through an N-point perspective (PnP method) method and a nearest point iteration (ICP method), wherein the optimization functions are as follows:
wherein u is i Is a three-dimensional feature point p i Two-dimensional coordinate values in the image coordinate system,is at different viewing angles with p i Associated three-dimensional features, K is a camera reference matrix, T * Namely the sensor pose obtained after optimization,is->2-norm square of (c);
extracting key image frames and predicting the pose under a world coordinate system according to the definition degree and the motion range of the images, and projecting key frame pixel points to a three-dimensional space through parallel calculation by using restored high-quality depth information to form dense point cloud;
the discrete expressions under different view angles are updated and fused in parallel by utilizing a voxel filtering mode, so that the reconstructed three-dimensional scene point cloud initial expression is obtained;
and encoding geometric-semantic features of the initial point cloud through the graph neural network model, and respectively predicting complete geometric information and point cloud semantic categories by using a geometric completion decoder and a semantic classification decoder.
Step 103, performing inverse twin construction on the three-dimensional fluid scene based on the physical perception to generate a twin fluid scene.
Sequential sequence h of fluid surface height field and 2D tag l distinguishing fluid from solid region s Splicing input stack self-encoder with layer jump connection, training network to output 2D velocity field u of fluid surface s The loss function adopts L 2 A norm;
combining the height field timing sequence h with the 2D tag l s Using convolutional neural networks G param From the surface velocity field u of the generated multiframe s {t,t+1,t+2} Medium estimated fluid viscosityNamely:
loss function uses L 1 Norms, at the same time, using 3D convolutional network G v Inferring internal 3D velocity field information from the surface 2D velocity field along the gravitational axis;
the final layer output of the network is dot multiplied by the obstacle mask so that the velocity in the non-fluid region is 0, creating a twinned fluid scene. A fluid parameter acquisition and evolution simulation flow diagram of some embodiments of the virtual-real fusion digital twin fluid phenomenon simulation method of the present disclosure is shown in fig. 2.
Step 104, continuously and updated calculating a tracking frame data set according to the human body trunk bone motion and the hand bone motion.
Continuously updating a tracking frame data set according to the motion of human body trunk bones and hand bones, wherein the tracking frame data in the tracking frame data set comprises a basic tracking data list, the basic tracking data list comprises a rotation matrix, a scaling factor and displacement which change among each tracking frame data, the position information, the speed information, the palm orientation, the palm ball radius and other hand tracking data of each frame of bones, and certain specific behavior actions and gestures are recognized according to multi-frame data;
processing the rotation of the bone particles, and calculating a rotation matrix relative to the center of the bone where the particles are positioned for each particle for each frame;
when the translation of the particles is processed, the speeds of bones of a human body trunk where the particles are positioned and each bone of a hand are respectively multiplied by different translation coefficients, the translation coefficients are larger as the bones are close to the far end, and conversely, the translation coefficients are smaller, and the movement formula of each particle is as follows:
v t+1 =v t ×ratio+(p t ×Rot-p t ),
wherein v is t For the speed of the particle at the t frame, ratio is the coefficient of the translational speed of the bone where the particle is located, p t Is the position of the particle at the t-th frame, rot is the rotation matrix of the particle.
Step 105, the control sensor measures the real scene to obtain the real environment color information, and establishes a plurality of virtual light sources in the twin fluid scene to simulate the real light field information so as to display the digital twin fluid scene.
The method comprises the steps that a sensor is controlled to measure a real scene to obtain real environment color information, wherein the real environment color information is light field information, and the light field information is stored in an image form;
processing the image by using an image space global illumination algorithm, searching for a region with high saturation in all channels, and dividing;
using a VPL (virtual point light) algorithm to find the outline of the radiation spot by using outline tracking;
calculating the spot area, and selecting a large spot as a light source;
and estimating the direction and the position of the incoming light according to the spot center position in the environment image, and establishing a plurality of virtual light sources in the scene to simulate the real light field information so as to display the digital twin fluid scene.
The foregoing description is only of the preferred embodiments of the present disclosure and description of the principles of the technology being employed. It will be appreciated by those skilled in the art that the scope of the invention in the embodiments of the present disclosure is not limited to the specific combination of the above technical features, but encompasses other technical features formed by any combination of the above technical features or their equivalents without departing from the spirit of the invention. Such as the above-described features, are mutually substituted with (but not limited to) the features having similar functions disclosed in the embodiments of the present disclosure.
Claims (6)
1. A digital twin fluid phenomenon simulation method of virtual-real fusion, comprising:
collecting an indoor scene depth map and an indoor scene RGB map, and carrying out complement processing on the indoor scene depth map according to the indoor scene RGB map so as to generate a complement indoor scene depth map;
reconstructing a three-dimensional scene point cloud according to the indoor scene RGB image, and determining the point cloud semantics of the three-dimensional scene point cloud;
based on physical perception, carrying out reverse twin construction on the three-dimensional fluid scene to generate a twin fluid scene;
continuously and updated calculating a tracking frame data set according to the human body trunk bone motion and the hand bone motion;
the method comprises the steps of controlling a sensor to measure a real scene, obtaining real environment color information, and establishing a plurality of virtual light sources in the twin fluid scene to simulate the real light field information so as to display the digital twin fluid scene.
2. The method of claim 1, wherein the acquiring the indoor scene depth map and the indoor scene RGB map, and the complementing the indoor scene depth map according to the indoor scene RGB map, to generate the complemented indoor scene depth map, comprises:
continuously acquiring an indoor scene depth map and an indoor scene RGB map of an indoor scene, taking effective depth information on the indoor scene depth map as priori, utilizing geometric range constraint provided by color information displayed by the indoor scene RGB map, and fusing features into an image reconstruction process of a self-encoder through a convolutional neural network to generate a complete and complemented indoor scene depth map so as to improve the depth information quality.
3. The method of claim 1, wherein the reconstructing a three-dimensional scene point cloud from the indoor scene RGB map and determining point cloud semantics of the three-dimensional scene point cloud comprises:
extracting ORB features with reasonable distribution and neighborhood ranges of the ORB features on an indoor scene RGB image by utilizing parallel calculation, and extracting high-dimensional features of multi-level RGB-D image pairs through a two-dimensional convolutional neural network;
acquiring depth information of feature points and neighborhood according to the depth map of the indoor scene after completion, carrying out matching association on adjacent frame features through the feature and neighborhood geometric distribution information, and then respectively calculating camera pose under the condition of 2D-3D and 3D-3D space feature matching through an N-point perspective method and a nearest point iteration method, wherein the optimization functions are as follows:
wherein u is i Is a three-dimensional feature point p i Two-dimensional coordinate values in the image coordinate system,is at different viewing angles with p i Associated three-dimensional features, K is a camera reference matrix, T * Namely the sensor pose obtained after optimization,is->2-norm square of (c);
extracting key image frames and predicting the pose under a world coordinate system according to the definition degree and the motion range of the images, and projecting key frame pixel points to a three-dimensional space through parallel calculation by using restored high-quality depth information to form dense point cloud;
the discrete expressions under different view angles are updated and fused in parallel by utilizing a voxel filtering mode, so that the reconstructed three-dimensional scene point cloud initial expression is obtained;
and encoding geometric-semantic features of the initial point cloud through the graph neural network model, and respectively predicting complete geometric information and point cloud semantic categories by using a geometric completion decoder and a semantic classification decoder.
4. The method of claim 1, wherein the reverse twinning the three-dimensional fluid scene based on the physical perception to generate a twinned fluid scene comprises:
sequential sequence h of fluid surface height field and 2D tag l distinguishing fluid from solid region s Splicing input stack self-encoder with layer jump connection, training network to output 2D velocity field u of fluid surface s The loss function adopts L 2 A norm;
combining the height field timing sequence h with the 2D tag l s Using convolutional neural networks G param From the surface velocity field u of the generated multiframe s {t,t+1,t+2} Medium estimated fluid viscosityNamely:
loss function uses L 1 Norms, at the same time, using 3D convolutional network G v Inferring internal 3D velocity field information from the surface 2D velocity field along the gravitational axis;
the final layer output of the network is dot multiplied by the obstacle mask so that the velocity in the non-fluid region is 0, creating a twinned fluid scene.
5. The method of claim 4, wherein the continuously updated computing tracking frame data sets from human torso skeletal motion and hand skeletal motion comprises:
continuously updating a tracking frame data set according to the motion of human body trunk bones and hand bones, wherein the tracking frame data in the tracking frame data set comprises a basic tracking data list, the basic tracking data list comprises a rotation matrix, a scaling factor and displacement which change among each tracking frame data, the position information, the speed information, the palm orientation, the palm ball radius and other hand tracking data of each frame of bones, and certain specific behavior actions and gestures are recognized according to multi-frame data;
processing the rotation of the bone particles, and calculating a rotation matrix relative to the center of the bone where the particles are positioned for each particle for each frame;
when the translation of the particles is processed, the speeds of bones of a human body trunk where the particles are positioned and each bone of a hand are respectively multiplied by different translation coefficients, the translation coefficients are larger as the bones are close to the far end, and conversely, the translation coefficients are smaller, and the movement formula of each particle is as follows:
v t+1 =v t ×ratio+(p t ×Rot-p t ),
wherein v is t For the speed of the particle at the t frame, ratio is the coefficient of the translational speed of the bone where the particle is located, p t Is the position of the particle at the t-th frame, rot is the rotation matrix of the particle.
6. The method of claim 1, wherein the controlling the sensor to measure the real scene, obtain real environment color information, and establish a number of virtual light sources in the twin fluid scene to simulate the real light field information for presentation of the digital twin fluid scene, comprises:
the method comprises the steps that a sensor is controlled to measure a real scene to obtain real environment color information, wherein the real environment color information is light field information, and the light field information is stored in an image form;
processing the image by using an image space global illumination algorithm, searching for a region with high saturation in all channels, and dividing;
using a VPL algorithm to find the outline of the radiation spot by using outline tracking;
calculating the spot area, and selecting a large spot as a light source;
and estimating the direction and the position of the incoming light according to the spot center position in the environment image, and establishing a plurality of virtual light sources in the scene to simulate the real light field information so as to display the digital twin fluid scene.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202311014639.0A CN117115398A (en) | 2023-08-11 | 2023-08-11 | Virtual-real fusion digital twin fluid phenomenon simulation method |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202311014639.0A CN117115398A (en) | 2023-08-11 | 2023-08-11 | Virtual-real fusion digital twin fluid phenomenon simulation method |
Publications (1)
Publication Number | Publication Date |
---|---|
CN117115398A true CN117115398A (en) | 2023-11-24 |
Family
ID=88803133
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202311014639.0A Pending CN117115398A (en) | 2023-08-11 | 2023-08-11 | Virtual-real fusion digital twin fluid phenomenon simulation method |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN117115398A (en) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN117809498A (en) * | 2024-01-09 | 2024-04-02 | 北京千乘科技有限公司 | Virtual-real interaction multidimensional twinning projection road network system |
-
2023
- 2023-08-11 CN CN202311014639.0A patent/CN117115398A/en active Pending
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN117809498A (en) * | 2024-01-09 | 2024-04-02 | 北京千乘科技有限公司 | Virtual-real interaction multidimensional twinning projection road network system |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
Chen et al. | A survey on 3d gaussian splatting | |
Li et al. | Vox-surf: Voxel-based implicit surface representation | |
EP3533218B1 (en) | Simulating depth of field | |
US11989900B2 (en) | Object recognition neural network for amodal center prediction | |
CN115222917A (en) | Training method, device and equipment for three-dimensional reconstruction model and storage medium | |
CN117115398A (en) | Virtual-real fusion digital twin fluid phenomenon simulation method | |
CN115115805A (en) | Training method, device and equipment for three-dimensional reconstruction model and storage medium | |
CN107018400B (en) | It is a kind of by 2D Video Quality Metrics into the method for 3D videos | |
CN115272608A (en) | Human hand reconstruction method and equipment | |
Ouyang et al. | Real-time neural character rendering with pose-guided multiplane images | |
Li et al. | Immersive neural graphics primitives | |
Wu et al. | [Retracted] 3D Film Animation Image Acquisition and Feature Processing Based on the Latest Virtual Reconstruction Technology | |
US20230290132A1 (en) | Object recognition neural network training using multiple data sources | |
Li et al. | Bringing instant neural graphics primitives to immersive virtual reality | |
CN117635801A (en) | New view synthesis method and system based on real-time rendering generalizable nerve radiation field | |
Liu et al. | Facial animation by optimized blendshapes from motion capture data | |
CN113673567B (en) | Panorama emotion recognition method and system based on multi-angle sub-region self-adaption | |
CN114255328A (en) | Three-dimensional reconstruction method for ancient cultural relics based on single view and deep learning | |
Yao et al. | Neural Radiance Field-based Visual Rendering: A Comprehensive Review | |
Huixuan et al. | Innovative Practice of Virtual Reality Technology in Animation Production | |
Liu | Light image enhancement based on embedded image system application in animated character images | |
Tian et al. | Research on Visual Design of Computer 3D Simulation Special Effects Technology in the Shaping of Sci-Fi Animation Characters | |
CA3143520C (en) | Method of computing simulated surfaces for animation generation and other purposes | |
US20240257375A1 (en) | Object recognition neural network for amodal center prediction | |
Baskar et al. | 3D Image Reconstruction and Processing for Augmented and Virtual Reality Applications: A Computer Generated Environment |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination |