CN112634439B - 3D information display method and device - Google Patents

3D information display method and device Download PDF

Info

Publication number
CN112634439B
CN112634439B CN202011566818.1A CN202011566818A CN112634439B CN 112634439 B CN112634439 B CN 112634439B CN 202011566818 A CN202011566818 A CN 202011566818A CN 112634439 B CN112634439 B CN 112634439B
Authority
CN
China
Prior art keywords
model
image
cloud data
test
target object
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202011566818.1A
Other languages
Chinese (zh)
Other versions
CN112634439A (en
Inventor
娄志云
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing QIYI Century Science and Technology Co Ltd
Original Assignee
Beijing QIYI Century Science and Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing QIYI Century Science and Technology Co Ltd filed Critical Beijing QIYI Century Science and Technology Co Ltd
Priority to CN202011566818.1A priority Critical patent/CN112634439B/en
Publication of CN112634439A publication Critical patent/CN112634439A/en
Application granted granted Critical
Publication of CN112634439B publication Critical patent/CN112634439B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/22—Matching criteria, e.g. proximity measures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/00—3D [Three Dimensional] image rendering
    • G06T15/04—Texture mapping
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G06T7/55—Depth or shape recovery from multiple images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2200/00—Indexing scheme for image data processing or generation, in general
    • G06T2200/08—Indexing scheme for image data processing or generation, in general involving all processing steps from image acquisition to 3D model generation

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Computer Graphics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Evolutionary Computation (AREA)
  • Evolutionary Biology (AREA)
  • General Engineering & Computer Science (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Artificial Intelligence (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Geometry (AREA)
  • Software Systems (AREA)
  • Processing Or Creating Images (AREA)

Abstract

The embodiment of the invention provides a 3D information display method and device, and relates to the technical field of data processing, wherein the method comprises the following steps: respective image groups of the target object are obtained. And reconstructing 3D models of the target object at different acquisition moments in the same fixed coordinate system according to the same fixed camera pose and fixed camera internal parameters according to each image group to obtain point cloud data of each 3D model. Under the condition that a 3D information display instruction aiming at the target object is received, according to each group of point cloud data, sequentially rendering 3D models corresponding to each acquisition time according to the sequence of the acquisition times corresponding to each group of point cloud data, and realizing 3D information display. By applying the scheme provided by the embodiment of the invention to display the 3D information, the dynamic information of the dynamic object can be reproduced through the 3D model.

Description

3D information display method and device
Technical Field
The invention relates to the technical field of data processing, in particular to a 3D information display method and device.
Background
The 2D image can record and reproduce information of a real object in the real world, but the 2D image can reproduce only two-dimensional plane information of the real object, and the information reproduction effect is poor. With the development of 3D technology, 3D models of real objects in the real world can be reconstructed by applying the 3D technology, three-dimensional stereo information of the real objects can be reproduced through the 3D models, and the information reproduction effect is good. Therefore, information reproducing a real object through a 3D model is widely used in VR (Virtual Reality) and other fields.
However, the prior art is often only capable of reconstructing a 3D model of an object in a 2D image based on the 2D image, and since the 2D image is a static image, only static information of the static object can be reproduced through the reconstructed 3D model. The real objects in the real world are often moving dynamic objects, e.g. moving people, animals, etc. And because the motion state of the dynamic object is controlled by the object itself and has randomness, a scheme for reproducing the dynamic object based on the 3D model needs to be provided.
Disclosure of Invention
The embodiment of the invention aims to provide a 3D information display method and device, so that dynamic information of a dynamic object is reproduced through a 3D model. The specific technical scheme is as follows:
in a first aspect of the embodiment of the present invention, there is first provided a 3D information display method, where the method includes:
obtaining each image group of a target object, wherein each image group comprises images which are acquired by a plurality of image acquisition devices at the same acquisition time from different acquisition angles and contain the target object, and the acquisition time corresponding to each image group is different;
reconstructing 3D models of the target object at different acquisition moments in the same fixed coordinate system according to the same fixed camera pose and fixed camera internal parameters according to each image group to obtain point cloud data of each 3D model, wherein the fixed camera pose is the simulation pose of a fixed simulation camera, the fixed camera internal parameters are the simulation internal parameters of the fixed simulation camera, and the acquisition range of the simulation camera is the union set of the acquisition ranges of each image acquisition device;
Under the condition that a 3D information display instruction aiming at the target object is received, according to each group of point cloud data, sequentially rendering 3D models corresponding to each acquisition time according to the sequence of the acquisition times corresponding to each group of point cloud data, and realizing 3D information display.
In one embodiment of the invention, the stationary coordinate system, stationary camera pose and stationary camera reference are obtained by:
selecting one reference image group from the image groups, and reconstructing a reference 3D model of the target object by using the parameter image groups;
and determining a coordinate system where the reconstructed reference 3D model is located and a simulation pose and a simulation internal reference used when the reference 3D model is reconstructed, wherein the simulation pose and the simulation internal reference are used as the fixed coordinate system, the fixed camera pose and the fixed camera internal reference.
In one embodiment of the present invention, reconstructing a 3D model of the target object at different acquisition moments in the same fixed coordinate system according to the same stationary camera pose and the same stationary camera internal reference according to each image group, including:
for each image group, reconstructing a 3D model of the target object corresponding to the image group by:
extracting the characteristics of each image in the image group to obtain characteristic pixel points in each image;
Determining matched pixels in different images corresponding to the same position on the target object based on first features of feature pixels in the respective images, wherein the first features include: pixel point positions of the pixel points and the first content sub-feature;
predicting second features of corresponding points of each matched pixel point in a 3D model in the fixed coordinate system according to the first features of the matched pixel points, the fixed camera pose and the fixed camera internal parameters, wherein the second features comprise: the three-dimensional position of the midpoint of the 3D model and the second content sub-feature;
obtaining depth information of each pixel point according to the pose of the stationary camera and the pixel coordinates of each pixel point in the participation image in the stationary camera;
and reconstructing a 3D model of the target object corresponding to the image group according to the second characteristics of the corresponding points of each matched pixel point in the 3D model, and the depth information and the first characteristics of each pixel point.
In one embodiment of the present invention, predicting the second feature of the corresponding point of each matched pixel in the 3D model in the fixed coordinate system according to the first feature of the matched pixel, the pose of the stationary camera and the internal reference of the stationary camera includes:
Predicting the three-dimensional position of the corresponding point of each matched pixel point in the 3D model in the fixed coordinate system according to the pixel point position in the first feature of each matched pixel point in the image, the position of the fixed camera and the internal reference of the fixed camera;
and for each matched pixel point, determining a second characteristic of the corresponding point of the matched pixel point in the 3D model according to the first content sub-characteristic in the first characteristic of the matched pixel point and the three-dimensional position of the corresponding point of the matched pixel point in the 3D model.
In one embodiment of the present invention, the obtaining depth information of each pixel point according to the pose of the stationary camera and the pixel coordinates of each pixel point in the participation image in the stationary camera includes:
performing image de-distortion treatment on each image to obtain a de-distorted image;
and obtaining depth information of each pixel point according to the pose of the stationary camera and the pixel coordinates of each pixel point in the image after the stationary camera participates in de-distortion.
In one embodiment of the present invention, reconstructing a 3D model of a target object corresponding to the image group according to the second feature of the corresponding point of each matched pixel point in the 3D model, the depth information of each pixel point, and the first feature, includes:
Generating a normal map corresponding to each image according to the depth information of each pixel point;
reconstructing a model shape of the 3D model based on the normal map and pixel locations in the first feature of the matching pixel points;
filling the content of the corresponding points of each pixel point in the model shape based on the first content sub-feature in the first feature of each pixel point;
and reconstructing a 3D model of the target object corresponding to the image group according to the model shape after the content filling.
In one embodiment of the present invention, before obtaining each image group of the target object, the method further includes:
obtaining each test image group of a test object, wherein each test image group comprises images containing the test object, which are acquired by a plurality of image acquisition devices at the same test acquisition time, and the test acquisition time corresponding to each test image group is different;
reconstructing test 3D models of the test object at different test acquisition moments according to each test image group to obtain test point cloud data of each test 3D model;
according to the cloud data of each group of test points, sequentially rendering a test 3D model corresponding to each test acquisition time according to the sequence of the test acquisition time corresponding to the cloud data of the test points, so as to realize 3D test information display;
If the displayed 3D test information is similar to the motion state of the test object, executing the step of obtaining each image group of the target object;
otherwise, the step of obtaining the respective test image groups of the test object is performed in return, in case it is determined that the position and/or orientation of the image acquisition device has changed.
In one embodiment of the present invention, according to each set of point cloud data, according to the sequence of the collection moments corresponding to each set of point cloud data, sequentially rendering the 3D model corresponding to each collection moment, including:
rendering a 3D model corresponding to the target point cloud data according to the target point cloud data, wherein the initial value of the target point cloud data is as follows: according to the sequence of the acquisition time of each group of point cloud data, the point cloud data positioned at the forefront end;
after displaying the rendered 3D model for a preset time, controlling the rendered 3D model to disappear;
if the current target point cloud data is not the point cloud data positioned at the rearmost end in the sequence of the acquisition time, determining the point cloud data positioned at the next position of the current target point cloud data in the sequence of the acquisition time as new target point cloud data, and returning to execute the step of rendering the 3D model corresponding to the target point cloud data according to the target point cloud data.
In a second aspect of the embodiment of the present invention, there is also provided a 3D information display apparatus, the apparatus including:
the image acquisition module is used for acquiring each image group of the target object, wherein each image group comprises a plurality of images which are acquired by the image acquisition equipment at the same acquisition time from different acquisition angles and contain the target object, and the acquisition time corresponding to each image group is different;
the data acquisition module is used for reconstructing 3D models of the target object at different acquisition moments in the same fixed coordinate system according to the same fixed camera pose and fixed camera internal parameters according to each image group to obtain point cloud data of each 3D model, wherein the fixed camera pose is the simulation pose of a fixed simulation camera, the fixed camera internal parameters are the simulation internal parameters of the fixed simulation camera, and the acquisition range of the simulation camera is the union of the acquisition ranges of each image acquisition device;
the information display module is used for sequentially rendering the 3D models corresponding to the acquisition moments according to the acquisition moment sequence corresponding to the point cloud data of each group according to the point cloud data of each group under the condition that the 3D information display instruction aiming at the target object is received, so that 3D information display is realized.
In one embodiment of the invention, the fixed coordinate system, the fixed camera pose and the fixed camera reference are obtained by the following modules:
a model reconstruction module, configured to select a reference image group from the image groups, and reconstruct a reference 3D model of the target object using the parameter image group;
and the parameter determining module is used for determining a coordinate system where the reconstructed reference 3D model is located and a simulation pose and a simulation internal reference used when the reference 3D model is reconstructed, and the simulation pose and the simulation internal reference are used as the fixed coordinate system, the fixed camera pose and the fixed camera internal reference.
In one embodiment of the present invention, the data obtaining module includes:
reconstructing a 3D model of a target object corresponding to each image group through the following submodules for each image group to obtain point cloud data of the 3D model:
the pixel point obtaining submodule is used for extracting the characteristics of each image in the image group to obtain characteristic pixel points in each image;
the pixel point matching sub-module is used for determining matching pixel points corresponding to the same position on the target object in different images based on first features of feature pixel points in each image, wherein the first features comprise: pixel point positions of the pixel points and the first content sub-feature;
The feature prediction sub-module is used for predicting second features of corresponding points of each matched pixel point in the 3D model in the fixed coordinate system according to the first features of the matched pixel points, the fixed camera pose and the fixed camera internal parameters, wherein the second features comprise: the three-dimensional position of the midpoint of the 3D model and the second content sub-feature;
the depth obtaining submodule is used for obtaining depth information of each pixel point according to the pose of the stationary camera and the pixel coordinates of each pixel point in the participation image in the stationary camera;
and the model reconstruction submodule is used for reconstructing the 3D model of the target object corresponding to the image group according to the second characteristic of the corresponding point of each matched pixel point in the 3D model, the depth information of each pixel point and the first characteristic to obtain the point cloud data of the 3D model.
In one embodiment of the present invention, the feature prediction submodule is specifically configured to:
predicting the three-dimensional position of the corresponding point of each matched pixel point in the 3D model in the fixed coordinate system according to the pixel point position in the first feature of each matched pixel point in the image, the position of the fixed camera and the internal reference of the fixed camera;
and for each matched pixel point, determining a second characteristic of the corresponding point of the matched pixel point in the 3D model according to the first content sub-characteristic in the first characteristic of the matched pixel point and the three-dimensional position of the corresponding point of the matched pixel point in the 3D model.
In one embodiment of the present invention, the depth obtaining submodule is specifically configured to:
performing image de-distortion treatment on each image to obtain a de-distorted image;
and obtaining depth information of each pixel point according to the pose of the stationary camera and the pixel coordinates of each pixel point in the image after the stationary camera participates in de-distortion.
In one embodiment of the present invention, the model reconstruction sub-module is specifically configured to:
generating a normal map corresponding to each image according to the depth information of each pixel point;
reconstructing a model shape of the 3D model based on the normal map and pixel locations in the first feature of the matching pixel points;
filling the content of the corresponding points of each pixel point in the model shape based on the first content sub-feature in the first feature of each pixel point;
and reconstructing a 3D model of the target object corresponding to the image group according to the model shape filled with the content, and obtaining point cloud data of the 3D model.
In one embodiment of the invention, the apparatus further comprises:
the system comprises a test image acquisition module, a test image acquisition module and a test image processing module, wherein the test image acquisition module is used for acquiring each test image group of a test object, each test image group comprises images which are acquired by a plurality of image acquisition devices at the same test acquisition time and contain the test object, and the test acquisition time corresponding to each test image group is different;
The test data acquisition module is used for reconstructing test 3D models of the test object at different test acquisition moments according to each test image group to obtain test point cloud data of each test 3D model;
and the test information display module is used for sequentially rendering the test 3D models corresponding to the test acquisition moments according to the sequence of the test acquisition moments corresponding to the test point cloud data according to the test point cloud data, realizing 3D test information display, triggering and executing the image acquisition module if the displayed 3D test information is similar to the motion state of the test object, otherwise, returning to triggering and executing the test image acquisition module under the condition that the position and/or the orientation of the image acquisition equipment are changed.
In one embodiment of the present invention, the information display module is specifically configured to:
under the condition that a 3D information display instruction aiming at the target object is received, rendering a 3D model corresponding to target point cloud data according to the target point cloud data, wherein the initial value of the target point cloud data is as follows: according to the sequence of the acquisition time of each group of point cloud data, the point cloud data positioned at the forefront end;
after displaying the rendered 3D model for a preset time, controlling the rendered 3D model to disappear;
If the current target point cloud data is not the point cloud data positioned at the rearmost end in the sequence of the acquisition time, determining the point cloud data positioned at the next position of the current target point cloud data in the sequence of the acquisition time as new target point cloud data, and returning to execute the step of rendering the 3D model corresponding to the target point cloud data according to the target point cloud data to realize 3D information display.
In a third aspect of the embodiment of the present invention, there is also provided an electronic device, including a processor, a communication interface, a memory, and a communication bus, where the processor, the communication interface, and the memory complete communication with each other through the communication bus;
a memory for storing a computer program;
a processor for implementing the method steps of any of the first aspects when executing a program stored on a memory.
In a further aspect of the present invention, there is also provided a computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the method steps of any of the first aspects described above.
In a further aspect of the invention there is also provided a computer program product comprising instructions which, when run on a computer, cause the computer to perform the method steps of any of the first aspects described above.
In the 3D information display scheme provided by the embodiment of the invention, the 3D models of the target object at different acquisition moments can be reconstructed according to the same fixed coordinate system, the fixed camera pose and the fixed camera internal parameters by different image groups of the target object acquired at different acquisition moments, so that the point cloud data of each 3D model can be obtained. Under the condition that a 3D information display instruction aiming at a target object is received, according to the point cloud data, sequentially rendering 3D models corresponding to all acquisition moments according to the sequence of the acquisition moments corresponding to all groups of point cloud data, so that 3D information display is realized.
From the above, if the target object is a dynamic object, the image groups of the target object acquired at different acquisition moments are different, and different 3D models of the target object at different acquisition moments can be reconstructed according to the different image groups, so as to obtain different point cloud data corresponding to the different acquisition moments. Different 3D models can be rendered according to different point cloud data, and the dynamic effect can be watched by a user by sequentially rendering different 3D models due to the persistence of vision of human eyes. Thus, the motion state of the dynamic target object can be reproduced through the reconstructed 3D model.
In addition, in the process of reconstructing the 3D models of the target object at different acquisition moments, the coordinate values of points corresponding to the same positions on the target object on each reconstructed 3D model are similar according to the same fixed coordinate system, the fixed camera pose and the fixed camera internal parameters. Therefore, in the process of displaying the 3D information, the position mutation of the 3D model can not occur, and the smoothness of the displayed 3D information is improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below.
Fig. 1 is a schematic flow chart of a first 3D information display method according to an embodiment of the present invention;
fig. 2 is a schematic flow chart of a second 3D information display method according to an embodiment of the present invention;
fig. 3 is a flow chart of a third method for displaying 3D information according to an embodiment of the present invention;
fig. 4 is a schematic structural diagram of a first 3D information display device according to an embodiment of the present invention;
fig. 5 is a schematic structural diagram of a second 3D information display device according to an embodiment of the present invention;
fig. 6 is a schematic structural diagram of a third 3D information display device according to an embodiment of the present invention;
Fig. 7 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be described below with reference to the accompanying drawings in the embodiments of the present invention.
Since the prior art only reproduces static information of static objects based on 3D models. In order to solve the problem, the embodiment of the invention provides a 3D information display method and device.
In one embodiment of the present invention, a 3D information display method is provided, where the method includes:
and obtaining each image group of the target object, wherein each image group comprises images which are acquired by a plurality of image acquisition devices at the same acquisition time from different acquisition angles and contain the target object, and the acquisition time corresponding to each image group is different.
And reconstructing 3D models of the target object at different acquisition moments in the same fixed coordinate system according to the same fixed camera pose and fixed camera internal parameters according to each image group to obtain point cloud data of each 3D model, wherein the fixed camera pose is the simulation pose of a fixed simulation camera, the fixed camera internal parameters are the simulation internal parameters of the fixed simulation camera, and the acquisition range of the simulation camera is the union set of the acquisition ranges of each image acquisition device.
Under the condition that a 3D information display instruction aiming at the target object is received, according to each group of point cloud data, sequentially rendering 3D models corresponding to each acquisition time according to the sequence of the acquisition times corresponding to each group of point cloud data, and realizing 3D information display.
From the above, if the target object is a dynamic object, the image groups of the target object acquired at different acquisition moments are different, and different 3D models of the target object at different acquisition moments can be reconstructed according to the different image groups, so as to obtain different point cloud data corresponding to the different acquisition moments. Different 3D models can be rendered according to different point cloud data, and the dynamic effect can be watched by a user by sequentially rendering different 3D models due to the persistence of vision of human eyes. Thus, the motion state of the dynamic target object can be reproduced through the reconstructed 3D model.
In addition, in the process of reconstructing the 3D models of the target object at different acquisition moments, the coordinate values of points corresponding to the same positions on the target object on each reconstructed 3D model are similar according to the same fixed coordinate system, the fixed camera pose and the fixed camera internal parameters. Therefore, in the process of displaying the 3D information, the position mutation of the 3D model can not occur, and the smoothness of the displayed 3D information is improved.
The method and the device for displaying 3D information provided by the embodiments of the present invention are described below through specific embodiments.
Referring to fig. 1, an embodiment of the present invention provides a flowchart of a first 3D information display method, which may be implemented through the following steps S101 to S103.
S101: respective image groups of the target object are obtained.
Each image group comprises a plurality of images which are acquired by the image acquisition equipment at the same acquisition time from different acquisition angles and contain the target object, and the corresponding acquisition time of each image group is different.
Specifically, the image acquisition device may be an industrial camera or other image acquisition devices with better image acquisition effects, and the better the image acquisition effect of the image acquisition device is, the higher the accuracy of the 3D model reconstructed according to the acquired image group is.
Each image acquisition device is used for acquiring images of the target object at different angles from different acquisition angles, and each image acquisition device is connected with a power supply. The respective image acquisition devices may be mounted around the target object, facing the target object. For example, the angles between virtual lines between different image capturing devices and the target object may be 20 °, 30 °, etc.
An intersection may exist between the acquisition range of each image acquisition device and the acquisition range of an adjacent image acquisition device, that is, the same region of the target object may exist in the images of the target object acquired by different image acquisition devices.
In one embodiment of the present invention, the respective image capturing devices may be controlled to synchronously capture images of the target object through an SDK (Software Development Kit ). The mounting position and orientation of the image acquisition device may remain unchanged during acquisition of different image sets.
S102: and reconstructing 3D models of the target object at different acquisition moments in the same fixed coordinate system according to the same fixed camera pose and fixed camera internal parameters according to each image group to obtain point cloud data of each 3D model.
The stationary camera pose is a simulated pose of a fixed simulated camera.
The fixed camera internal reference is an analog internal reference of a fixed analog camera.
The acquisition range of the analog camera is the union of the acquisition ranges of the image acquisition devices.
Specifically, the simulation camera is a virtual camera representing each real image acquisition device, and is kept unchanged in the process of constructing the 3D model of the target object at different acquisition moments, namely, the adopted simulation pose and the adopted simulation internal parameters are kept unchanged in the process of constructing the 3D model of the target object at different acquisition moments, namely, the same stationary phase pose and the same stationary phase internal parameters are kept. And reconstructing the 3D models at different acquisition moments in a fixed coordinate system, so that the consistency of the reconstructed 3D models of the target object at different acquisition moments can be maintained.
The stationary camera pose and the stationary camera internal participation fixed coordinate system can be preset simulation poses and preset simulation internal participation preset coordinate systems. The preset coordinate system may be a world coordinate system or any other three-dimensional coordinate system.
In one embodiment of the present invention, the fixed coordinate system, the fixed camera pose and the fixed camera reference can be obtained through the following steps a-B.
Step A: and selecting one reference image group from the image groups, and reconstructing a reference 3D model of the target object by using the parameter image group.
Specifically, the reference image group may be any one of the image groups, or may be an image group located at the front end, or an image group located at the rear end, or an image group located in the middle according to the acquisition time.
The above-described reference 3D model may be reconstructed by a collmap algorithm or other algorithm.
And (B) step (B): and determining a coordinate system in which the reconstructed reference 3D model is located and a simulation pose and a simulation internal reference used when the reference 3D model is reconstructed, wherein the simulation pose and the simulation internal reference are used as the fixed coordinate system, the fixed camera pose and the fixed camera internal reference.
Specifically, the model_controller command line in the collmap algorithm can be used for outputting the simulation pose, the simulation internal reference and the coordinate system where the reference 3D model is located.
In another embodiment of the present invention, the 3D model may be reconstructed by a collap algorithm or other algorithms, the point cloud data may be point cloud data in ply format, and different point cloud data may be stored in different folders in a point cloud database.
In addition, after reconstructing the 3D models of the target object at different acquisition moments, smoothing processing may be performed on each 3D model, so as to obtain a smoothed 3D model.
In one embodiment of the present invention, the step S102 may be implemented by steps S102A-S102E, which are not described in detail herein.
S103: under the condition that a 3D information display instruction aiming at the target object is received, according to each group of point cloud data, sequentially rendering 3D models corresponding to each acquisition time according to the sequence of the acquisition times corresponding to each group of point cloud data, and realizing 3D information display.
The collection time corresponding to the point cloud data may be the collection time of the image group in which the point cloud data is reconstructed.
Specifically, 3D models corresponding to all the acquisition moments can be rendered through the open source library opengl according to the sequence from front to back of the acquisition moments corresponding to all the sets of point cloud data, so that 3D information display of the target object is realized. And if the 3D models corresponding to all the acquisition moments are rendered according to the sequence from back to front of the acquisition moments corresponding to all the sets of point cloud data, the inverted display of the 3D information of the target object can be realized.
In one embodiment of the invention, the 3D model corresponding to each acquisition time can be sequentially rendered through the following steps C-E.
Step C: and rendering a 3D model corresponding to the target point cloud data according to the target point cloud data.
The initial value of the target point cloud data is as follows: and according to the sequence of the acquisition time of each group of point cloud data, the point cloud data positioned at the forefront end.
Step D: and after displaying the rendered 3D model for a preset time, controlling the rendered 3D model to disappear.
Specifically, according to the persistence of vision effect of the human eye, when the preset duration of the rendered 3D model is displayed to be shorter, the user can observe the dynamic effect by sequentially rendering each 3D model, so that the preset duration can be 0.03s, 0.04s, and the like.
Step E: and C, if the current target point cloud data is not the point cloud data positioned at the rearmost end in the sequence of the acquisition time, determining the point cloud data positioned next to the current target point cloud data in the sequence of the acquisition time as new target point cloud data, and returning to execute the step A.
If the current target point cloud data is the point cloud data positioned at the rearmost end in the sequence of the acquisition time, the 3D information display is finished, otherwise, the point cloud data positioned at the next position of the current target point cloud data in the sequence of the acquisition time is required to be rendered.
From the above, if the target object is a dynamic object, the image groups of the target object acquired at different acquisition moments are different, and different 3D models of the target object at different acquisition moments can be reconstructed according to the different image groups, so as to obtain different point cloud data corresponding to the different acquisition moments. Different 3D models can be rendered according to different point cloud data, and the dynamic effect can be watched by a user by sequentially rendering different 3D models due to the persistence of vision of human eyes. Thus, the motion state of the dynamic target object can be reproduced through the reconstructed 3D model.
In addition, in the process of reconstructing the 3D models of the target object at different acquisition moments, the coordinate values of points corresponding to the same positions on the target object on each reconstructed 3D model are similar according to the same fixed coordinate system, the fixed camera pose and the fixed camera internal parameters. Therefore, in the process of displaying the 3D information, the position mutation of the 3D model can not occur, and the smoothness of the displayed 3D information is improved.
Referring to fig. 2, a flow chart of a second 3D information display method according to an embodiment of the present invention is shown, and compared with the embodiment shown in fig. 1, the above step S102 may be implemented for each image group through the following steps S102A-S102E.
S102A: and extracting the characteristics of each image in the image group to obtain characteristic pixel points in each image.
Specifically, feature pixel points in each image can be obtained through an algorithm in the prior art. For example, SUSAN (Small univalue segment assimilating nucleus, corner feature detection) operator, doG (Difference of Gaussian, gaussian difference) operator, RANSAC (random sample consensus ) algorithm, and the like, to which embodiments of the present invention are not limited.
S102B: and determining matched pixel points corresponding to the same position on the target object in different images based on the first features of the feature pixel points in each image.
Wherein the first feature includes: the pixel location of the pixel and the first content sub-feature.
Specifically, the pixel point position may be represented by a pixel coordinate of the pixel point in the image, and the first content sub-feature includes a color value, a brightness value, a material of a point on the target object corresponding to the pixel point, and the like.
In one embodiment of the invention, the similarity between the first features of the feature pixel points in different images can be calculated, and the feature pixel points with the calculated similarity greater than the preset similarity are determined as the matched pixel points.
S102C: and predicting second characteristics of corresponding points of each matched pixel point in the 3D model in the fixed coordinate system according to the first characteristics of the matched pixel points, the fixed camera pose and the fixed camera internal parameters.
Wherein the second feature includes: the three-dimensional position of the midpoint of the 3D model is associated with the second content sub-feature.
In one embodiment of the present invention, the second feature of the corresponding point in the 3D model in the fixed coordinate system may be predicted by the following steps F to G.
Step F: and predicting the three-dimensional position of the corresponding point of each matched pixel point in the 3D model in the fixed coordinate system according to the pixel point position in the first feature of each matched pixel point in the image, the fixed camera pose and the fixed camera internal reference.
Specifically, the three-dimensional coordinates of the three-dimensional position can be calculated by the following coordinate conversion formula:
wherein Z is c For homogeneous scale, u is the abscissa in the pixel coordinates of the matching pixel point, v is the ordinate in the pixel coordinates of the matching pixel point, d x Representing the size of a unit pixel of an image on the horizontal axis, d y Representing the size of a unit pixel of an image on the vertical axis, gamma being a distortion parameter, u 0 Is the abscissa, v, of the optical center pixel point in the image 0 Is the ordinate of the optical center pixel point in the image, f is the focal length, R 1 To represent an orthogonal rotation matrix, T 1 To represent a translation matrix, X i1 To match the x-axis coordinates of the corresponding point of the pixel point in the 3D model in a fixed coordinate system, Y i1 To match the y-axis coordinates of the corresponding points of the pixel points in the 3D model in a fixed coordinate system, Z i1 To match the z-axis coordinates of the corresponding points of the pixel points in the 3D model in a fixed coordinate system. Wherein the positive direction of the x-axis is the horizontal direction,the positive direction of the y-axis is the vertical direction, and the positive direction of the z-axis is the depth direction. The origin of the coordinate system may be any point, the x-axis direction and the y-axis direction may be any directions perpendicular to each other, and the z-axis direction may be determined according to the right-hand rule.
Specifically, d in the above formula x 、d y 、γ、u 0 、v 0 And f is a parameter contained in a fixed camera reference, R 1 And T is 1 Is a parameter contained in the stationary phase pose.
Step G: and for each matched pixel point, determining a second characteristic of the corresponding point of the matched pixel point in the 3D model according to the first content sub-characteristic in the first characteristic of the matched pixel point and the three-dimensional position of the corresponding point of the matched pixel point in the 3D model.
Specifically, the calculated three-dimensional position may be used as the three-dimensional position in the second feature, and the first content sub-feature is used as the second content sub-feature in the second feature, so as to determine the second feature of the corresponding point of the matched pixel point in the 3D model.
S102D: and obtaining depth information of each pixel point according to the pose of the stationary camera and the pixel coordinates of each pixel point in the participation image in the stationary camera.
In one embodiment of the present invention, the depth information of each pixel point may be obtained through algorithms in the prior art, such as a multi-view stereo algorithm, a photometric stereo algorithm, a defocus inference algorithm, and the like, which is not limited in this embodiment of the present invention.
In another embodiment of the present invention, since the image acquired by the image acquisition device may have distortion, if depth information of each pixel point in the image acquired by the image acquisition device is directly determined, the obtained depth information may be inaccurate. Therefore, the image de-distortion processing can be carried out on each image to obtain a de-distorted image, and the depth information of each pixel point is determined according to the pose of the stationary camera and the pixel coordinates of each pixel point in the image after the stationary camera participates in de-distortion.
Specifically, the image de-distortion processing can be completed by using an image de-distortion function in OpenCV, such as an undristor image function, an initunderstator electifypap function, a remap function, and other algorithms.
S102E: and reconstructing a 3D model of the target object corresponding to the image group according to the second characteristics of the corresponding points of each matched pixel point in the 3D model, and the depth information and the first characteristics of each pixel point.
Specifically, since the pixel position of the pixel in the image is two-dimensional position information and the depth information is three-dimensional position information, the three-dimensional position of the corresponding point of each pixel in the 3D model can be determined according to the pixel position and the depth information in the first feature of the pixel. And taking the first content sub-feature in the first feature of the pixel point as the second content sub-feature of the corresponding point of the pixel point in the 3D model. Therefore, the 3D model of the target object corresponding to the image group can be reconstructed through the determined three-dimensional positions of the corresponding points of the pixel points in the 3D model and the information such as color, brightness, materials and the like contained in the second content sub-features.
In one embodiment of the present invention, step S102E described above may be implemented by a dense reconstruction process in the SFM algorithm.
In another embodiment of the present invention, the above step S102F may also be implemented by the following step H-step K.
Step H: and generating a normal map corresponding to each image according to the depth information of each pixel point.
Specifically, the normal map is used for representing the concave-convex condition of the surface of the target object, and the normal map includes the normal marked with the direction by the RGB color channels.
The depth information of each pixel point may represent a distance between a point on the target object corresponding to the pixel point and the image acquisition device, and the smaller the distance between the point on the target object and the image acquisition device is, the more convex the point on the target object is, or the more concave the point on the target object is, so that a normal map corresponding to each image may be generated according to the depth information of each pixel point.
Step I: the model shape of the 3D model is reconstructed based on the normal map and pixel locations in the first feature of the matching pixel points.
Specifically, the three-dimensional position of the matching pixel point at the corresponding point of the 3D model may be predicted according to the pixel position in the first feature of the matching pixel point, and the corresponding point of the matching pixel point in the normal map may be determined, so as to determine the three-dimensional position of the corresponding point in the normal map. And reconstructing the model shape of the 3D model by taking the three-dimensional position of the corresponding point of the matched pixel point in the found map as a reference and combining the concave-convex condition of the surface of the target object represented by the normal map.
The model shape may represent a surface of the target object.
Step J: and filling the content of the corresponding point of each pixel point in the model shape based on the first content sub-feature in the first feature of each pixel point.
Specifically, the content value of the corresponding point of each pixel point in the model shape may be assigned to the first content sub-feature, where the first content sub-feature includes the color value and the brightness value of each pixel point, and the material of the corresponding point in the model shape is set to be the material represented by the first content sub-feature.
Step K: and reconstructing a 3D model of the target object corresponding to the image group according to the model shape after the content filling.
Specifically, the model shape after the content filling may be used as a model surface of the reconstructed 3D model, so as to reconstruct the 3D model of the target object corresponding to the image group.
From the above, it can be determined, according to the first features of the feature pixel points of each image in the image group, the matching pixel points corresponding to the same position on the target object in different images. The images of the target object at different angles can thus be combined with each other by matching pixel points. And converting the pixel point positions in the first characteristic of the matched pixel points into three-dimensional positions, and taking the three-dimensional positions as positions of corresponding points of the matched pixel points in the 3D model. And determining the positions of corresponding points of other pixel points in the 3D model by taking the three-dimensional positions of the matched pixel points as a reference. And taking the first content sub-feature contained in the first characteristic of the pixel point as the second content sub-feature of the corresponding point of the pixel point in the 3D model, thereby reconstructing the 3D model of the target object.
Referring to fig. 3, a flow chart of a third 3D information display method according to an embodiment of the present invention further includes the following steps S104-S106 before step S101, compared with the embodiment shown in fig. 1.
S104: respective sets of test images of the test object are obtained.
Each test image group comprises images which are acquired by a plurality of image acquisition devices at the same test acquisition time and contain the test object, and the test acquisition time corresponding to each test image group is different.
S105: and reconstructing the test 3D models of the test object at different test acquisition moments according to each test image group to obtain test point cloud data of each test 3D model.
S106: according to the cloud data of each group of test points, according to the sequence of the test collection time corresponding to the cloud data of the test points, the test 3D model corresponding to each test collection time is sequentially rendered, and 3D test information display is achieved.
Specifically, the implementation manners of the steps S104 to S106 are similar to the implementation manners of the steps S101 to S103, and will not be repeated here.
If it is determined that the displayed 3D test information is similar to the motion state of the test object, the above step S101 is performed. Otherwise, in the case where it is determined that the position and/or orientation of the image capturing apparatus is changed, the above-described step S104 is executed back.
In the embodiment of the invention, the user can watch the displayed 3D test information and compare the 3D test information with the motion state of the test object, and the user determines whether the displayed 3D test information is similar to the motion state of the test object. The user can input an instruction indicating whether the motion states of the displayed 3D test information and the test object are similar, and the electronic device executing the 3D information display method can determine whether the motion states of the displayed 3D test information and the test object are similar according to the received instruction.
If the displayed 3D test information is dissimilar to the motion state of the test object, it is indicated that the current displayed 3D test information is different from the actual motion state of the test object, and the motion state of the test object cannot be accurately reproduced according to the image group acquired by the image acquisition device at the current position and/or orientation, so that the position and/or orientation of the image acquisition device needs to be changed.
After the position and/or orientation of the image acquisition device are changed, the acquired test image group is changed, the reconstructed test 3D model is changed, 3D test information can be displayed according to the new test 3D model until the displayed 3D test information is similar to the motion state of the test object, and the motion state of the object can be accurately reproduced by the position and/or orientation of the image acquisition device after adjustment. Step S101 may thus be performed with the position and orientation of the image capturing apparatus maintained to reproduce the motion state of the target object.
In the above, before the 3D information of the target object is displayed, the 3D test information of the test object is displayed, whether the position and/or the orientation of the image acquisition device meets the requirements is detected, if not, the position and/or the orientation of the image acquisition device is changed, and if yes, the image group of the target object is acquired under the condition that the position and the orientation of the image acquisition device are maintained, so that the 3D information of the target object is displayed. To ensure accuracy of the presented 3D information.
Corresponding to the foregoing 3D information display method, referring to fig. 4, an embodiment of the present invention further provides a schematic structural diagram of a first 3D information display device, where the device includes:
an image obtaining module 401, configured to obtain each image group of a target object, where each image group includes images including the target object, which are collected by a plurality of image collecting devices at the same collection time from different collection angles, and the collection times corresponding to each image group are different;
the data obtaining module 402 is configured to reconstruct 3D models of the target object at different acquisition moments in the same fixed coordinate system according to the same stationary camera pose and fixed camera internal parameters according to each image group, so as to obtain point cloud data of each 3D model, where the stationary camera pose is a simulation pose of a fixed simulation camera, the fixed camera internal parameters are simulation internal parameters of the fixed simulation camera, and an acquisition range of the simulation camera is a union of acquisition ranges of each image acquisition device;
The information display module 403 is configured to sequentially render, according to each set of point cloud data and according to the order of the collection moments corresponding to each set of point cloud data, a 3D model corresponding to each collection moment under the condition that a 3D information display instruction for the target object is received, so as to realize 3D information display.
From the above, if the target object is a dynamic object, the image groups of the target object acquired at different acquisition moments are different, and different 3D models of the target object at different acquisition moments can be reconstructed according to the different image groups, so as to obtain different point cloud data corresponding to the different acquisition moments. Different 3D models can be rendered according to different point cloud data, and the dynamic effect can be watched by a user by sequentially rendering different 3D models due to the persistence of vision of human eyes. Thus, the motion state of the dynamic target object can be reproduced through the reconstructed 3D model.
In addition, in the process of reconstructing the 3D models of the target object at different acquisition moments, the coordinate values of points corresponding to the same positions on the target object on each reconstructed 3D model are similar according to the same fixed coordinate system, the fixed camera pose and the fixed camera internal parameters. Therefore, in the process of displaying the 3D information, the position mutation of the 3D model can not occur, and the smoothness of the displayed 3D information is improved.
In one embodiment of the invention, the fixed coordinate system, the fixed camera pose and the fixed camera reference are obtained by the following modules:
a model reconstruction module, configured to select a reference image group from the image groups, and reconstruct a reference 3D model of the target object using the parameter image group;
and the parameter determining module is used for determining a coordinate system where the reconstructed reference 3D model is located and a simulation pose and a simulation internal reference used when the reference 3D model is reconstructed, and the simulation pose and the simulation internal reference are used as the fixed coordinate system, the fixed camera pose and the fixed camera internal reference.
Referring to fig. 5, a schematic structural diagram of a second 3D information display device according to an embodiment of the present invention, compared with the embodiment shown in fig. 4, the data obtaining module 402 includes:
reconstructing a 3D model of a target object corresponding to each image group through the following submodules for each image group to obtain point cloud data of the 3D model:
a pixel point obtaining submodule 402A, configured to perform feature extraction on each image in the image group to obtain feature pixel points in each image;
a pixel matching sub-module 402B, configured to determine, based on first features of feature pixels in each image, matching pixels in different images corresponding to a same position on the target object, where the first features include: pixel point positions of the pixel points and the first content sub-feature;
The feature prediction submodule 402C is configured to predict, according to the first feature of the matched pixel points, the stationary camera pose and the stationary camera internal reference, a second feature of a corresponding point of each matched pixel point in the 3D model in the stationary coordinate system, where the second feature includes: the three-dimensional position of the midpoint of the 3D model and the second content sub-feature;
a depth obtaining submodule 402D, configured to obtain depth information of each pixel point according to the pose of the stationary camera and pixel coordinates of each pixel point in the participation image in the stationary camera;
the model reconstruction submodule 402E is configured to reconstruct a 3D model of the target object corresponding to the image group according to the second feature of the corresponding point in the 3D model of each matched pixel point, the depth information of each pixel point, and the first feature, so as to obtain point cloud data of the 3D model.
From the above, it can be determined, according to the first features of the feature pixel points of each image in the image group, the matching pixel points corresponding to the same position on the target object in different images. The images of the target object at different angles can thus be combined with each other by matching pixel points. And converting the pixel point positions in the first characteristic of the matched pixel points into three-dimensional positions, and taking the three-dimensional positions as positions of corresponding points of the matched pixel points in the 3D model. And determining the positions of corresponding points of other pixel points in the 3D model by taking the three-dimensional positions of the matched pixel points as a reference. And taking the first content sub-feature contained in the first characteristic of the pixel point as the second content sub-feature of the corresponding point of the pixel point in the 3D model, thereby reconstructing the 3D model of the target object.
In one embodiment of the present invention, the feature prediction submodule 402C is specifically configured to:
predicting the three-dimensional position of the corresponding point of each matched pixel point in the 3D model in the fixed coordinate system according to the pixel point position in the first feature of each matched pixel point in the image, the position of the fixed camera and the internal reference of the fixed camera;
and for each matched pixel point, determining a second characteristic of the corresponding point of the matched pixel point in the 3D model according to the first content sub-characteristic in the first characteristic of the matched pixel point and the three-dimensional position of the corresponding point of the matched pixel point in the 3D model.
In one embodiment of the present invention, the depth obtaining submodule 402D is specifically configured to:
performing image de-distortion treatment on each image to obtain a de-distorted image;
and obtaining depth information of each pixel point according to the pose of the stationary camera and the pixel coordinates of each pixel point in the image after the stationary camera participates in de-distortion.
In one embodiment of the present invention, the model reconstruction submodule 402E is specifically configured to:
generating a normal map corresponding to each image according to the depth information of each pixel point;
reconstructing a model shape of the 3D model based on the normal map and pixel locations in the first feature of the matching pixel points;
Filling the content of the corresponding points of each pixel point in the model shape based on the first content sub-feature in the first feature of each pixel point;
and reconstructing a 3D model of the target object corresponding to the image group according to the model shape filled with the content, and obtaining point cloud data of the 3D model.
Referring to fig. 6, a schematic structural diagram of a third 3D information display device according to an embodiment of the present invention, compared with the embodiment shown in fig. 4, the device further includes:
a test image obtaining module 404, configured to obtain each test image group of a test object, where each test image group includes images including the test object collected by a plurality of image collecting devices at the same test collection time, and the test collection times corresponding to each test image group are different;
the test data obtaining module 405 is configured to reconstruct a test 3D model of the test object at different test acquisition moments according to each test image group, so as to obtain test point cloud data of each test 3D model;
the test information display module 406 is configured to sequentially render, according to the test collection time sequence corresponding to the test point cloud data, a test 3D model corresponding to each test collection time according to each set of test point cloud data, so as to implement 3D test information display, and if it is determined that the displayed 3D test information is similar to the motion state of the test object, trigger to execute the image obtaining module 401, otherwise, return to trigger to execute the test image obtaining module 404 under the condition that it is determined that the position and/or orientation of the image collection device is changed.
In the above, before the 3D information of the target object is displayed, the 3D test information of the test object is displayed, whether the position and/or the orientation of the image acquisition device meets the requirements is detected, if not, the position and/or the orientation of the image acquisition device is changed, and if yes, the image group of the target object is acquired under the condition that the position and the orientation of the image acquisition device are maintained, so that the 3D information of the target object is displayed. To ensure accuracy of the presented 3D information.
In one embodiment of the present invention, the information display module 403 is specifically configured to:
under the condition that a 3D information display instruction aiming at the target object is received, rendering a 3D model corresponding to target point cloud data according to the target point cloud data, wherein the initial value of the target point cloud data is as follows: according to the sequence of the acquisition time of each group of point cloud data, the point cloud data positioned at the forefront end;
after displaying the rendered 3D model for a preset time, controlling the rendered 3D model to disappear;
if the current target point cloud data is not the point cloud data positioned at the rearmost end in the sequence of the acquisition time, determining the point cloud data positioned at the next position of the current target point cloud data in the sequence of the acquisition time as new target point cloud data, and returning to execute the step of rendering the 3D model corresponding to the target point cloud data according to the target point cloud data to realize 3D information display.
The embodiment of the present invention further provides an electronic device, as shown in fig. 7, including a processor 701, a communication interface 702, a memory 703 and a communication bus 704, where the processor 701, the communication interface 702, and the memory 703 perform communication with each other through the communication bus 704,
a memory 703 for storing a computer program;
the processor 701 is configured to implement the above-described 3D information display method when executing the program stored in the memory 703.
When the electronic equipment provided by the embodiment of the invention is used for displaying 3D information, if the target object is a dynamic object, the image groups of the target object acquired at different acquisition moments are different, and different 3D models of the target object at different acquisition moments can be reconstructed according to the different image groups, so that different point cloud data corresponding to the different acquisition moments are obtained. Different 3D models can be rendered according to different point cloud data, and the dynamic effect can be watched by a user by sequentially rendering different 3D models due to the persistence of vision of human eyes. Thus, the motion state of the dynamic target object can be reproduced through the reconstructed 3D model.
In addition, in the process of reconstructing the 3D models of the target object at different acquisition moments, the coordinate values of points corresponding to the same positions on the target object on each reconstructed 3D model are similar according to the same fixed coordinate system, the fixed camera pose and the fixed camera internal parameters. Therefore, in the process of displaying the 3D information, the position mutation of the 3D model can not occur, and the smoothness of the displayed 3D information is improved.
The communication bus mentioned by the above terminal may be a peripheral component interconnect standard (Peripheral Component Interconnect, abbreviated as PCI) bus or an extended industry standard architecture (Extended Industry Standard Architecture, abbreviated as EISA) bus, etc. The communication bus may be classified as an address bus, a data bus, a control bus, or the like. For ease of illustration, the figures are shown with only one bold line, but not with only one bus or one type of bus.
The communication interface is used for communication between the terminal and other devices.
The memory may include random access memory (Random Access Memory, RAM) or non-volatile memory (non-volatile memory), such as at least one disk memory. Optionally, the memory may also be at least one memory device located remotely from the aforementioned processor.
The processor may be a general-purpose processor, including a central processing unit (Central Processing Unit, CPU for short), a network processor (Network Processor, NP for short), etc.; but also digital signal processors (Digital Signal Processor, DSP for short), application specific integrated circuits (Application Specific Integrated Circuit, ASIC for short), field-programmable gate arrays (Field-Programmable Gate Array, FPGA for short) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
In yet another embodiment of the present invention, a computer readable storage medium is provided, where a computer program is stored, the computer program implementing the 3D information presentation method according to any one of the above embodiments when being executed by a processor.
When the computer program stored in the computer readable storage medium provided by the embodiment is used for displaying 3D information, if the target object is a dynamic object, the image groups of the target object acquired at different acquisition moments are different, and the 3D model of the target object at different acquisition moments can be reconstructed according to the different image groups, so that different point cloud data corresponding to the different acquisition moments are obtained. Different 3D models can be rendered according to different point cloud data, and the dynamic effect can be watched by a user by sequentially rendering different 3D models due to the persistence of vision of human eyes. Thus, the motion state of the dynamic target object can be reproduced through the reconstructed 3D model.
In addition, in the process of reconstructing the 3D models of the target object at different acquisition moments, the coordinate values of points corresponding to the same positions on the target object on each reconstructed 3D model are similar according to the same fixed coordinate system, the fixed camera pose and the fixed camera internal parameters. Therefore, in the process of displaying the 3D information, the position mutation of the 3D model can not occur, and the smoothness of the displayed 3D information is improved.
In a further embodiment of the present invention, a computer program product comprising instructions, which when run on a computer, causes the computer to perform the 3D information presentation method according to any of the above embodiments is also provided.
When the computer program product provided by the embodiment is executed to display 3D information, if the target object is a dynamic object, the image groups of the target object acquired at different acquisition moments are different, and different 3D models of the target object at different acquisition moments can be reconstructed according to the different image groups, so that different point cloud data corresponding to the different acquisition moments are obtained. Different 3D models can be rendered according to different point cloud data, and the dynamic effect can be watched by a user by sequentially rendering different 3D models due to the persistence of vision of human eyes. Thus, the motion state of the dynamic target object can be reproduced through the reconstructed 3D model.
In addition, in the process of reconstructing the 3D models of the target object at different acquisition moments, the coordinate values of points corresponding to the same positions on the target object on each reconstructed 3D model are similar according to the same fixed coordinate system, the fixed camera pose and the fixed camera internal parameters. Therefore, in the process of displaying the 3D information, the position mutation of the 3D model can not occur, and the smoothness of the displayed 3D information is improved.
In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, produces a flow or function in accordance with embodiments of the present invention, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions may be stored in or transmitted from one computer-readable storage medium to another, for example, by wired (e.g., coaxial cable, optical fiber, digital Subscriber Line (DSL)), or wireless (e.g., infrared, wireless, microwave, etc.). The computer readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that contains an integration of one or more available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid State Disk (SSD)), etc.
It is noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
In this specification, each embodiment is described in a related manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for an apparatus, an electronic device, a computer-readable storage medium and a computer program product, the description is relatively simple, as it is substantially similar to the method embodiments, as relevant see the partial description of the method embodiments.
The foregoing description is only of the preferred embodiments of the present invention and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims (9)

1. A method for displaying 3D information, the method comprising:
obtaining each image group of a target object, wherein each image group comprises images which are acquired by a plurality of image acquisition devices at the same acquisition time from different acquisition angles and contain the target object, and the acquisition time corresponding to each image group is different;
for each image group, reconstructing a 3D model of the target object corresponding to the image group by: extracting the characteristics of each image in the image group to obtain characteristic pixel points in each image; determining matched pixel points corresponding to the same position on the target object in different images based on first features of the feature pixel points in each image; predicting second characteristics of corresponding points of each matched pixel point in a 3D model in a fixed coordinate system according to the first characteristics of the matched pixel points, the fixed camera pose and the fixed camera internal parameters; obtaining depth information of each pixel point according to the pose of the stationary camera and the pixel coordinates of each pixel point in the participation image in the stationary camera; reconstructing a 3D model of a target object corresponding to the image group according to second features of corresponding points of each matched pixel point in the 3D model, depth information of each pixel point and first features, and obtaining point cloud data of each 3D model, wherein the first features comprise: the pixel point position of the pixel point and the first content sub-feature, and the second feature comprises: the three-dimensional position of the midpoint of the 3D model and the second content sub-feature, the fixed camera pose is the simulation pose of a fixed simulation camera, the fixed camera internal reference is the simulation internal reference of the fixed simulation camera, and the acquisition range of the simulation camera is the union of the acquisition ranges of all image acquisition devices;
Under the condition that a 3D information display instruction aiming at the target object is received, according to each group of point cloud data, sequentially rendering 3D models corresponding to each acquisition time according to the sequence of the acquisition times corresponding to each group of point cloud data, and realizing 3D information display.
2. The method of claim 1, wherein the stationary coordinate system, stationary camera pose, and stationary camera reference are obtained by:
selecting one reference image group from the image groups, and reconstructing a reference 3D model of the target object by using the reference image group;
and determining a coordinate system where the reconstructed reference 3D model is located and a simulation pose and a simulation internal reference used when the reference 3D model is reconstructed, wherein the simulation pose and the simulation internal reference are used as the fixed coordinate system, the fixed camera pose and the fixed camera internal reference.
3. The method according to claim 1, wherein predicting the second feature of the corresponding point of each matching pixel in the 3D model in the fixed coordinate system based on the first feature of the matching pixel, the stationary camera pose and the stationary camera reference comprises:
predicting the three-dimensional position of the corresponding point of each matched pixel point in the 3D model in the fixed coordinate system according to the pixel point position in the first feature of each matched pixel point in the image, the position of the fixed camera and the internal reference of the fixed camera;
And for each matched pixel point, determining a second characteristic of the corresponding point of the matched pixel point in the 3D model according to the first content sub-characteristic in the first characteristic of the matched pixel point and the three-dimensional position of the corresponding point of the matched pixel point in the 3D model.
4. The method according to claim 1, wherein the obtaining depth information of each pixel point according to the stationary camera pose and the pixel coordinates of each pixel point in the stationary-camera-in-camera-participation image includes:
performing image de-distortion treatment on each image to obtain a de-distorted image;
and obtaining depth information of each pixel point according to the pose of the stationary camera and the pixel coordinates of each pixel point in the image after the stationary camera participates in de-distortion.
5. The method according to claim 1, wherein reconstructing the 3D model of the target object corresponding to the image group according to the second feature of the corresponding point in the 3D model of each matched pixel point, the depth information of each pixel point, and the first feature comprises:
generating a normal map corresponding to each image according to the depth information of each pixel point;
reconstructing a model shape of the 3D model based on the normal map and pixel locations in the first feature of the matching pixel points;
Filling the content of the corresponding points of each pixel point in the model shape based on the first content sub-feature in the first feature of each pixel point;
and reconstructing a 3D model of the target object corresponding to the image group according to the model shape after the content filling.
6. The method according to any one of claims 1-5, further comprising, prior to obtaining the respective image sets of the target object:
obtaining each test image group of a test object, wherein each test image group comprises images containing the test object, which are acquired by a plurality of image acquisition devices at the same test acquisition time, and the test acquisition time corresponding to each test image group is different;
reconstructing test 3D models of the test object at different test acquisition moments according to each test image group to obtain test point cloud data of each test 3D model;
according to the cloud data of each group of test points, sequentially rendering a test 3D model corresponding to each test acquisition time according to the sequence of the test acquisition time corresponding to the cloud data of the test points, so as to realize 3D test information display;
if the displayed 3D test information is similar to the motion state of the test object, executing the step of obtaining each image group of the target object;
Otherwise, the step of obtaining the respective test image groups of the test object is performed in return, in case it is determined that the position and/or orientation of the image acquisition device has changed.
7. The method according to any one of claims 1-5, wherein sequentially rendering the 3D model corresponding to each acquisition time according to the order of the acquisition time corresponding to each set of point cloud data according to each set of point cloud data, comprises:
rendering a 3D model corresponding to the target point cloud data according to the target point cloud data, wherein the initial value of the target point cloud data is as follows: according to the sequence of the acquisition time of each group of point cloud data, the point cloud data positioned at the forefront end;
after displaying the rendered 3D model for a preset time, controlling the rendered 3D model to disappear;
if the current target point cloud data is not the point cloud data positioned at the rearmost end in the sequence of the acquisition time, determining the point cloud data positioned at the next position of the current target point cloud data in the sequence of the acquisition time as new target point cloud data, and returning to execute the step of rendering the 3D model corresponding to the target point cloud data according to the target point cloud data.
8. A 3D information presentation apparatus, the apparatus comprising:
The image acquisition module is used for acquiring each image group of the target object, wherein each image group comprises a plurality of images which are acquired by the image acquisition equipment at the same acquisition time from different acquisition angles and contain the target object, and the acquisition time corresponding to each image group is different;
a data obtaining module, configured to reconstruct, for each image group, a 3D model of a target object corresponding to the image group by: extracting the characteristics of each image in the image group to obtain characteristic pixel points in each image; determining matched pixel points corresponding to the same position on the target object in different images based on first features of the feature pixel points in each image; predicting second characteristics of corresponding points of each matched pixel point in a 3D model in a fixed coordinate system according to the first characteristics of the matched pixel points, the fixed camera pose and the fixed camera internal parameters; obtaining depth information of each pixel point according to the pose of the stationary camera and the pixel coordinates of each pixel point in the participation image in the stationary camera; reconstructing a 3D model of a target object corresponding to the image group according to second features of corresponding points of each matched pixel point in the 3D model, depth information of each pixel point and first features, wherein the first features comprise: the pixel point position of the pixel point and the first content sub-feature, and the second feature comprises: the three-dimensional position of the midpoint of the 3D model and the second content sub-feature, the fixed camera pose is the simulation pose of a fixed simulation camera, the fixed camera internal reference is the simulation internal reference of the fixed simulation camera, and the acquisition range of the simulation camera is the union of the acquisition ranges of all image acquisition devices;
The information display module is used for sequentially rendering the 3D models corresponding to the acquisition moments according to the acquisition moment sequence corresponding to the point cloud data of each group according to the point cloud data of each group under the condition that the 3D information display instruction aiming at the target object is received, so that 3D information display is realized.
9. The apparatus of claim 8, wherein the stationary coordinate system, stationary camera pose, and stationary camera reference are obtained by:
a model reconstruction module, configured to select a reference image group from the image groups, and reconstruct a reference 3D model of the target object using the reference image group;
and the parameter determining module is used for determining a coordinate system where the reconstructed reference 3D model is located and a simulation pose and a simulation internal reference used when the reference 3D model is reconstructed, and the simulation pose and the simulation internal reference are used as the fixed coordinate system, the fixed camera pose and the fixed camera internal reference.
CN202011566818.1A 2020-12-25 2020-12-25 3D information display method and device Active CN112634439B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202011566818.1A CN112634439B (en) 2020-12-25 2020-12-25 3D information display method and device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202011566818.1A CN112634439B (en) 2020-12-25 2020-12-25 3D information display method and device

Publications (2)

Publication Number Publication Date
CN112634439A CN112634439A (en) 2021-04-09
CN112634439B true CN112634439B (en) 2023-10-31

Family

ID=75325380

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202011566818.1A Active CN112634439B (en) 2020-12-25 2020-12-25 3D information display method and device

Country Status (1)

Country Link
CN (1) CN112634439B (en)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113144613B (en) * 2021-05-08 2024-06-21 成都乘天游互娱网络科技有限公司 Model-based method for generating volume cloud

Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106780576A (en) * 2016-11-23 2017-05-31 北京航空航天大学 A kind of camera position and orientation estimation method towards RGBD data flows
CN106941605A (en) * 2017-04-27 2017-07-11 华南理工大学 The image vision monitoring apparatus and method of a kind of emery wheel dressing finishing
JP2018181047A (en) * 2017-04-17 2018-11-15 凸版印刷株式会社 Three-dimensional shape model generating device, three-dimensional shape model generating method and program
CN109509226A (en) * 2018-11-27 2019-03-22 广东工业大学 Three dimensional point cloud method for registering, device, equipment and readable storage medium storing program for executing
CN110874818A (en) * 2018-08-31 2020-03-10 阿里巴巴集团控股有限公司 Image processing and virtual space construction method, device, system and storage medium
WO2020078250A1 (en) * 2018-10-15 2020-04-23 华为技术有限公司 Data processing method and device for virtual scene
CN111311742A (en) * 2020-03-27 2020-06-19 北京百度网讯科技有限公司 Three-dimensional reconstruction method, three-dimensional reconstruction device and electronic equipment
CN111369666A (en) * 2020-03-02 2020-07-03 中国电子科技集团公司第五十二研究所 Dynamic target reconstruction method and device based on multiple RGBD cameras
KR20200122870A (en) * 2019-04-19 2020-10-28 광운대학교 산학협력단 Acquisition method for high quality 3-dimension spatial information using photogrammetry
WO2020243962A1 (en) * 2019-06-06 2020-12-10 深圳市大疆创新科技有限公司 Object detection method, electronic device and mobile platform

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR102178239B1 (en) * 2017-12-14 2020-11-12 캐논 가부시끼가이샤 3D model generation device, generation method, and program

Patent Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106780576A (en) * 2016-11-23 2017-05-31 北京航空航天大学 A kind of camera position and orientation estimation method towards RGBD data flows
JP2018181047A (en) * 2017-04-17 2018-11-15 凸版印刷株式会社 Three-dimensional shape model generating device, three-dimensional shape model generating method and program
CN106941605A (en) * 2017-04-27 2017-07-11 华南理工大学 The image vision monitoring apparatus and method of a kind of emery wheel dressing finishing
CN110874818A (en) * 2018-08-31 2020-03-10 阿里巴巴集团控股有限公司 Image processing and virtual space construction method, device, system and storage medium
WO2020078250A1 (en) * 2018-10-15 2020-04-23 华为技术有限公司 Data processing method and device for virtual scene
CN109509226A (en) * 2018-11-27 2019-03-22 广东工业大学 Three dimensional point cloud method for registering, device, equipment and readable storage medium storing program for executing
KR20200122870A (en) * 2019-04-19 2020-10-28 광운대학교 산학협력단 Acquisition method for high quality 3-dimension spatial information using photogrammetry
WO2020243962A1 (en) * 2019-06-06 2020-12-10 深圳市大疆创新科技有限公司 Object detection method, electronic device and mobile platform
CN111369666A (en) * 2020-03-02 2020-07-03 中国电子科技集团公司第五十二研究所 Dynamic target reconstruction method and device based on multiple RGBD cameras
CN111311742A (en) * 2020-03-27 2020-06-19 北京百度网讯科技有限公司 Three-dimensional reconstruction method, three-dimensional reconstruction device and electronic equipment

Also Published As

Publication number Publication date
CN112634439A (en) 2021-04-09

Similar Documents

Publication Publication Date Title
CN110675489B (en) Image processing method, device, electronic equipment and storage medium
WO2018119889A1 (en) Three-dimensional scene positioning method and device
CN113689578B (en) Human body data set generation method and device
CN104715479A (en) Scene reproduction detection method based on augmented virtuality
CN111080776B (en) Human body action three-dimensional data acquisition and reproduction processing method and system
CN111161398B (en) Image generation method, device, equipment and storage medium
CN106797458A (en) The virtual change of real object
JP2018026064A (en) Image processor, image processing method, system
CN109906600A (en) Simulate the depth of field
CN113763231A (en) Model generation method, image perspective determination device, image perspective determination equipment and medium
CN116109765A (en) Three-dimensional rendering method and device for labeling objects, computer equipment and storage medium
CN112634439B (en) 3D information display method and device
CN112652056B (en) 3D information display method and device
CN115131507B (en) Image processing method, image processing device and meta space three-dimensional reconstruction method
CN115937299B (en) Method for placing virtual object in video and related equipment
CN115272575B (en) Image generation method and device, storage medium and electronic equipment
CN116469101A (en) Data labeling method, device, electronic equipment and storage medium
JP5926626B2 (en) Image processing apparatus, control method therefor, and program
US20120162215A1 (en) Apparatus and method for generating texture of three-dimensional reconstructed object depending on resolution level of two-dimensional image
CN114332356A (en) Virtual and real picture combining method and device
CN114387378A (en) Image generation method and device based on digital twin rendering engine and electronic equipment
CN114596407A (en) Resource object three-dimensional model generation interaction method and device, and display method and device
CN113744361A (en) Three-dimensional high-precision map construction method and device based on trinocular vision
CN110490977B (en) Image processing method, system and storage medium for holographic device
Degen et al. Stereoscopic camera-sensor model for the development of highly automated driving functions within a virtual test environment

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant