CN112907733A - Method and device for reconstructing three-dimensional model and three-dimensional model acquisition and reconstruction system - Google Patents
Method and device for reconstructing three-dimensional model and three-dimensional model acquisition and reconstruction system Download PDFInfo
- Publication number
- CN112907733A CN112907733A CN202110200760.7A CN202110200760A CN112907733A CN 112907733 A CN112907733 A CN 112907733A CN 202110200760 A CN202110200760 A CN 202110200760A CN 112907733 A CN112907733 A CN 112907733A
- Authority
- CN
- China
- Prior art keywords
- dimensional
- resolution
- voxel data
- color
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 56
- 238000003062 neural network model Methods 0.000 claims abstract description 30
- 238000012545 processing Methods 0.000 claims description 25
- 238000007500 overflow downdraw method Methods 0.000 claims description 10
- 238000012549 training Methods 0.000 claims description 10
- 238000004590 computer program Methods 0.000 claims description 6
- 238000000605 extraction Methods 0.000 claims description 4
- 238000004364 calculation method Methods 0.000 abstract description 8
- 230000008569 process Effects 0.000 description 9
- 238000010586 diagram Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 4
- 238000013528 artificial neural network Methods 0.000 description 3
- 238000004422 calculation algorithm Methods 0.000 description 3
- 230000004927 fusion Effects 0.000 description 3
- 230000009466 transformation Effects 0.000 description 3
- 230000004075 alteration Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000009499 grossing Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000005070 sampling Methods 0.000 description 2
- 230000011218 segmentation Effects 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 238000011160 research Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T1/00—General purpose image data processing
- G06T1/20—Processor architectures; Processor configuration, e.g. pipelining
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
Landscapes
- Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Graphics (AREA)
- Geometry (AREA)
- Software Systems (AREA)
- Image Processing (AREA)
Abstract
The invention discloses a method and a device for reconstructing a three-dimensional model and a three-dimensional model acquisition and reconstruction system, wherein the method for reconstructing the three-dimensional model comprises the following steps: acquiring three-dimensional color point cloud data of a target object; generating three-dimensional voxel data under a first resolution according to the three-dimensional color point cloud data; inputting the three-dimensional voxel data at the first resolution into a super-resolution neural network model to generate three-dimensional voxel data at a second resolution, wherein the first resolution is less than the second resolution; and obtaining a three-dimensional model of the target object according to the three-dimensional voxel data under the second resolution. The method can improve the resolution of the three-dimensional model under the condition of not improving the calculation power of the GPU.
Description
Technical Field
The present invention relates to the field of three-dimensional technology, and in particular, to a method for reconstructing a three-dimensional model, an apparatus for reconstructing a three-dimensional model, a computer storage medium, and a three-dimensional model acquisition and reconstruction system.
Background
Three-dimensional acquisition and reconstruction of digital objects such as human bodies is an important research problem in the field of virtual reality, and with the development of technologies, dynamic acquisition of human bodies through low-cost acquisition equipment becomes a very promising application. However, due to the factors of fast dynamic speed, complex motion situation, variable surface situation and the like of the dynamic object, no good method is available for tracking and modeling the dynamic object. Meanwhile, the accuracy of modeling the dynamic object depends on the sampling accuracy and the real-time processing capability of a computer, and the existing hardware condition is very limited.
In the related art, a Dynamic object is modeled by using Dynamic Fusion (Dynamic Fusion), but the resolution of a model finally obtained is not high due to a speed limitation in this method. This speed limitation is mainly due to the voxel modeling method used in the calculation process of the method, voxel modeling is a method commonly used in computer Graphics, the final precision and resolution of the model depend on the number of cubes used by the voxel model, and the number of cubes that a computer can process in real time depends on the computing power of a GPU (Graphics Processing Unit). For example, with the Dynamic Fusion algorithm, only 256 can be processed in real time with a 1080Ti GPU3A number of voxel models, and this resolution is quite limited in high precision applications.
Disclosure of Invention
The present invention is directed to solving at least one of the problems of the prior art. To this end, it is an object of the present invention to propose a method of reconstructing a three-dimensional model that allows to increase the resolution of the three-dimensional model without increasing the computational power of the GPU.
The second objective of the present invention is to provide an apparatus for reconstructing a three-dimensional model.
It is a further object of the present invention to provide a computer storage medium.
The fourth objective of the present invention is to provide a three-dimensional model acquisition and reconstruction system.
In order to solve the above problem, an embodiment of the first aspect of the present invention provides a method for reconstructing a three-dimensional model, including: acquiring three-dimensional color point cloud data of a target object; generating three-dimensional voxel data under a first resolution according to the three-dimensional color point cloud data; inputting the three-dimensional voxel data at the first resolution into a super-resolution neural network model to generate three-dimensional voxel data at a second resolution, wherein the first resolution is less than the second resolution; and obtaining a three-dimensional model of the target object according to the three-dimensional voxel data under the second resolution.
According to the method for reconstructing the three-dimensional model, provided by the embodiment of the invention, under the condition that the GPU computing power is not basically improved, the three-dimensional voxel data under the first resolution is input into the super-resolution neural network model by adopting the super-resolution neural network model to generate the three-dimensional voxel data under the second resolution, wherein the first resolution is smaller than the second resolution, namely, the low-resolution voxel data is generated into the high-resolution voxel data, and then the three-dimensional model of the target object is obtained by the high-resolution voxel data, so that the resolution of the three-dimensional model can be improved.
In some embodiments, the super-resolution neural network model is obtained by training: acquiring a color three-dimensional point cloud sequence of an object; solving a deformation field based on a dynamic fusion method to register the color three-dimensional point cloud sequence; obtaining a first set of voxel data of the sequence of colored three-dimensional point clouds at the first resolution and a second set of voxel data of the sequence of colored three-dimensional point clouds at the second resolution; and training a basic network model by taking the first group of voxel data as input and the second group of voxel data as reference so as to obtain the super-resolution neural network model.
In some embodiments, acquiring color three-dimensional point cloud data of a target object comprises: acquiring a continuous depth image sequence and a continuous color image sequence on a target object time sequence; processing each frame of depth image to obtain three-dimensional point cloud data, and processing each frame of color image to obtain color data; and registering the three-dimensional point cloud data of the corresponding frame image with the color data to obtain the three-dimensional color point cloud data.
In some embodiments, generating three-dimensional voxel data at a first resolution from the three-dimensional color point cloud data comprises: and solving the deformation field based on a dynamic fusion method, and registering the three-dimensional color point cloud data to generate three-dimensional voxel data under the first resolution.
In some embodiments, obtaining a three-dimensional model of the target object from the three-dimensional voxel data at the second resolution comprises: and obtaining a three-dimensional model of the target object from the three-dimensional voxel data at the second resolution by a grid extraction method.
An embodiment of a second aspect of the present invention provides an apparatus for reconstructing a three-dimensional model, including: the point cloud data acquisition module is configured to acquire three-dimensional color point cloud data of a target object; the first voxel data generation module is configured to obtain three-dimensional voxel data under a first resolution ratio according to the three-dimensional color point cloud data; a second voxel data generation module configured to input the three-dimensional voxel data at the first resolution into a super-resolution neural network model to generate three-dimensional voxel data at a second resolution, wherein the first resolution is less than the second resolution; a model reconstruction module configured to obtain a three-dimensional model of the target object from the three-dimensional voxel data at the second resolution.
According to the device for reconstructing the three-dimensional model provided by the embodiment of the invention, under the condition that the GPU computing power is not basically improved, the super-resolution neural network model is adopted, the second voxel data generation module inputs the three-dimensional voxel data under the first resolution into the super-resolution neural network model to generate the three-dimensional voxel data under the second resolution, wherein the first resolution is smaller than the second resolution, namely, the low-resolution voxel data is generated into the high-resolution voxel data, and then the model reconstruction module obtains the three-dimensional model of the target object by the high-resolution voxel data, so that the resolution of the three-dimensional model can be improved.
A third embodiment of the present invention provides a computer storage medium, on which a computer program is stored, wherein the computer program is executed by a processor to implement the method for reconstructing a three-dimensional model according to the above embodiments.
An embodiment of a fourth aspect of the present invention provides a three-dimensional model acquisition and reconstruction system, including: a depth camera for acquiring a depth image of a target object; a color camera for acquiring a color image of the target object; and the data processing equipment is respectively connected with the depth camera and the color camera and is used for executing the method for reconstructing the three-dimensional model in the embodiment.
According to the three-dimensional model acquisition and reconstruction system provided by the embodiment of the invention, the data processing equipment adopts the method for reconstructing the three-dimensional model, so that the resolution of the three-dimensional model can be improved without improving the calculation power of the GPU.
In some embodiments, the depth camera comprises a ToF camera.
In some embodiments, the color camera comprises an RGB color camera.
Additional aspects and advantages of the invention will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the invention.
Drawings
The above and/or additional aspects and advantages of the present invention will become apparent and readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings of which:
FIG. 1 is a flow diagram of a method of reconstructing a three-dimensional model according to one embodiment of the invention;
FIG. 2 is a schematic diagram of a neural network training process, according to one embodiment of the present invention;
FIG. 3 is a block diagram of an apparatus for reconstructing a three-dimensional model according to an embodiment of the present invention;
fig. 4 is a block diagram of a three-dimensional model acquisition and reconstruction system according to an embodiment of the present invention.
Detailed Description
Embodiments of the present invention will be described in detail below, the embodiments described with reference to the drawings being illustrative, and the embodiments of the present invention will be described in detail below.
In order to solve the above problems, an object of the present invention is to provide a method for reconstructing a three-dimensional model, which can improve the resolution of the three-dimensional model without increasing the GPU computation power.
In the method for reconstructing a three-dimensional model provided in the embodiment of the first aspect of the present invention, in the embodiment, when the same three-dimensional color point cloud data is input, the set resolution parameters of the voxel data are different, and the resolution of the finally obtained model is also different, but the calculation amount is larger if the resolution is higher. Based on the above principle, the basic idea of the embodiment of the invention is as follows: under the condition that the GPU computing capacity is not basically improved, the voxel data under the low resolution ratio is used for computing, the voxel data under the high resolution ratio is further obtained through the super-resolution neural network model, the three-dimensional model of the target object is reconstructed through the voxel data under the high resolution ratio, and therefore compared with a method for modeling a dynamic object through directly adopting the voxel data under the low resolution ratio, the resolution ratio of the three-dimensional model can be improved.
As shown in fig. 1, the method of the embodiment of the present invention at least includes steps S1 to S4.
Step S1, three-dimensional color point cloud data of the target object is acquired.
The three-dimensional color point cloud data is formed by fusing two modal information of point cloud data and color image data.
In an embodiment, the point cloud data may be obtained by a depth camera for recording geometric position information of the target object; the color image data may be obtained by a color camera for recording color texture information of the target object. And further, processing the acquired point cloud data and the color image data to obtain three-dimensional color point cloud data of the target object.
And step S2, generating three-dimensional voxel data under the first resolution according to the three-dimensional color point cloud data.
Wherein the first resolution is a preset value, which depends on the calculation processing capability of the GPU, and can be processed to the highest degree in real time, for example, 256 times in real time in case of 1080Ti GPU3A quantity of voxel data.
In the embodiment of the invention, under the condition that the GPU computing capacity is not basically improved, the voxel data under the low resolution can be computed by adopting a dynamic fusion method, namely the three-dimensional voxel data under the first resolution is generated according to the three-dimensional color point cloud data.
Step S3, inputting the three-dimensional voxel data at the first resolution into a super-resolution neural network model to generate three-dimensional voxel data at the second resolution.
Wherein the first resolution is less than the second resolution. The second resolution is a predetermined value which depends on the resolution of the desired model of the system, e.g. 256 for the first resolution3The second resolution is 5123。
The super-resolution is to improve the resolution of the original image by a hardware or software method. The super-resolution neural network model can perform super-resolution processing on the low-resolution image to be evaluated to obtain the high-resolution reference image, and the processing process can be understood as the super-resolution neural network model learns the operation process similar to an interpolation algorithm, so that the low-resolution image to be evaluated can be converted into the high-resolution reference image.
Specifically, with the three-dimensional voxel data at the second resolution as a final target result, performing super-resolution neural network training on the three-dimensional voxel data at the first resolution, that is, inputting the low-resolution voxel data into a super-resolution neural network model to obtain the high-resolution voxel data.
Step S4, a three-dimensional model of the target object is obtained from the three-dimensional voxel data at the second resolution.
Specifically, compared with a mode of directly reconstructing the three-dimensional model of the target object by using the three-dimensional voxel data at the first resolution, the embodiment of the present invention uses the super-resolution neural network model to generate the three-dimensional voxel data at the first resolution into the three-dimensional voxel data at the second resolution, that is, to generate the low-resolution voxel data into the high-resolution voxel data, and reconstructs the three-dimensional model of the target object by using the high-resolution voxel data, that is, the three-dimensional voxel data at the second resolution. Thus, the resolution of the three-dimensional model may be increased without increasing the computational power of the GPU.
For example, in the case of 1080Ti GPU, 256 real-time processing can be performed only by using the dynamic object modeling method in the prior art3Quantity of voxel data, andthe method of the embodiment of the invention utilizes the super-resolution neural network model to convert the low-resolution voxel data 256 into the high-resolution voxel data 2563Generated as high resolution voxel data 5123So that the resolution of the final model reaches 5123I.e. the lateral resolution is improved by 1 time and the volume resolution is improved by 7 times. Therefore, the method according to the embodiment of the invention can improve the resolution of the three-dimensional model by up-sampling the voxel data.
In addition, the super-resolution neural network model is adopted, so that the calculation amount required by the reconstruction of the three-dimensional model can be reduced under the condition of the same resolution. For example, for a resolution of 163The required calculation amount of the three-dimensional model is 8 under the condition of the prior art38 times, but the calculation amount can be limited within 2 times by adopting the method of the embodiment of the invention, that is, if the acquisition resolution is 16 times3The embodiment of the invention can be applied to 83The low-resolution voxel data is calculated and obtained by using a super-resolution neural network model 163To perform three-dimensional model reconstruction. Therefore, the method for reconstructing the three-dimensional model can realize the modeling of the high-resolution voxel model with less computing power.
According to the method for reconstructing the three-dimensional model, provided by the embodiment of the invention, under the condition that the GPU computing power is not basically improved, the three-dimensional voxel data under the first resolution is input into the super-resolution neural network model by adopting the super-resolution neural network model to generate the three-dimensional voxel data under the second resolution, wherein the first resolution is smaller than the second resolution, namely, the low-resolution voxel data is generated into the high-resolution voxel data, and then the three-dimensional model of the target object is obtained by the high-resolution voxel data, so that the resolution of the three-dimensional model can be improved.
In some embodiments, embodiments of the invention are obtained by training a super-resolution neural network model by: acquiring a color three-dimensional point cloud sequence of an object; solving the deformation field based on a dynamic fusion method to register the color three-dimensional point cloud sequence; obtaining a first group of voxel data of the color three-dimensional point cloud sequence at a first resolution, and obtaining a second group of voxel data of the color three-dimensional point cloud sequence at a second resolution; and training a basic network model by taking the first group of voxel data as input and the second group of voxel data as reference so as to obtain a super-resolution neural network model.
Specifically, for the used super-resolution neural network model, the training process includes generating two sets of voxel data for the color three-dimensional point cloud sequence under the conditions of low resolution, i.e., first resolution, and high resolution, i.e., second resolution, respectively, and then training the neural network with the low resolution data, i.e., the first set of voxel data, as input and the high resolution data, i.e., the second set of voxel data, as reference, for example, as shown in fig. 2.
In some embodiments, acquiring color three-dimensional point cloud data of a target object may include acquiring a depth image sequence and a color image sequence which are continuous in time sequence of the target object; processing each frame of depth image to obtain three-dimensional point cloud data, and processing each frame of color image to obtain color data; and registering the three-dimensional point cloud data of the corresponding frame image with the color data to obtain three-dimensional color point cloud data.
The processing of the obtained depth image sequence and color image sequence may include denoising, smoothing, foreground and background segmentation, and the like. Specifically, since a background, a non-processed target in the environment, and the like are usually captured in the initially obtained depth image, in order to accurately reconstruct a three-dimensional model of an actual target object in a computer environment, an image of the target object may be obtained through denoising processing and a smoothing algorithm. And separating the three-dimensional modeling target object from the background through foreground and background segmentation processing. And by acquiring the continuous depth image sequence and the continuous color image sequence on the target object time sequence and processing each frame of image, the three-dimensional model of the target object can be conveniently and dynamically reconstructed in real time.
In some embodiments, for generating voxel data at a first resolution from the three-dimensional color point cloud data, the embodiment of the invention may include, solving the deformation field based on a dynamic fusion method, and registering the three-dimensional color point cloud data to generate voxel data at the first resolution.
The core of the dynamic fusion method is to establish a model under a key frame, and then changes of scenes can be corresponded to the model through geometric transformation, namely, the depth image which is newly read each time is fused into the model under the key frame after being geometrically transformed, and the transformation process is equivalent to the cancellation of the transformation of the scenes. Therefore, the scene becomes more real and full gradually by adopting a dynamic fusion method.
In the embodiment, during shooting, only partial information of a human body is shot by a shot frame of image, a plurality of frames of images shot through different angles contain a certain common part, and therefore, the images need to be registered when a complete three-dimensional model is generated. Specifically, a deformation field is solved based on a dynamic fusion method, and a common part is used as a reference, and multi-frame images acquired under different photographing parameters such as time, angle and illumination are overlapped and matched into a unified coordinate system to generate three-dimensional voxel data under a first resolution.
In some embodiments, a three-dimensional model of the target object is obtained from the three-dimensional voxel data at the second resolution by a mesh extraction method.
In the embodiment, the three-dimensional voxel data contains the adjacency relation between the corresponding spatial points, and whether the adjacent points are spliced or not is determined through different judgment modes. The embodiment of the invention adopts a mesh extraction method to obtain a three-dimensional model of a target object, for example, triangular meshes are established, points of the same plane or points with the distance within a threshold value range are connected into the triangular meshes according to the distance relation of space points in space, and after the meshes of each area are established, the meshes are spliced to generate the three-dimensional model. Alternatively, the mesh with different shapes may be established according to the actual depth image, which is not limited in this respect.
As shown in fig. 2, an apparatus 10 according to an embodiment of the present invention includes a point cloud data obtaining module 1, a first voxel data generating module 2, a second voxel data generating module 3, and a model reconstructing module 4.
The point cloud data acquisition module 1 is configured to acquire three-dimensional color point cloud data of a target object; the first voxel data generation module 2 is configured to obtain three-dimensional voxel data at a first resolution according to the three-dimensional color point cloud data; the second voxel data generation module 3 is configured to input the three-dimensional voxel data at the first resolution into the super-resolution neural network model to generate the three-dimensional voxel data at the second resolution, wherein the first resolution is smaller than the second resolution; the model reconstruction module 4 is configured to obtain a three-dimensional model of the target object from the three-dimensional voxel data at the second resolution.
It should be noted that a specific implementation manner of the apparatus 10 for reconstructing a three-dimensional model according to the embodiment of the present invention is similar to a specific implementation manner of the method for reconstructing a three-dimensional model according to any of the above embodiments of the present invention, and please refer to the description of the method part specifically, and details are not described here for reducing redundancy.
According to the apparatus 10 for reconstructing a three-dimensional model of an embodiment of the present invention, without substantially increasing the GPU computation power, the second voxel data generation module 3 inputs the three-dimensional voxel data at the first resolution to the super-resolution neural network model by using the super-resolution neural network model to generate the three-dimensional voxel data at the second resolution, where the first resolution is smaller than the second resolution, that is, the low-resolution voxel data is generated as the high-resolution voxel data, and the model reconstruction module 4 obtains the three-dimensional model of the target object from the high-resolution voxel data, thereby increasing the resolution of the three-dimensional model.
A third embodiment of the present invention provides a computer storage medium, on which a computer program is stored, wherein the computer program, when executed by a processor, implements the method for reconstructing a three-dimensional model provided by the above embodiments.
A fourth embodiment of the present invention provides a three-dimensional model acquisition and reconstruction system, as shown in fig. 3, the three-dimensional model acquisition and reconstruction system 20 according to the embodiment of the present invention includes a depth camera 5, a color camera 6 and a data processing device 7.
Wherein, the depth camera 5 is used for acquiring a depth image of the target object; the color camera 6 is used for acquiring a color image of the target object; the data processing device 7 is connected to the depth camera 5 and the color camera 6, respectively, for performing the method of reconstructing a three-dimensional model as provided in the above embodiments.
It should be noted that a specific implementation manner of the data processing device 7 is similar to a specific implementation manner of the method for reconstructing a three-dimensional model according to any of the above embodiments of the present invention, and please refer to the description of the method part specifically, and details are not described here for reducing redundancy.
According to the three-dimensional model acquisition and reconstruction system 20 of the embodiment of the present invention, the data processing device 7 can improve the resolution of the three-dimensional model without improving the GPU computation power by using the method for reconstructing the three-dimensional model provided in the above-described embodiment.
In some embodiments, the depth camera 5 may comprise a ToF camera.
In some embodiments, the color camera 6 may comprise an RGB color camera.
In the description of this specification, any process or method descriptions in flow charts or otherwise described herein may be understood as representing modules, segments, or portions of code which include one or more executable instructions for implementing steps of custom logic functions or processes, and alternate implementations are included within the scope of the preferred embodiment of the present invention in which functions may be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved, as would be understood by those reasonably skilled in the art of the embodiments of the present invention.
The logic and/or steps represented in the flowcharts or otherwise described herein, e.g., an ordered listing of executable instructions that can be considered to implement logical functions, can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For the purposes of this description, a "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of the computer-readable medium would include the following: an electrical connection (electronic device) having one or more wires, a portable computer diskette (magnetic device), a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium could even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, via for instance optical scanning of the paper or other medium, then compiled, interpreted or otherwise processed in a suitable manner if necessary, and then stored in a computer memory.
It should be understood that portions of the present invention may be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, the various steps or methods may be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. If implemented in hardware, as in another embodiment, any one or combination of the following techniques, which are known in the art, may be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application specific integrated circuit having an appropriate combinational logic gate circuit, a Programmable Gate Array (PGA), a Field Programmable Gate Array (FPGA), or the like.
It will be understood by those skilled in the art that all or part of the steps carried by the method for implementing the above embodiments may be implemented by hardware related to instructions of a program, which may be stored in a computer readable storage medium, and when the program is executed, the program includes one or a combination of the steps of the method embodiments.
In addition, functional units in the embodiments of the present invention may be integrated into one processing module, or each unit may exist alone physically, or two or more units are integrated into one module. The integrated module can be realized in a hardware mode, and can also be realized in a software functional module mode. The integrated module, if implemented in the form of a software functional module and sold or used as a stand-alone product, may also be stored in a computer readable storage medium.
The storage medium mentioned above may be a read-only memory, a magnetic or optical disk, etc. Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention, and that variations, modifications, substitutions and alterations can be made to the above embodiments by those of ordinary skill in the art within the scope of the present invention.
In the description herein, references to the description of the term "one embodiment," "some embodiments," "an illustrative embodiment," "an example," "a specific example," or "some examples" or the like mean that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, the schematic representations of the terms used above do not necessarily refer to the same embodiment or example.
While embodiments of the invention have been shown and described, it will be understood by those of ordinary skill in the art that: various changes, modifications, substitutions and alterations can be made to the embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.
Claims (10)
1. A method of reconstructing a three-dimensional model, comprising:
acquiring three-dimensional color point cloud data of a target object;
generating three-dimensional voxel data under a first resolution according to the three-dimensional color point cloud data;
inputting the three-dimensional voxel data at the first resolution into a super-resolution neural network model to generate three-dimensional voxel data at a second resolution, wherein the first resolution is less than the second resolution;
and obtaining a three-dimensional model of the target object according to the three-dimensional voxel data under the second resolution.
2. The method of reconstructing a three-dimensional model according to claim 1, wherein the super-resolution neural network model is obtained by training:
acquiring a color three-dimensional point cloud sequence of an object;
solving a deformation field based on a dynamic fusion method to register the color three-dimensional point cloud sequence;
obtaining a first set of voxel data of the sequence of colored three-dimensional point clouds at the first resolution and a second set of voxel data of the sequence of colored three-dimensional point clouds at the second resolution;
and training a basic network model by taking the first group of voxel data as input and the second group of voxel data as reference so as to obtain the super-resolution neural network model.
3. The method of reconstructing a three-dimensional model according to claim 1, wherein acquiring color three-dimensional point cloud data of a target object comprises:
acquiring a continuous depth image sequence and a continuous color image sequence on a target object time sequence;
processing each frame of depth image to obtain three-dimensional point cloud data, and processing each frame of color image to obtain color data;
and registering the three-dimensional point cloud data of the corresponding frame image with the color data to obtain the three-dimensional color point cloud data.
4. The method of reconstructing a three-dimensional model according to claim 1, wherein generating three-dimensional voxel data at a first resolution from the three-dimensional color point cloud data comprises:
and solving the deformation field based on a dynamic fusion method, and registering the three-dimensional color point cloud data to generate three-dimensional voxel data under the first resolution.
5. The method of reconstructing a three-dimensional model according to claim 1, wherein obtaining a three-dimensional model of the target object from the three-dimensional voxel data at the second resolution comprises:
and obtaining a three-dimensional model of the target object from the three-dimensional voxel data at the second resolution by a grid extraction method.
6. An apparatus for reconstructing a three-dimensional model, comprising:
the point cloud data acquisition module is configured to acquire three-dimensional color point cloud data of a target object;
the first voxel data generation module is configured to obtain three-dimensional voxel data under a first resolution ratio according to the three-dimensional color point cloud data;
a second voxel data generation module configured to input the three-dimensional voxel data at the first resolution into a super-resolution neural network model to generate three-dimensional voxel data at a second resolution, wherein the first resolution is less than the second resolution;
a model reconstruction module configured to obtain a three-dimensional model of the target object from the three-dimensional voxel data at the second resolution.
7. A computer storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method of reconstructing a three-dimensional model of any of claims 1-5.
8. A three-dimensional model acquisition and reconstruction system, comprising:
a depth camera for acquiring a depth image of a target object;
a color camera for acquiring a color image of the target object;
a data processing device, connected to said depth camera and said color camera, respectively, for performing the method of reconstructing a three-dimensional model of any one of claims 1-5.
9. The three-dimensional model acquisition and reconstruction system of claim 8, wherein the depth camera comprises a ToF camera.
10. The three-dimensional model acquisition and reconstruction system according to claim 8 or 9, wherein said color camera comprises an RGB color camera.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110200760.7A CN112907733A (en) | 2021-02-23 | 2021-02-23 | Method and device for reconstructing three-dimensional model and three-dimensional model acquisition and reconstruction system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110200760.7A CN112907733A (en) | 2021-02-23 | 2021-02-23 | Method and device for reconstructing three-dimensional model and three-dimensional model acquisition and reconstruction system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112907733A true CN112907733A (en) | 2021-06-04 |
Family
ID=76124387
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110200760.7A Pending CN112907733A (en) | 2021-02-23 | 2021-02-23 | Method and device for reconstructing three-dimensional model and three-dimensional model acquisition and reconstruction system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112907733A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115239891A (en) * | 2022-09-22 | 2022-10-25 | 无锡日联科技股份有限公司 | Three-dimensional reconstruction method and device, electronic equipment and storage medium |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104778659A (en) * | 2015-04-15 | 2015-07-15 | 杭州电子科技大学 | Single-frame image super-resolution reconstruction method on basis of deep learning |
| CN106600553A (en) * | 2016-12-15 | 2017-04-26 | 华中科技大学 | DEM super-resolution method based on convolutional neural network |
| CN110060332A (en) * | 2019-04-09 | 2019-07-26 | 上海科技大学 | High-precision three-dimensional based on airborne acquisition equipment builds figure and modeling |
| CN110335340A (en) * | 2019-05-14 | 2019-10-15 | 广东康云科技有限公司 | Object threedimensional model generation method, system and storage medium based on super-pixel |
| CN110390638A (en) * | 2019-07-22 | 2019-10-29 | 北京工商大学 | A kind of high-resolution three-dimension voxel model method for reconstructing |
| CN110428378A (en) * | 2019-07-26 | 2019-11-08 | 北京小米移动软件有限公司 | Processing method, device and the storage medium of image |
| US20200258300A1 (en) * | 2018-12-20 | 2020-08-13 | Carl Zeiss Vision International Gmbh | Method and apparatus for generating a 3d reconstruction of an object |
| CN111968238A (en) * | 2020-08-22 | 2020-11-20 | 晋江市博感电子科技有限公司 | Human body color three-dimensional reconstruction method based on dynamic fusion algorithm |
-
2021
- 2021-02-23 CN CN202110200760.7A patent/CN112907733A/en active Pending
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104778659A (en) * | 2015-04-15 | 2015-07-15 | 杭州电子科技大学 | Single-frame image super-resolution reconstruction method on basis of deep learning |
| CN106600553A (en) * | 2016-12-15 | 2017-04-26 | 华中科技大学 | DEM super-resolution method based on convolutional neural network |
| US20200258300A1 (en) * | 2018-12-20 | 2020-08-13 | Carl Zeiss Vision International Gmbh | Method and apparatus for generating a 3d reconstruction of an object |
| CN110060332A (en) * | 2019-04-09 | 2019-07-26 | 上海科技大学 | High-precision three-dimensional based on airborne acquisition equipment builds figure and modeling |
| CN110335340A (en) * | 2019-05-14 | 2019-10-15 | 广东康云科技有限公司 | Object threedimensional model generation method, system and storage medium based on super-pixel |
| CN110390638A (en) * | 2019-07-22 | 2019-10-29 | 北京工商大学 | A kind of high-resolution three-dimension voxel model method for reconstructing |
| CN110428378A (en) * | 2019-07-26 | 2019-11-08 | 北京小米移动软件有限公司 | Processing method, device and the storage medium of image |
| CN111968238A (en) * | 2020-08-22 | 2020-11-20 | 晋江市博感电子科技有限公司 | Human body color three-dimensional reconstruction method based on dynamic fusion algorithm |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115239891A (en) * | 2022-09-22 | 2022-10-25 | 无锡日联科技股份有限公司 | Three-dimensional reconstruction method and device, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109285215B (en) | Human body three-dimensional model reconstruction method and device and storage medium | |
| CN109003325B (en) | Three-dimensional reconstruction method, medium, device and computing equipment | |
| CN106803267B (en) | Kinect-based indoor scene three-dimensional reconstruction method | |
| CN112330729B (en) | Image depth prediction method, device, terminal equipment and readable storage medium | |
| Lim et al. | Surface reconstruction techniques: a review | |
| US9984498B2 (en) | Sparse GPU voxelization for 3D surface reconstruction | |
| Tretschk et al. | Demea: Deep mesh autoencoders for non-rigidly deforming objects | |
| CN108898630A (en) | A kind of three-dimensional rebuilding method, device, equipment and storage medium | |
| US20170278302A1 (en) | Method and device for registering an image to a model | |
| CN114863038B (en) | Real-time dynamic free visual angle synthesis method and device based on explicit geometric deformation | |
| CN112562082A (en) | Three-dimensional face reconstruction method and system | |
| CN110942512B (en) | Indoor scene reconstruction method based on meta-learning | |
| CN109191554A (en) | A kind of super resolution image reconstruction method, device, terminal and storage medium | |
| KR20220120674A (en) | 3D reconstruction method, apparatus, apparatus and storage medium | |
| CN114387392B (en) | Method for reconstructing three-dimensional human body posture according to human shadow | |
| CN117671138A (en) | Digital twin modeling method and system based on SAM large model and NeRF | |
| CN114972634A (en) | Multi-view three-dimensional deformable human face reconstruction method based on feature voxel fusion | |
| CN108655571A (en) | A kind of digital-control laser engraving machine, control system and control method, computer | |
| Pesavento et al. | Super-resolution 3D human shape from a single low-resolution image | |
| CN112907733A (en) | Method and device for reconstructing three-dimensional model and three-dimensional model acquisition and reconstruction system | |
| CN116091705A (en) | Variable topology dynamic scene reconstruction and editing method and device based on nerve radiation field | |
| Ren et al. | Shape recovery of endoscopic videos by shape from shading using mesh regularization | |
| CN114332156A (en) | Real-time three-dimensional motion completion method based on graph convolution neural network | |
| CN115239559A (en) | Depth map super-resolution method and system for fusion view synthesis | |
| Szirmay-Kalos | Filtering and gradient estimation for distance fields by quadratic regression |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |