CN116645471A - Modeling method, system, equipment and storage medium for extracting foreground object - Google Patents
Modeling method, system, equipment and storage medium for extracting foreground object Download PDFInfo
- Publication number
- CN116645471A CN116645471A CN202310628354.XA CN202310628354A CN116645471A CN 116645471 A CN116645471 A CN 116645471A CN 202310628354 A CN202310628354 A CN 202310628354A CN 116645471 A CN116645471 A CN 116645471A
- Authority
- CN
- China
- Prior art keywords
- foreground
- data
- point
- background
- color
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 39
- 238000003860 storage Methods 0.000 title claims abstract description 16
- 238000005070 sampling Methods 0.000 claims description 32
- 238000009877 rendering Methods 0.000 claims description 17
- 230000006870 function Effects 0.000 claims description 13
- 238000012545 processing Methods 0.000 claims description 9
- 238000000926 separation method Methods 0.000 claims description 4
- 239000003086 colorant Substances 0.000 claims description 3
- 238000012549 training Methods 0.000 claims description 2
- 238000002372 labelling Methods 0.000 abstract description 5
- 238000010586 diagram Methods 0.000 description 12
- 238000012360 testing method Methods 0.000 description 8
- 238000004590 computer program Methods 0.000 description 7
- 230000011218 segmentation Effects 0.000 description 6
- 238000005516 engineering process Methods 0.000 description 5
- 238000010276 construction Methods 0.000 description 4
- 230000000694 effects Effects 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 238000004088 simulation Methods 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 2
- 238000006243 chemical reaction Methods 0.000 description 2
- 239000011159 matrix material Substances 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000003672 processing method Methods 0.000 description 2
- 230000009466 transformation Effects 0.000 description 2
- 238000013473 artificial intelligence Methods 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/194—Segmentation; Edge detection involving foreground-background segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10004—Still image; Photographic image
- G06T2207/10012—Stereo images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10024—Color image
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Geometry (AREA)
- Computer Graphics (AREA)
- Artificial Intelligence (AREA)
- Computing Systems (AREA)
- Databases & Information Systems (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Multimedia (AREA)
- Image Analysis (AREA)
Abstract
The embodiment of the application provides a modeling method, a modeling system, modeling equipment and a modeling storage medium for extracting a foreground object, and belongs to the technical field of three-dimensional reconstruction. The modeling method for extracting the foreground object comprises the following steps: acquiring image point cloud data of a target object and camera pose information; separating a foreground from a background in the image point cloud data to obtain a foreground object point cloud; determining a minimum bounding box of the foreground according to the extracted foreground object point cloud; based on the data, foreground point data and background point data are determined; based on foreground point data and background point data, determining foreground reconstruction data and background reconstruction data by using a preset reconstruction network; and generating and displaying a target object model according to the foreground reconstruction data and the background reconstruction data. The embodiment of the application utilizes the minimum bounding box to calibrate the foreground and the background, reduces the cost of manual labeling, predicts the model data by utilizing the preset reconstruction network, and improves the accuracy and the precision of model reconstruction.
Description
Technical Field
The present application relates to the field of three-dimensional reconstruction technology, and in particular, to a modeling method for extracting a foreground object, a modeling system for extracting a foreground object, an electronic device, and a computer storage medium.
Background
With the rapid development of artificial intelligence, automatic driving of automobiles has been developed. At present, many automobile companies already apply the automatic driving technology to the auxiliary driving function, and the testing method of the automatic driving algorithm mainly comprises two kinds of real road scene testing and virtual simulation scene testing. The test based on the real road scene has high cost, low efficiency and high risk coefficient, and the test based on the simulation scene can improve the efficiency and the safety to a certain extent, and can reduce the cost. Thus, most autopilot algorithm researchers will test autopilot algorithms using a simulation test scenario.
The three-dimensional object model of the scene is required to be established based on the test of the simulation scene, the foreground and the background are segmented by utilizing a segmentation network in the existing three-dimensional object modeling technology, but the accuracy of the segmentation result based on the method is not high, the requirement of segmentation labeling is of a pixel level, and a large amount of labor cost is consumed.
Disclosure of Invention
The application aims to provide a modeling method for extracting a foreground object, so as to solve the technical problem of low segmentation accuracy in the prior art; secondly, a modeling system for extracting a foreground object is provided; a third object is to provide an electronic device; a fourth object is to provide a computer storage medium.
In order to achieve the above purpose, the technical scheme adopted by the application is as follows:
a modeling method of extracting a foreground object, the method comprising:
acquiring image point cloud data of a target object and camera pose information;
separating a foreground from a background in the image point cloud data to obtain a foreground object point cloud;
determining a minimum bounding box of the foreground according to the extracted foreground object point cloud;
determining foreground point data and background point data based on the camera pose information, the foreground minimum bounding box and the image point cloud data;
based on foreground point data and background point data, determining foreground reconstruction data and background reconstruction data by using a preset reconstruction network;
and generating and displaying a target object model according to the foreground reconstruction data and the background reconstruction data.
According to the technical characteristics, the foreground is extracted from the background by utilizing the minimum bounding box to obtain the reconstruction area of the foreground object and the background, then the foreground point and the background point are determined according to the pose information, the minimum bounding box of the foreground and the image point cloud data, and then the foreground point and the background point are reconstructed by utilizing a preset reconstruction network, so that a target object model is finally obtained, and the segmentation accuracy and the model construction accuracy are improved.
Further, the preset reconstruction network comprises an implicit surface reconstruction network and a color prediction network;
the generating foreground reconstruction data based on the foreground point data by using a preset reconstruction network includes:
inputting the foreground point data into the implicit surface reconstruction network, and determining the distance from each foreground point in the foreground object to a preset implicit surface;
inputting the foreground point data into the color prediction network to determine the color of each foreground point in the foreground object;
and obtaining reconstruction data of the foreground objects according to the distance from each foreground point to the preset implicit surface and the color of each foreground point in the foreground objects.
According to the technical characteristics, the implicit surface reconstruction network and the color prediction network are utilized to predict the distance from each foreground point in the foreground object to the preset implicit surface and the color of the foreground point, so that foreground object reconstruction data are obtained, and a data base is provided for subsequent model construction.
Further, the determining the background reconstruction data based on the background point data by using a preset reconstruction network includes:
inputting the background points into the implicit surface reconstruction network, and determining the distance between each background point in the background and a preset implicit surface;
inputting the background points into the color prediction network to determine the color of each background point in the background;
and generating the background reconstruction data according to the distance from each background point in the background to the preset implicit surface and the color of each background point.
According to the technical characteristics, the implicit surface reconstruction network and the color prediction network are utilized to predict the distance from each background point in the background to the preset implicit surface and the color of the background point, so that background reconstruction data is obtained, and a data base is provided for the subsequent model construction.
Further, the determining foreground point data and background point data based on the camera pose information, the foreground minimum bounding box, and the image point cloud data includes:
generating light rays passing through a foreground minimum bounding box based on the image point cloud data and the camera pose information;
sampling the light passing through the minimum bounding box of the foreground to obtain light sampling data;
calculating to obtain the position information of each light sampling point based on the light sampling data;
judging whether the position information of the light sampling point meets preset foreground position information or not;
if yes, judging the light sampling point as a foreground point, and obtaining foreground point data;
if not, judging the light sampling point as a background point, and obtaining the background point data.
According to the technical characteristics, the light rays which pass through the reconstruction space and correspond to the pixels are generated, namely, the light ray directions which correspond to the pixels are determined, the position of each pixel can be calculated, and the background point and the foreground point are determined according to the calculated positions of the pixels, so that the accuracy of dividing the foreground and the background is improved.
Further, rendering the target object model based on the color of each foreground point in the foreground object and the color of each background point in the background, and calculating color data of each pixel point after rendering by using the formula (1):
in the formula (1), the components are as follows,representing pixel color; n represents the number of sampling points on ray and N 1 Indicating the number of foreground points on the light, N 2 N represents the number of background points on the light 1 +N 2 ＝N；w(p i ) Color expression weight of the ith foreground point on the light is represented; w (p) j ) Color expression weights representing the jth background point on the light; c (p) i ) Representing foreground point p i Is a predicted color of (2); c (p) j ) Representing background point p j Is used for the prediction color of (a).
Further, the generating and displaying a target object model according to the foreground reconstruction data and the background reconstruction data includes:
reconstructing to obtain an initial object model based on the distance from each foreground point to the preset implicit surface and the distance from each background point to the preset implicit surface;
calculating a color loss value according to the color data of each pixel point after rendering;
judging whether the color loss value is smaller than a preset color loss value or not;
if the color loss value is smaller than the preset color loss value, the current initial object model is saved as a target object model;
and displaying the target object model.
According to the technical characteristics, calculating the color loss value of each pixel point after rendering, and if the color loss value is smaller than the preset color loss value, storing the current target object model so as to ensure the effect of model rendering and improve the model reduction degree.
Further, the calculating a color loss value according to the color data of each pixel point after the rendering includes:
calculating a color loss value according to the color data of each pixel point after rendering by using the formula (2):
l sum ＝l color +β·l reg (2)
in the formula (2), l sum Is a color loss value; l (L) color Is a color loss function, expressed as Representing the rendered pixel color; c (ray) is the true color of the pixel; l (L) reg Constraint loss function for regular term, expressed as +.>M represents the batch size of light rays which are sent to training in the color prediction network; n represents the number of sampling points on each light ray; />Representing the predicted point p of the implicit surface reconstruction network i Distance S to preset implicit surface to point p i Beta is a hyper-parameter.
A modeling system for extracting foreground objects, the system comprising:
the data acquisition module is used for acquiring image point cloud data of the target object and camera pose information;
the image separation module is used for separating the foreground from the background in the image point cloud data to obtain foreground object point clouds;
the bounding box determining module is used for determining a minimum foreground bounding box according to the extracted foreground object point cloud;
the data determining module is used for determining foreground point data and background point data based on the camera pose information, the minimum foreground bounding box and the image point cloud data;
the data processing module is used for determining foreground reconstruction data and background reconstruction data by utilizing a preset reconstruction network based on the foreground point data and the background point data;
and the model display module is used for generating and displaying a target object model according to the foreground reconstruction data and the background reconstruction data.
An electronic device configured to perform the modeling method of extracting a foreground object described above.
A computer readable storage medium having instructions stored thereon for causing a machine to perform the steps described in any one of the possible implementations of the modeling method of extracting a foreground object described above.
The application has the beneficial effects that:
(1) According to the method, the model data required by the model reconstruction is predicted by using the preset reconstruction network, so that the accuracy and the accuracy of the model reconstruction are improved;
(2) The application utilizes the minimum bounding box to calibrate the reconstruction range of the foreground object and the background, does not need to carry out pixel-level labeling, and reduces the cost of manual labeling;
(3) The method provided by the application has better universality and is suitable for most virtual engines and simulation software.
Drawings
The accompanying drawings are included to provide a further understanding of embodiments of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and together with the description serve to explain, without limitation, the embodiments of the application. In the drawings:
FIG. 1 is a flow diagram schematically illustrating a modeling method of extracting a foreground object according to an embodiment of the present application;
FIG. 2 is a schematic diagram schematically illustrating foreground minimum bounding box calibration;
FIG. 3 is a schematic diagram schematically illustrating an axis alignment bounding box transformation;
FIG. 4 is a schematic diagram schematically illustrating ray generation;
fig. 5 is an architecture diagram schematically illustrating a modeling system for extracting foreground objects according to an embodiment of the present application.
Detailed Description
Further advantages and effects of the present application will become readily apparent to those skilled in the art from the disclosure herein, by referring to the accompanying drawings and the preferred embodiments. The application may be practiced or carried out in other embodiments that depart from the specific details, and the details of the present description may be modified or varied from the spirit and scope of the present application. It should be understood that the preferred embodiments are presented by way of illustration only and not by way of limitation.
It should be noted that the illustrations provided in the following embodiments merely illustrate the basic concept of the present application by way of illustration, and only the components related to the present application are shown in the drawings and are not drawn according to the number, shape and size of the components in actual implementation, and the form, number and proportion of the components in actual implementation may be arbitrarily changed, and the layout of the components may be more complicated.
The present embodiment provides a radar data processing method, and referring to fig. 1, fig. 1 is a schematic flow chart of the radar data processing method provided in the present embodiment.
Step S100: and acquiring image point cloud data of the target object and camera pose information.
Specifically, a multi-view two-dimensional image of the target object is collected, camera pose information of the multi-view image is estimated based on the multi-view two-dimensional image, image point cloud data with background noise is determined by using a sparse reconstruction method, for example, the multi-view two-dimensional image of the target object is obtained, and camera focus coordinates o epsilon R of each two-dimensional image are estimated 3 Converting the focus coordinate into a world coordinate system according to a conversion matrix of the camera coordinate system, wherein the conversion matrix is T epsilon R 4×4 Wherein R is a real number.
Step S200: and separating the foreground from the background in the image point cloud data to obtain a foreground object point cloud.
The method comprises the steps of identifying image point cloud data, identifying background point cloud and foreground object point cloud, removing the background point cloud from an image to finish segmentation of a foreground object and a background, dividing the foreground object point cloud and the background point cloud, and determining reconstruction ranges of the foreground object and the background.
Step S300: and determining a minimum foreground bounding box according to the extracted foreground object point cloud.
The minimum bounding box is also called an external minimum rectangle, and is an algorithm for solving the optimal bounding space of the discrete point set.
It can be understood that, in order to improve the accuracy of the division of the minimum bounding box, in this embodiment, the foreground minimum bounding box is subjected to an axial alignment bounding box transformation, the calibrated foreground minimum bounding box is shown in fig. 2, and the foreground minimum bounding box is translated and rotated, so that the center of the foreground minimum bounding box is located at the origin of the world coordinate system, and the length, width and height of the foreground minimum bounding box are respectively parallel to the x, y and z axes of the world coordinate system (as shown in fig. 3), and are respectively denoted as a, b and c, and the pose of the image is correspondingly spatially transformed according to the translation and rotation of the foreground minimum bounding box, and the background and the foreground object are accurately separated through the calibration of the foreground minimum bounding box.
Step S400: and determining foreground point data and background point data based on the camera pose information, the minimum foreground bounding box and the image point cloud data.
Specifically, as shown in fig. 4, according to the image point cloud data and the camera position information, the present embodiment generates a plurality of light rays passing through the corresponding pixel d from the camera focus o and passing through the minimum bounding box of the foreground, and can determine that the direction of the light rays is expressed asSampling the light according to the starting point and the direction of the light to obtain light sampling data, and calculating the position information of each sampling point based on the light sampling data by using the following formulaAnd (3) extinguishing:
wherein p is the position of the sampling point, o is the focal coordinates of the camera, t is the relative distance of each sampling point from the light ray starting point,is the direction of light.
After determining the position information of each sampling point, the embodiment also provides that whether the position information of the light sampling point meets the preset foreground position information is judged; if yes, judging the light sampling point as a foreground point, and obtaining foreground point data, wherein the foreground point data comprises foreground point position, color and other data; if the data does not meet the requirement, the light sampling points are judged to be background points, so that background point data are obtained, and the background point data comprise data such as the positions and the colors of the background points. For example, the length, width and height of the foreground minimum bounding box are a, b and c respectively, and when the coordinates (x, y, z) of the sampling point p satisfy the following conditions simultaneouslyDividing the point p into foreground points in the foreground points to further obtain foreground point data; otherwise, dividing the background point into background points to further obtain background point data.
Step S500: and determining foreground reconstruction data and background reconstruction data by using a preset reconstruction network based on the foreground point data and the background point data.
The preset reconstruction network comprises an implicit surface reconstruction network and a color prediction network, the implicit surface reconstruction network and the color prediction network are realized based on a multi-layer perception neural network, and the implicit surface reconstruction network adopts a signed distance function to represent the distance from any point in the space to the preset implicit surface so as to generate the distance from a foreground point to the preset implicit surface and the distance from a background point to the preset implicit surface; the color prediction network predicts the color of the sampling point on the light passing through the minimum bounding box of the foreground to obtain the color of the foreground and the color of the background. It will be appreciated that the foreground reconstruction data includes the distance of the foreground points to the preset implicit surface and the color of the foreground points, and the background reconstruction data includes the distance of the background points to the preset implicit surface and the color of the background points.
The generation of reconstruction data for foreground objects specifically includes: inputting foreground point data into an implicit surface reconstruction network, and predicting the distance S epsilon R from the coordinate information p of each foreground point to a preset implicit surface 1 Feature vector f e R 256 Normal n e R 3 The implicit reconstruction network can be expressed as p→ (S, f, n), the foreground point p is input into the color prediction network, and the color prediction network predicts the color of the foreground point according to the coordinate information p of the foreground point and the light directionThe normal n and the feature vector f in the implicit surface reconstruction network predict the color of each foreground point, and the color prediction network can be expressed as +.>c is the color of the foreground point, and c epsilon R 3 The method comprises the steps of carrying out a first treatment on the surface of the The step of generating the background reconstruction data is the same as the step of generating the foreground reconstruction data, and will not be described in detail here.
Step S600: and generating and displaying a target object model according to the foreground reconstruction data and the background reconstruction data.
In this step, based on the distance from each foreground point to the preset implicit surface and the distance from each background point to the preset implicit surface, an initial object model is obtained by reconstruction, and since the obtained initial object model is not subjected to color rendering, the embodiment also needs to render the initial object model based on the color result predicted by the color prediction model. Specifically, the color of each pixel point after rendering can be calculated by using the following formula:
wherein,,representing pixel color; n represents the number of sampling points on ray and N 1 Indicating the number of foreground points on the light, N 2 N represents the number of background points on the light 1 +N 2 ＝N；w(p i ) Color expression weight of the ith foreground point on the light is represented; w (p) j ) Color expression weights representing the jth background point on the light; c (p) i ) Representing foreground point p i Is a predicted color of (2); c (p) j ) Representing background point p j Is used for the prediction color of (a).
After the color of each pixel point after rendering is obtained, the embodiment also compares the true color of the pixel point with the color of each pixel point after rendering, calculates a color loss value, judges whether the color loss value is smaller than a preset color loss value after obtaining the color loss value of each pixel, if so, indicates that the current color prediction effect is good, and stores the current target object model, if not, indicates that the current color prediction effect is bad, and needs to reconstruct the model, thereby improving the accuracy of model reconstruction and ensuring the restoration degree of model reconstruction; wherein, the color loss value can be calculated by the following formula:
l sum ＝l color +β·l reg ，
wherein l sum For the color loss value, l color Is a color loss function, and can be expressed asM represents the light batch size trained into the color prediction network, < >>Representing the color of the pixel after rendering, c (ray) is the true color of the pixel, and it can be understood that, in order to ensure the reasonability of the deformation space, the embodiment adds a regularization term constraint loss function l reg Can be expressed as->M represents the batch size of light trained in the color prediction network, N represents the number of sampling points on each light, and +.>Representing the predicted point p of the implicit surface reconstruction network i Distance S to preset implicit surface to point p i Beta is a hyper-parameter representing the weight of the regular term loss.
The method can adopt a Marching cube algorithm (Marching Cubes) to extract the target object model into an explicit three-dimensional model so as to complete model construction and display.
According to the embodiment, through obtaining image point cloud data and camera pose information of a target object, a foreground and a background are separated to obtain foreground point cloud, a foreground minimum bounding box is determined according to the extracted foreground point cloud, foreground point data and background point data are partitioned based on the camera pose information, the foreground minimum bounding box and the image point cloud data, foreground reconstruction data and background reconstruction data are determined by using a preset reconstruction network further based on the foreground point data and the background point data, and finally a target object model is generated and displayed according to the foreground reconstruction data and the background reconstruction data. The embodiment predicts the model data required by the model reconstruction by using the preset reconstruction network, improves the accuracy and the accuracy of the model reconstruction, and also calibrates the reconstruction range of the foreground object and the background by using the minimum bounding box of the foreground, reduces the manual labeling cost and improves the model reconstruction efficiency.
The present embodiment proposes a modeling system 200 for extracting foreground objects, and referring to fig. 5, fig. 5 is a schematic system architecture diagram of the modeling system 200 for extracting foreground objects provided in the present embodiment.
The modeling system 200 for extracting foreground objects includes a data acquisition module 210, an image separation module 220, a bounding box determination module 230, a data determination module 240, a data processing module 250, and a model display module 260.
The data acquisition module 210 is configured to acquire image point cloud data of a target object and camera pose information;
the image separation module 220 is configured to separate a foreground from a background in the image point cloud data to obtain a foreground object point cloud;
the bounding box determining module 230 is configured to determine a minimum bounding box of the foreground according to the extracted point cloud of the foreground object;
the data determining module 240 is configured to determine foreground point data and background point data based on camera pose information, a minimum foreground bounding box, and image point cloud data;
the data processing module 250 is configured to determine foreground reconstruction data and background reconstruction data by using a preset reconstruction network based on the foreground point data and the background point data;
the model display module 260 is configured to generate and display a target object model according to the foreground reconstruction data and the background reconstruction data.
It will be apparent to those skilled in the art that, for convenience and brevity of description, only the above-described division of the functional units and modules is illustrated, and in practical application, the above-described functional distribution may be performed by different functional units and modules according to needs, i.e. the internal structure of the apparatus is divided into different functional units or modules to perform all or part of the above-described functions. The functional units and modules in the embodiment may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit, where the integrated units may be implemented in a form of hardware or a form of a software functional unit. In addition, the specific names of the functional units and modules are only for distinguishing from each other, and are not used for limiting the protection scope of the present application.
The present embodiment also provides an electronic device that in one typical configuration includes one or more processors (CPUs), an input/output interface, a network interface, and memory.
The memory may include volatile memory in a computer-readable medium, random Access Memory (RAM) and/or nonvolatile memory, etc., such as Read Only Memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.
Computer readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computing device. Computer-readable media, as defined herein, does not include transitory computer-readable media (transmission media), such as modulated data signals and carrier waves.
The present embodiment also provides a computer readable storage medium having stored thereon instructions for a program adapted to perform the steps of a modeling method of extracting a foreground object when executed by a processor.
It will be appreciated by those skilled in the art that embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In addition, the specific features described in the above embodiments may be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, various possible combinations of embodiments of the present application are not described in detail.
In addition, the functional modules in the embodiments of the present application may be integrated together to form a single part, or each module may exist alone, or two or more modules may be integrated to form a single part.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article or apparatus that comprises an element.
The foregoing is merely exemplary of the present application and is not intended to limit the present application. Various modifications and variations of the present application will be apparent to those skilled in the art. Any modification, equivalent replacement, improvement, etc. which come within the spirit and principles of the application are to be included in the scope of the claims of the present application.
Claims (10)
1. A modeling method of extracting a foreground object, the method comprising:
acquiring image point cloud data of a target object and camera pose information;
separating a foreground from a background in the image point cloud data to obtain a foreground object point cloud;
determining a minimum bounding box of the foreground according to the extracted foreground object point cloud;
determining foreground point data and background point data based on the camera pose information, the foreground minimum bounding box and the image point cloud data;
based on foreground point data and background point data, determining foreground reconstruction data and background reconstruction data by using a preset reconstruction network;
and generating and displaying a target object model according to the foreground reconstruction data and the background reconstruction data.
2. The modeling method of extracting a foreground object of claim 1, wherein the preset reconstruction network comprises an implicit surface reconstruction network and a color prediction network;
the generating foreground reconstruction data based on the foreground point data by using a preset reconstruction network includes:
inputting the foreground point data into the implicit surface reconstruction network, and determining the distance from each foreground point in the foreground object to a preset implicit surface;
inputting the foreground point data into the color prediction network to determine the color of each foreground point in the foreground object;
and obtaining reconstruction data of the foreground objects according to the distance from each foreground point to the preset implicit surface and the color of each foreground point in the foreground objects.
3. The modeling method for extracting a foreground object according to claim 2, wherein the determining the background reconstruction data based on the background point data using a preset reconstruction network includes:
inputting the background points into the implicit surface reconstruction network, and determining the distance between each background point in the background and a preset implicit surface;
inputting the background points into the color prediction network to determine the color of each background point in the background;
and generating the background reconstruction data according to the distance from each background point in the background to the preset implicit surface and the color of each background point.
4. The modeling method for extracting a foreground object according to claim 1, wherein the determining foreground point data and background point data based on the camera pose information, the foreground minimum bounding box, and the image point cloud data includes:
generating light rays passing through a foreground minimum bounding box based on the image point cloud data and the camera pose information;
sampling the light passing through the minimum bounding box of the foreground to obtain light sampling data;
calculating to obtain the position information of each light sampling point based on the light sampling data;
judging whether the position information of the light sampling point meets preset foreground position information or not;
if yes, judging the light sampling point as a foreground point, and obtaining foreground point data;
if not, judging the light sampling point as a background point, and obtaining the background point data.
5. A modeling method of extracting a foreground object as defined in claim 3, further comprising:
rendering the target object model based on the colors of the foreground points in the foreground object and the colors of the background points in the background, and calculating color data of each pixel point after rendering by using the formula (1):
in the formula (1), the components are as follows,representing pixel color; n represents the number of sampling points on ray and N 1 Indicating the number of foreground points on the light, N 2 N represents the number of background points on the light 1 +N 2 ＝N；w(p i ) Color expression weight of the ith foreground point on the light is represented; w (p) j ) Color expression weights representing the jth background point on the light; c (p) i ) Representing foreground point p i Is a predicted color of (2); c (p) j ) Representing background point p j Is used for the prediction color of (a).
6. The modeling method for extracting a foreground object as defined in claim 5, wherein generating and displaying a target object model from the foreground object reconstruction data and the background reconstruction data comprises:
reconstructing to obtain an initial object model based on the distance from each foreground point to the preset implicit surface and the distance from each background point to the preset implicit surface;
calculating a color loss value according to the color data of each pixel point after rendering;
judging whether the color loss value is smaller than a preset color loss value or not;
if the color loss value is smaller than the preset color loss value, the current initial object model is saved as a target object model;
and displaying the target object model.
7. The modeling method for extracting a foreground object as defined in claim 6, wherein calculating a color loss value from the color data of each pixel after rendering comprises:
calculating a color loss value according to the color data of each pixel point after rendering by using the formula (2):
l sum ＝l color +β·l reg (2)
in the formula (2), l sum Is a color loss value; l (L) color Is a color loss function, expressed as Representing the rendered pixel color; c (ray) is the true color of the pixel; l (L) reg Constraint loss function for regular term, expressed as +.>M represents the batch size of light rays which are sent to training in the color prediction network; n represents the number of sampling points on each light ray; />Representing the predicted point p of the implicit surface reconstruction network i Distance S to preset implicit surface to point p i Is beta is superParameters.
8. A modeling system for extracting a foreground object, the system comprising:
the data acquisition module is used for acquiring image point cloud data of the target object and camera pose information;
the image separation module is used for separating the foreground from the background in the image point cloud data to obtain foreground object point clouds;
the bounding box determining module is used for determining a minimum foreground bounding box according to the extracted foreground object point cloud;
the data determining module is used for determining foreground point data and background point data based on the camera pose information, the minimum foreground bounding box and the image point cloud data;
the data processing module is used for determining foreground reconstruction data and background reconstruction data by utilizing a preset reconstruction network based on the foreground point data and the background point data;
and the model display module is used for generating and displaying a target object model according to the foreground reconstruction data and the background reconstruction data.
9. An electronic device, comprising: a processor and a memory storing machine-readable instructions executable by the processor, which when executed by the processor perform the modeling method of extracting a foreground object of any of claims 1-7.
10. A computer-readable storage medium having instructions stored thereon for causing a machine to perform the modeling method of extracting a foreground object of any of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310628354.XA CN116645471A (en) | 2023-05-30 | 2023-05-30 | Modeling method, system, equipment and storage medium for extracting foreground object |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310628354.XA CN116645471A (en) | 2023-05-30 | 2023-05-30 | Modeling method, system, equipment and storage medium for extracting foreground object |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116645471A true CN116645471A (en) | 2023-08-25 |
Family
ID=87643076
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310628354.XA Pending CN116645471A (en) | 2023-05-30 | 2023-05-30 | Modeling method, system, equipment and storage medium for extracting foreground object |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116645471A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116993924A (en) * | 2023-09-25 | 2023-11-03 | 北京渲光科技有限公司 | Three-dimensional scene modeling method and device, storage medium and computer equipment |
-
2023
- 2023-05-30 CN CN202310628354.XA patent/CN116645471A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116993924A (en) * | 2023-09-25 | 2023-11-03 | 北京渲光科技有限公司 | Three-dimensional scene modeling method and device, storage medium and computer equipment |
| CN116993924B (en) * | 2023-09-25 | 2023-12-15 | 北京渲光科技有限公司 | Three-dimensional scene modeling method and device, storage medium and computer equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110070139B (en) | Small sample in-loop learning system and method facing automatic driving environment perception | |
| US11954813B2 (en) | Three-dimensional scene constructing method, apparatus and system, and storage medium | |
| CN108198145B (en) | Method and device for point cloud data restoration | |
| CN111524135B (en) | Method and system for detecting defects of tiny hardware fittings of power transmission line based on image enhancement | |
| CN113468967B (en) | Attention mechanism-based lane line detection method, attention mechanism-based lane line detection device, attention mechanism-based lane line detection equipment and attention mechanism-based lane line detection medium | |
| CN111161349A (en) | Object attitude estimation method, device and equipment | |
| CN112861729B (en) | Real-time depth completion method based on pseudo-depth map guidance | |
| CN108230235B (en) | Disparity map generation system, method and storage medium | |
| CN113052109A (en) | 3D target detection system and 3D target detection method thereof | |
| US11756308B2 (en) | Neural network device and method using a neural network for sensor fusion | |
| CN116645471A (en) | Modeling method, system, equipment and storage medium for extracting foreground object | |
| CN110335330A (en) | Image simulation generation method and its system, deep learning algorithm training method and electronic equipment | |
| CN116433903A (en) | Instance segmentation model construction method, system, electronic equipment and storage medium | |
| CN115170741A (en) | Rapid radiation field reconstruction method under sparse visual angle input | |
| CN117274515A (en) | Visual SLAM method and system based on ORB and NeRF mapping | |
| CN115082450A (en) | Pavement crack detection method and system based on deep learning network | |
| CN113920538A (en) | Object detection method, device, equipment, storage medium and computer program product | |
| CN112396657A (en) | Neural network-based depth pose estimation method and device and terminal equipment | |
| CN115457195A (en) | Two-dimensional and three-dimensional conversion method, system, equipment and medium for distribution network engineering drawings | |
| CN116027951A (en) | Visualization method and device for vehicle acquisition data and storage medium | |
| CN114998507A (en) | Luminosity three-dimensional reconstruction method based on self-supervision learning | |
| CN112634174B (en) | Image representation learning method and system | |
| CN117115786B (en) | Depth estimation model training method for joint segmentation tracking and application method | |
| CN114462486A (en) | Training method of image processing model, image processing method and related device | |
| US20220327811A1 (en) | System and method for composite training in machine learning architectures |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |