CN113744410A

CN113744410A - Grid generation method and device, electronic equipment and computer readable storage medium

Info

Publication number: CN113744410A
Application number: CN202111068931.1A
Authority: CN
Inventors: 姜翰青; 章国锋; 鲍虎军
Original assignee: Zhejiang Shangtang Technology Development Co Ltd
Current assignee: Zhejiang Shangtang Technology Development Co Ltd; Zhejiang Sensetime Technology Development Co Ltd
Priority date: 2021-09-13
Filing date: 2021-09-13
Publication date: 2021-12-03
Also published as: TW202312100A; WO2023035509A1

Abstract

The present disclosure provides a mesh generation method, apparatus, electronic device, and computer-readable storage medium, which determine a corresponding voxel of a pixel in a preset voxel space based on an image coordinate and a depth value of the pixel in a current depth image; determining structured data increment information corresponding to neighborhood voxels based on the structured data corresponding to the neighborhood voxels and the pre-depth image of the voxels; and finally, generating a target grid corresponding to the current depth image based on the structured data increment information corresponding to the neighborhood voxels. According to the method, the grid is extracted based on the incremental information of the structured data corresponding to the voxel, the target grid is generated, the grid is not extracted from all the structured data, the calculation amount of grid extraction can be reduced, the time consumption of grid extraction is reduced, and the grid extraction efficiency is effectively improved.

Description

Grid generation method and device, electronic equipment and computer readable storage medium

Technology neighborhood

The present disclosure relates to the field of image processing technologies, and in particular, to a grid generation method, an apparatus, an electronic device, and a computer-readable storage medium.

Background

Augmented reality is an important topic in the field of three-dimensional perception, and the use of dense grids in augmented reality applications can more completely describe three-dimensional scene information. Based on the known dense grid, some common 3D scene interactions or special effects can be achieved, such as occlusion, collision, shadow mapping of objects, etc. According to a traditional dense grid generation algorithm, depth maps of different visual angles are sequentially fused into structured data according to internal and external parameters of a camera, and corresponding dense grids are extracted.

In the traditional dense grid generation algorithm, after each depth image is fused into the structured data, grids are extracted from all the structured data, and the computation amount is large, the time consumption is long, and the efficiency is poor.

Disclosure of Invention

The embodiment of the disclosure at least provides a grid generation method and device.

In a first aspect, an embodiment of the present disclosure provides a mesh generation method, including:

acquiring a current depth image and structured data corresponding to a previous depth image used for grid generation before the current depth image;

determining a corresponding voxel of a pixel in a preset voxel space based on the image coordinate and the depth value of the pixel in the current depth image;

determining structured data increment information corresponding to the neighborhood voxels of the voxels based on the neighborhood voxels of the voxels and the structured data corresponding to the pre-depth image;

and generating a target grid corresponding to the current depth image based on the structured data increment information corresponding to the neighborhood voxels of the voxels.

In this aspect, the grid is extracted based on the incremental information of the structured data corresponding to the voxel to generate the target grid, instead of extracting the grid for all structured data, so that the calculation amount of grid extraction can be reduced, the time consumption of grid extraction can be reduced, and the grid extraction efficiency can be effectively improved.

In one possible embodiment, the determining incremental structured data information corresponding to a neighborhood voxel of the voxel based on the neighborhood voxel of the voxel and the structured data corresponding to the pre-depth image includes:

determining bias information corresponding to the neighborhood voxels based on voxel coordinates of the neighborhood voxels in the preset voxel space;

determining key values of the neighborhood voxels based on the bias information of the neighborhood voxels;

and determining structured data increment information corresponding to the neighborhood voxels of the voxels based on the key values of the neighborhood voxels and the structured data corresponding to the front depth image.

According to the embodiment, the generation of the key value of the voxel by using the bias information can avoid hash collision when the voxel outside the voxel space is inquired. In addition, the more accurate incremental information of the structured data can be determined by comparing the key values of the voxels with the structured data corresponding to the front depth image.

In a possible implementation, the structured data corresponding to the front depth image includes key values of voxels corresponding to the front depth image;

determining structured data increment information corresponding to the neighborhood voxels of the voxels based on the key values of the neighborhood voxels and the structured data corresponding to the front depth image, including:

projecting the neighborhood voxel onto the current depth image in response to a key value of the neighborhood voxel being included in the structured data;

in response to the fact that the neighborhood voxels are projected onto the current depth image and the voxel depth information obtained by projecting the neighborhood voxels onto the current depth image is within a preset threshold range, determining structured data increment information corresponding to the neighborhood voxels based on the voxel depth information corresponding to the neighborhood voxels and the pixel depth information of the pixels corresponding to the neighborhood voxels in the current depth image; alternatively, the first and second electrodes may be,

and in response to the condition that the key value of the neighborhood voxel is not included in the structured data, determining structured data increment information corresponding to the neighborhood voxel based on the voxel depth information corresponding to the neighborhood voxel.

According to the embodiment, structured data increment information can be determined more accurately by using the projection result of the neighborhood voxels and the voxel depth information and/or pixel depth information corresponding to the voxels.

In one possible embodiment, the structured data increment information includes state information of the corresponding voxel and a truncated sign distance function increment value of the corresponding voxel;

determining structured data increment information corresponding to the neighborhood voxels based on the voxel depth information corresponding to the neighborhood voxels and the pixel depth information of the pixels corresponding to the neighborhood voxels in the current depth image, including:

taking the difference value of the voxel depth information corresponding to the neighborhood voxels and the pixel depth information corresponding to the neighborhood voxels as the increment value of the truncated symbol distance function corresponding to the neighborhood voxels;

and setting the state information of the neighborhood voxels to be an update state.

determining structured data increment information corresponding to the neighborhood voxels based on the voxel depth information corresponding to the neighborhood voxels, including:

taking the voxel depth information as a truncated sign distance function increment value corresponding to the neighborhood voxel;

and setting the state information of the neighborhood voxels to be an adding state.

According to the embodiment, the structured data increment information can be determined more accurately by using the voxel depth information and/or the pixel depth information corresponding to the voxel.

In a possible embodiment, the determining incremental structured data information corresponding to a neighborhood voxel of the voxel based on key values of the neighborhood voxel and structured data corresponding to the front depth image further includes:

and setting the state information of the neighborhood voxels to a deleted state in response to the condition that the neighborhood voxels are not projected onto the current depth image or the voxel depth information obtained by projecting the neighborhood voxels onto the current depth image is not in the preset threshold range.

According to the embodiment, the voxels which cannot be projected on the current depth image or the voxels whose depth information is not within the preset threshold range are set to the deleted state, so that the voxels with low accuracy can be deleted, and the quality of the generated grid can be improved.

In one possible embodiment, the bias information includes first and second bias information for corresponding voxels;

determining bias information corresponding to the neighborhood voxels based on the voxel coordinates of the neighborhood voxels in the preset voxel space, including:

screening a coordinate item with the maximum absolute value from the voxel coordinates;

determining the first bias information based on the coordinate item with the maximum absolute value and the number of unit voxels included in the preset voxel space;

determining the second bias information based on the first bias information and the number of unit voxels included in the preset voxel space.

In this embodiment, the bias information can be generated more accurately based on the voxel coordinates.

In one possible embodiment, the generating a target mesh corresponding to a current depth image based on the structured data increment information corresponding to the neighborhood voxel of the voxel includes:

determining target structured data for the voxel based on the structured data delta information for the voxel and the structured data;

and generating a target grid corresponding to the current depth image based on the target structured data of the voxel.

In the embodiment, the incremental information of the structured data is added to the original structured data to generate the target structured data, and the target structured data can be used for generating the accurate target grid.

generating a target grid corresponding to the current depth image based on the target structured data of the voxels, including:

responding to the condition that the state information of the voxel of the neighborhood voxel of the voxel is in a preset state, and generating an incremental grid corresponding to the voxel based on the target structured data of the voxel when the condition that the number of times that the state information of the voxel of the neighborhood voxel is in the preset state is determined to be greater than the preset number of times;

and generating a target grid corresponding to the current depth image based on the increment grid corresponding to the voxel and the grid corresponding to the front depth image.

According to the embodiment, when the updating times of the state information in the preset state are greater than the preset times, the grids are extracted from the corresponding target structured data, so that repeated grid extraction is avoided, the calculation amount of grid extraction is reduced, the calculation resources are saved, the grid extraction efficiency is improved, and the grid extraction noise is reduced.

In a possible implementation, the above mesh generation method further includes:

acquiring an optimized pose parameter corresponding to shooting equipment for shooting the depth image and a pose parameter when the shooting equipment shoots each depth image;

for each depth image, determining a pose change value based on the pose parameter corresponding to the depth image and the optimized pose parameter;

in response to the situation that the pose change value is larger than a preset pose change threshold value, removing the structured data increment information of the voxel corresponding to the depth image from the target structured data, and determining the final structured data increment information of the voxel corresponding to the depth image based on an optimized pose parameter;

and determining the structured data corresponding to the depth image based on the final structured data increment information of the voxel corresponding to the depth image.

According to the embodiment, when the pose change of the shooting equipment is large, the structured data increment information generated according to the unoptimized pose parameters is removed from the target structured data, new and more accurate structured data increment information is generated based on the optimized pose parameter information, more accurate structured data can be generated based on the structured data increment information, and the grid layering dislocation phenomenon can be reduced.

In one possible embodiment, the structured data increment information includes state information of the corresponding voxel, a truncated sign distance function increment value of the corresponding voxel;

the mesh generation method further includes:

and in response to the condition that the truncated sign distance function value of the voxel corresponding to the depth image is smaller than zero after the structured data increment information of the voxel corresponding to the depth image is removed from the target structured data, setting the state information of the voxel corresponding to the depth image to be in a deleted state.

According to the embodiment, the voxels with the truncated symbol distance function value smaller than zero are set to be in the deleted state, the voxels with low accuracy can be deleted, the quality of the generated grid is improved, and the phenomenon of grid layering and dislocation is reduced.

In a possible embodiment, before the generating a target mesh corresponding to a current depth image based on the structured data increment information corresponding to the neighborhood voxel of the voxel, the method further includes:

screening target voxels with empty structured data increment information from the voxels corresponding to the current depth image;

projecting the target voxel onto the current depth image;

and in response to the fact that the target voxel is projected onto the current depth image, and in the case that voxel depth information obtained by projecting the target voxel onto the current depth image is within a preset threshold range, determining structured data increment information corresponding to the target voxel based on the voxel depth information corresponding to the target voxel and pixel depth information of a pixel corresponding to the target voxel in the current depth image.

According to the embodiment, the moving object is determined by using the non-updated structured data and the depth image, and the corresponding incremental information of the structured data is updated, so that the accuracy of grid extraction can be improved.

determining structured data increment information corresponding to the target voxel based on the voxel depth information corresponding to the target voxel and the pixel depth information of the pixel corresponding to the target voxel in the current depth image, including:

in response to the condition that the difference value obtained by subtracting the pixel depth information corresponding to the target voxel from the voxel depth information corresponding to the target voxel is less than zero, taking the difference value obtained by subtracting the pixel depth information from the voxel depth information as the increment value of the truncated sign distance function corresponding to the target voxel;

and setting the state information of the target voxel to be an update state.

According to the embodiment, the structured data increment information of the voxel with the difference value smaller than zero between the voxel depth information and the corresponding pixel depth information is updated, the structured data increment information of the moving object is substantially updated, and the grid extraction accuracy can be improved.

In a possible implementation, before the determining, based on the image coordinates of the pixel in the current depth image, the corresponding voxel of the pixel in a preset voxel space, the method further includes:

calculating a pixel depth difference value between the pixel depth of each pixel in the current depth image and the pixel depth of a neighborhood pixel of the pixel;

and in response to the condition that the pixel depth difference value corresponding to the pixel is not within the preset depth difference value range, setting the pixel as invalid.

According to the embodiment, the depth consistency of the neighborhood pixels of the pixels in the current depth image is checked, the noise in the current depth image is eliminated, and the grid extraction accuracy is improved.

In a second aspect, an embodiment of the present disclosure further provides a mesh generation apparatus, including:

the data acquisition module is used for acquiring a current depth image and structured data corresponding to a front depth image used for grid generation before the current depth image;

a voxel determining module, configured to determine, based on image coordinates and depth values of pixels in the current depth image, a voxel corresponding to the pixel in a preset voxel space;

an incremental information determining module, configured to determine structured data incremental information corresponding to a neighborhood voxel of the voxel based on the neighborhood voxel of the voxel and structured data corresponding to the pre-depth image;

and the grid generating module is used for generating a target grid corresponding to the current depth image based on the structured data increment information corresponding to the neighborhood voxels of the voxels.

In a third aspect, an embodiment of the present disclosure further provides an electronic device, including: a processor, a memory and a bus, the memory storing machine-readable instructions executable by the processor, the processor and the memory communicating via the bus when the electronic device is running, the machine-readable instructions when executed by the processor performing the steps of the first aspect described above, or any possible implementation of the first aspect.

In a fourth aspect, this disclosed embodiment also provides a computer-readable storage medium, on which a computer program is stored, where the computer program is executed by a processor to perform the steps in the first aspect or any one of the possible implementation manners of the first aspect.

For the description of the effects of the above mesh generation apparatus, electronic device, and computer-readable storage medium, reference is made to the description of the above mesh generation method, which is not repeated here.

In order to make the aforementioned objects, features and advantages of the present disclosure more comprehensible, preferred embodiments accompanied with figures are described in detail below.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings required for use in the embodiments will be briefly described below, and the drawings herein incorporated in and forming a part of the specification illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the technical solutions of the present disclosure. It is to be understood that the following drawings depict only certain embodiments of the disclosure and are therefore not to be considered limiting of its scope, for the person skilled in the art will readily appreciate that other related drawings may be derived therefrom without the benefit of the inventive faculty.

Fig. 1 shows a flowchart of a mesh generation method provided by an embodiment of the present disclosure;

FIG. 2 illustrates a schematic diagram of neighborhood voxels in an embodiment of the present disclosure;

FIG. 3 illustrates a schematic diagram of a preset voxel space in an embodiment of the present disclosure;

FIG. 4 shows a schematic diagram of an extracted mesh in an embodiment of the present disclosure;

fig. 5 is a schematic diagram of a mesh generation apparatus provided in an embodiment of the present disclosure;

fig. 6 shows a schematic diagram of an electronic device provided by an embodiment of the present disclosure.

Detailed Description

In order to make the objects, technical solutions and advantages of the embodiments of the present disclosure more clear, the technical solutions of the embodiments of the present disclosure will be described clearly and completely with reference to the drawings in the embodiments of the present disclosure, and it is obvious that the described embodiments are only a part of the embodiments of the present disclosure, not all of the embodiments. The components of the embodiments of the present disclosure, generally described and illustrated in the figures herein, can be arranged and designed in a wide variety of different configurations. Thus, the following detailed description of the embodiments of the present disclosure, presented in the figures, is not intended to limit the scope of the claimed disclosure, but is merely representative of selected embodiments of the disclosure. All other embodiments, which can be derived by the skilled person without inventive step based on the embodiments of the present disclosure, are within the scope of the present disclosure.

It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, it need not be further defined and explained in subsequent figures.

The term "and/or" herein merely describes an associative relationship, meaning that three relationships may exist, e.g., a and/or B, may mean: a exists alone, A and B exist simultaneously, and B exists alone. In addition, the term "at least one" herein means any one of a plurality or any combination of at least two of a plurality, for example, including at least one of A, B, C, and may mean including any one or more elements selected from the group consisting of A, B and C.

In the prior art, when grid extraction is carried out, after each depth image is fused into structured data, grids are extracted from all the structured data, and the grid extraction method is large in calculation amount, long in time consumption and poor in efficiency. In view of the above drawbacks, embodiments of the present disclosure provide a grid generation method and apparatus, an electronic device, and a computer-readable storage medium, where a grid is extracted based on structured data increment information corresponding to voxels, and a target grid is generated, instead of extracting the grid for all structured data, so that the amount of computation for grid extraction can be reduced, the time consumed for grid extraction is reduced, and the grid extraction efficiency is effectively improved.

The mesh generation method, the mesh generation apparatus, the electronic device, and the storage medium according to the present disclosure are described below with specific embodiments.

As shown in fig. 1, the embodiment of the present disclosure discloses a mesh generation method, which may be applied to a device such as a server that needs to generate a mesh, and is used to generate a target mesh based on a depth image. Specifically, the mesh generation method may include the steps of:

s110, obtaining a current depth image and structured data corresponding to a previous depth image used for grid generation before the current depth image.

The depth image may be captured by a depth camera and transmitted to a device that performs the mesh generation method. The front depth image may include one or more depth images, and is an image in which structured data is generated before the current depth image and mesh extraction is performed based on the extracted structured data.

The present embodiment is described with respect to how the mesh is extracted from the current depth image, and the method for extracting the mesh from the previous depth image may be the same as the method for extracting the mesh from the current depth image.

The "mesh" in the mesh extraction and generation target mesh in this embodiment refers to a dense mesh, and the extracted or generated dense mesh may be used to generate an augmented reality AR image or an AR special effect.

When the grid is generated or extracted, firstly, the structured data corresponding to the depth image is fused into the structured data obtained after the previous depth image is fused, and then grid extraction operation is carried out on the fused structured data to obtain the grid. Therefore, when the mesh is generated in the front depth image, the corresponding structured data and the fused structured data are generated firstly.

And S120, determining a corresponding voxel of the pixel in a preset voxel space based on the image coordinate of the pixel in the current depth image.

Here, as shown in fig. 3, the preset voxel space may be flexibly set according to the actual application scenario, for example, the preset voxel space may be set to be N³N is 500 in the voxel space formed by the unit voxels. The voxel coordinate system of the voxel space has the range of [ -N/2, N/2]The unit voxel corresponds to an actual size of ζ, which is 0.06 meters. The actual size of the preset voxel space is N ζ.

In fig. 3, a voxel located in the predetermined voxel space is a voxel 301, and a voxel located outside the predetermined voxel space is a voxel 302.

When a voxel of a certain pixel is specifically determined, the following steps can be utilized to implement:

when determining the voxel coordinate of a pixel, the depth value of the pixel needs to be combinedPhysically, acquiring pixel coordinates p (i, j) of a pixel with the depth d (i, j), and projecting the pixel into a world coordinate system by using internal and external parameters of a shooting device for shooting a current depth image to obtain three-dimensional coordinates

Then the three-dimensional coordinates are measured

Projected to voxel coordinates

The voxel coordinates (x, y, z) are obtained.

In order to save memory, only the transformed voxels need to be managed, and the whole preset voxel space does not need to be maintained.

S130, determining structured data increment information corresponding to the neighborhood voxels of the voxels based on the neighborhood voxels of the voxels and the structured data corresponding to the front depth image.

As shown in fig. 2, the neighborhood voxels of a voxel may include voxels 201 located at 8 vertices of a cube corresponding to the unit voxel where the voxel is located.

In a particular embodiment, the structured data delta information may include state information of the corresponding voxel and a truncated sign distance function tsdf delta value of the corresponding voxel. The state information may include: an add state, an update state, and a delete state. The truncated sign distance function increment value may be a difference between voxel depth information obtained by projecting a certain voxel onto the current depth image and pixel depth information of a pixel corresponding to the voxel.

When the structured data increment information is determined, firstly, key values of neighborhood voxels are calculated, then whether the calculated key values exist in structured data obtained by fusing front depth images or not is judged, if not, the structured data of the neighborhood voxels are used as the structured data increment information, and if so, the structured increment information is determined based on the voxel coordinates favorable for the voxels and the corresponding pixel coordinates.

And S140, generating a target grid corresponding to the current depth image based on the structured data increment information corresponding to the neighborhood voxels of the voxels.

Specifically, the structured data increment information of a certain voxel may be fused with the structured data corresponding to the voxel in the structured data obtained by fusing the previous depth image to obtain the target structured data of the voxel, and then the target mesh may be generated based on the mesh obtained by mesh extraction of the target structured data.

In a specific implementation, the incremental structured data information may be added to the structured data corresponding to the corresponding voxel by means of weighted summation, so as to obtain the target structured data.

And adding the incremental information of the structured data to the original structured data to generate target structured data, and generating a more accurate target grid by using the target structured data.

In some embodiments, the determining incremental information of the structured data corresponding to the neighborhood voxel of the voxel based on the neighborhood voxel of the voxel and the structured data corresponding to the pre-depth image may specifically be implemented by using the following steps:

and step 11, determining the bias information corresponding to the neighborhood voxels based on the voxel coordinates of the neighborhood voxels in the preset voxel space.

In a specific implementation, the bias information may include first bias information and second bias information of neighboring voxels. The method can be realized by the following steps:

and a substep 111 of screening a coordinate item with the maximum absolute value from the voxel coordinates of the neighborhood voxels.

The coordinate term with the largest absolute value can be specifically screened by using the following formula:

indicating that the coordinate item with the largest absolute value is selected.

The substep 112 determines the first bias information based on the coordinate item having the largest absolute value and the number of unit voxels included in the preset voxel space.

In that

Then, the first bias information may be calculated using the following formula:

in the formula, s_lIndicating the first bias information.

Then, the first bias information may be calculated using the following formula:

substep 113 determines the second bias information based on the first bias information and the number of unit voxels included in the preset voxel space.

Here, the second offset information may be specifically calculated by using the following formula:

in the formula, s_GIndicating the second bias information and k the index value of the voxel.

And step 12, determining the key value of the neighborhood voxel based on the bias information of the neighborhood voxel.

In specific implementation, the following formula can be used to calculate the key value of the neighborhood voxel:

wherein the content of the first and second substances,

in the formula (I), the compound is shown in the specification,

representing the key values of the neighborhood voxels.

The key value calculation mode can avoid Hash collision when the voxels outside the preset voxel space are inquired, the inquiry range is limited in the preset voxel space, and the voxels obtained by depth image conversion can be obtained at any time, which means that the grid generation is not limited by the set generation range any more.

And step 13, determining structured data increment information corresponding to the neighborhood voxels of the voxels based on the key values of the neighborhood voxels and the structured data corresponding to the front depth image.

Here, the structured data includes key values of voxels to which the front depth image corresponds. After obtaining the key values of the neighborhood voxels corresponding to the current depth image, judging whether the key values of the neighborhood voxels are included in the structured data, and projecting the neighborhood voxels onto the current depth image under the condition that the key values of the neighborhood voxels are included in the structured data. And then, if the neighborhood voxel is projected onto the current depth image and the voxel depth information obtained by projecting the neighborhood voxel onto the current depth image is within a preset threshold range, determining the structured data increment information corresponding to the neighborhood voxel based on the voxel depth information corresponding to the neighborhood voxel and the pixel depth information of the pixel corresponding to the neighborhood voxel in the current depth image.

Taking the difference value between the voxel depth information corresponding to the neighborhood voxels and the pixel depth information corresponding to the neighborhood voxels as the increment value of the truncated symbolic distance function corresponding to the neighborhood voxels; and setting the state information of the neighborhood voxels to be an update state.

The preset threshold range can be flexibly set according to specific application scenarios, and can be set to [0.4m,2.5m ], for example.

And if the key value of the neighborhood voxel is not included in the structured data, determining the structured data increment information corresponding to the neighborhood voxel based on the voxel depth information corresponding to the neighborhood voxel.

Specifically, the voxel depth information is used as a truncated sign distance function increment value corresponding to the neighborhood voxel; and setting the state information of the neighborhood voxels to be an adding state.

The structured data increment information can be accurately determined by utilizing the voxel depth information and/or the pixel depth information corresponding to the voxels.

Voxels which cannot be projected onto the current depth image or voxels whose depth information is not within the preset threshold range are set to a deleted state, so that voxels with low accuracy can be deleted, which is beneficial to improving the quality of the generated grid.

In the embodiment, the structured data increment information can be more accurately determined by using the projection result of the neighborhood voxel and the voxel depth information and/or pixel depth information corresponding to the voxel.

Generating the key value of the voxel by using the bias information can avoid hash collision when the voxel outside the voxel space is inquired. In addition, the more accurate incremental information of the structured data can be determined by comparing the key values of the voxels with the structured data corresponding to the front depth image.

After the above steps are completed, a counter may be used to record the update condition of the state information of each neighborhood voxel, specifically, if the state information of a certain neighborhood voxel is in an added state, the counter of the neighborhood voxel is set to 1, if the state information of a certain neighborhood voxel is in an updated state, the counter of the neighborhood voxel is added with 1 on the basis of the original value, and if the state information of a certain neighborhood voxel is in a deleted state, the counter of the neighborhood voxel is set to 0.

Determining whether to perform grid extraction on the neighborhood voxel based on the value of the neighborhood voxel recorded by the counter, which may specifically be implemented by using the following steps:

step 21, in response to that the state information of the voxel of the neighborhood voxel of the voxel is in a preset state, for example, an added state or an updated state, and when it is determined that the number of times that the state information of the voxel of the neighborhood voxel is in the preset state is greater than a preset number of times, that is, when the value in the counter corresponding to the neighborhood voxel is greater than the preset number of times, generating an incremental grid corresponding to the voxel based on the target structured data of the voxel.

The preset number of times can be flexibly set according to a specific application scene, and for example, can be set to 3.

When extracting the grid from the target structured data, the method can be specifically realized by using a Marching Cubes algorithm.

After the grid is extracted for a certain voxel, the state information of the voxel is set to a common state.

And step 22, generating a target grid corresponding to the current depth image based on the incremental grid corresponding to the voxel and the grid corresponding to the front depth image.

Specifically, the incremental mesh generated by each voxel is added to the mesh generated by the front depth image to obtain the target mesh.

As shown in fig. 4, an incremental grid 401 is added to an existing grid 402, the grid having an increasingly larger area.

When the updating times of the state information in the preset state are greater than the preset times, the grid is extracted from the corresponding target structured data, so that repeated grid extraction is avoided, the calculation amount of grid extraction is reduced, the calculation resources are saved, the grid extraction efficiency is improved, and the grid extraction noise is reduced.

And if the state information of one voxel exists in the neighborhood voxels of the voxel, a grid is not extracted from the voxel.

The pose of the shooting device for shooting the depth image may change, and if the pose of the shooting device is optimized and has a large change with the pose used for grid extraction, the structured data generated before needs to be deleted or updated, which can be specifically realized by the following steps:

and 31, acquiring an optimized pose parameter corresponding to shooting equipment for shooting the depth image and a pose parameter when the shooting equipment shoots each depth image.

In particular implementations, the extrinsic parameters of a capture device, such as a camera, may be denoted as T_i ^tThe external parameters include six parameters, the first three parameters are translation parameters of the camera, the last three parameters are rotation parameters of the camera, the i represents the frame number of the depth image, and the t represents the time sequence number of the depth image.

And step 32, aiming at each depth image, determining a pose change value based on the pose parameter corresponding to the depth image and the optimized pose parameter.

After receiving the optimized pose parameters, calculating pose change values for the pose parameters corresponding to each frame of depth image, specifically calculating the pose change values by using the following formula:

ΔT_i ^t′＝||T_i ^t′-T_i ^t||₂

in the formula, T_i ^t′Representing optimized pose parameters, | × | | luminance₂Representing the L2 norm.

And step 33, in response to the situation that the pose change value is larger than a preset pose change threshold value, removing the structured data increment information of the voxel corresponding to the depth image from the target structured data, and determining the final structured data increment information of the voxel corresponding to the depth image based on the optimized pose parameter.

The preset pose change threshold value can be flexibly set according to a specific application scene, and can be set to 0.01, for example.

When structured data increment information is eliminated, the position and pose parameters corresponding to the depth image can be used for determining the voxel coordinates of each voxel in the depth image, then the structured data increment information corresponding to each voxel is calculated by the same method as the embodiment, and the calculated structured data increment information is deleted from the target structured data. The structured data increment information corresponding to the stored depth image can be directly used for deleting the structured data increment information from the target structured data.

After the structured data increment information of the voxel corresponding to the depth image is removed from the target structured data, if the truncation sign distance function value of the voxel corresponding to the depth image is smaller than zero, the state information of the voxel corresponding to the depth image is set to be a deletion state.

The voxels with the truncated symbol distance function value less than zero are set to be in a deleted state, the voxels with low accuracy can be deleted, the quality of the generated grid is improved, and the phenomenon of grid layering and dislocation is reduced.

Then, the optimized pose parameters are used to determine the voxel coordinates of each voxel in the depth image, and the structured data increment information corresponding to each voxel is calculated by the same method as the above embodiment.

And step 34, determining the structured data corresponding to the depth image based on the final structured data increment information of the voxel corresponding to the depth image.

Here, the incremental structured data information may be added to the structured data corresponding to the corresponding voxel by means of weighted summation.

The grid generation technology in the prior art does not perform special treatment on the dynamic objects in the scene, the dynamic objects in the scene cannot be updated in real time, and the generated grid is not accurate enough. The embodiment of the present disclosure may specifically implement processing of a dynamic object by using the following steps:

before generating a target grid corresponding to a current depth image based on the structured data increment information corresponding to the neighborhood voxels of the voxels, screening target voxels with empty structured data increment information from the voxels corresponding to the current depth image; the target voxel is then projected onto the current depth image. And when the target voxel is projected onto the current depth image and the voxel depth information obtained by projecting the target voxel onto the current depth image is within a preset threshold range, determining the structured data increment information corresponding to the target voxel based on the voxel depth information corresponding to the target voxel and the pixel depth information of the pixel corresponding to the target voxel in the current depth image.

When the target voxel is projected to the current depth image, the pose parameter of the shooting device can be specifically used for implementation.

The preset threshold range can be flexibly set according to a specific application scenario, and can be set to [0.4m,2.5m ], for example.

The determining the structured data increment information corresponding to the target voxel based on the voxel depth information corresponding to the target voxel and the pixel depth information of the pixel corresponding to the target voxel in the current depth image may specifically be:

in response to the condition that the difference value obtained by subtracting the pixel depth information corresponding to the target voxel from the voxel depth information corresponding to the target voxel is less than zero, taking the difference value obtained by subtracting the pixel depth information from the voxel depth information as the increment value of the truncated sign distance function corresponding to the target voxel; and setting the state information of the target voxel to be an update state.

The structured data increment information of the voxel with the difference value smaller than zero between the voxel depth information and the corresponding pixel depth information is updated, the structured data increment information of the moving object is substantially updated, and the grid extraction accuracy can be improved.

When processing a dynamic object, it should be noted that although the state information of a voxel is updated, the value in the counter corresponding to the voxel is not modified.

The grid generation technology in the prior art is mainly used for processing a high-precision depth image acquired by a depth sensor, and the generated grid noise is more for the depth image with strong noise and poor quality; the embodiment of the present disclosure may perform noise reduction processing before processing the depth image to improve the accuracy of the generated mesh, and specifically may be implemented by using the following steps:

step 41, before determining the corresponding voxel of the pixel in the preset voxel space based on the image coordinates of the pixel in the current depth image, for each pixel in the current depth image, calculating a pixel depth difference value between the pixel depth of the pixel and the pixel depth of the pixel in the neighborhood of the pixel.

The neighborhood pixels may include a range of pixels adjacent to the pixel, for example, a pixel may have coordinates p (i, j), and the neighborhood pixels of the pixel may be pixels having coordinates within { i + Δ i, j + Δ j }. Where Δ i ∈ (-R, R), Δ j ∈ (-R, R), the neighborhood radius may be set to 3 pixels.

And 42, in response to the condition that the depth difference value of the pixel corresponding to the pixel is not within the preset depth difference value range, setting the pixel as invalid.

The preset depth difference may be set to a fixed value according to a specific scene, or the preset depth difference may be determined according to the depth information of the pixel, for example, the preset depth difference is set to 15% of the depth value corresponding to the pixel.

And performing neighborhood pixel depth consistency check on pixels in the current depth image, and eliminating noise in the current depth image, thereby being beneficial to improving the accuracy of grid extraction.

In the grid generation method in the above embodiment, only when the cumulative update frequency of the state information in each depth image reaches a certain value, the grid is extracted from the corresponding target structured data and merged into the previous grid, so that repeated access to all structured data is avoided, and the generated grid noise is greatly reduced. The embodiment reduces the amount of structured data needing to be maintained, is beneficial to grid extraction, does not generate voxel access conflict, can be arbitrarily expanded to the outside of a preset voxel space, and does not need to set the grid generation range in advance. Further, the embodiment solves the problem of high memory occupation in the traditional method, and the grid generating method can be operated in real time even by a middle-low-end mobile phone. Furthermore, whether a dynamic object exists in the depth image is judged in the process of generating the structured data, and if the dynamic object exists, the corresponding incremental information of the structured data is updated and removed, so that the accuracy of generating the grid can be improved. Further, in the above embodiment, before processing the depth image, consistency check is performed to remove noise in the depth image, thereby reducing noise of the generated mesh. Furthermore, in the above embodiment, the frame with a large change in the optimized pose parameter is fused again, that is, the incremental information of the structured data fused with the old pose parameter of the frame is deleted, and the pose parameter after the frame is optimized and the optimized pose parameter are fused into the incremental information of the structured data again, which is helpful for reducing the grid layering phenomenon.

Corresponding to the above mesh generation method, the present disclosure also discloses a mesh generation apparatus, where each module in the apparatus can implement each step in the mesh generation method of each embodiment, and can obtain the same beneficial effect, and therefore, the description of the same part is not repeated here. Specifically, as shown in fig. 5, the mesh generation apparatus includes:

a data obtaining module 510, configured to obtain a current depth image and a structured number corresponding to a previous depth image that is used for mesh generation before the current depth image.

A voxel determining module 520, configured to determine a voxel corresponding to a pixel in a preset voxel space based on the image coordinate and the depth value of the pixel in the current depth image.

An incremental information determining module 530, configured to determine incremental information of the structured data corresponding to the neighborhood voxel of the voxel based on the neighborhood voxel of the voxel and the structured data corresponding to the pre-depth image.

And a grid generating module 540, configured to generate a target grid corresponding to the current depth image based on the structured data increment information corresponding to the neighborhood voxel of the voxel.

In some embodiments, the incremental information determination module 530, when determining the structured data incremental information corresponding to the neighborhood voxel of the voxel based on the neighborhood voxel of the voxel and the structured data corresponding to the pre-depth image, is configured to:

In some embodiments, the structured data comprises key values of voxels to which the front depth image corresponds;

the incremental information determining module 530, when determining the structured data incremental information corresponding to the neighborhood voxel of the voxel based on the key value of the neighborhood voxel and the structured data corresponding to the previous depth image, is configured to:

In some embodiments, the structured data delta information comprises state information of the corresponding voxel and a truncated symbolic distance function delta value of the corresponding voxel;

the incremental information determining module 530, when determining the structured data incremental information corresponding to the neighborhood voxel based on the voxel depth information corresponding to the neighborhood voxel and the pixel depth information of the pixel corresponding to the neighborhood voxel in the current depth image, is configured to:

setting the state information of the neighborhood voxels to be an updated state;

In some embodiments, the incremental information determining module 530 is configured to, when determining the structured data incremental information corresponding to the neighborhood voxel of the voxel based on the key values of the neighborhood voxel and the structured data corresponding to the previous depth image:

In some embodiments, the bias information comprises first and second bias information for the corresponding voxel;

the incremental information determining module 530, when determining the bias information corresponding to the neighborhood voxel based on the voxel coordinate of the neighborhood voxel in the preset voxel space, is configured to:

In some embodiments, the mesh generation module 540, when generating the target mesh corresponding to the current depth image based on the structured data increment information corresponding to the neighborhood voxels of the voxel, is configured to:

the mesh generation module 540, when generating the target mesh corresponding to the current depth image based on the target structured data of the voxel, is configured to:

In some embodiments, the incremental information determining module 530 is further configured to:

In some embodiments, the structured data delta information comprises state information of the corresponding voxel, a truncated symbolic distance function delta value of the corresponding voxel;

the incremental information determination module 530 is further configured to:

In some embodiments, the incremental information determining module 530 is further configured to, before the generating the target mesh corresponding to the current depth image based on the structured data incremental information corresponding to the neighborhood voxels of the voxels:

projecting the target voxel onto the current depth image;

the incremental information determining module 530, when determining the structured data incremental information corresponding to the target voxel based on the voxel depth information corresponding to the target voxel and the pixel depth information of the pixel corresponding to the target voxel in the current depth image, is configured to:

and setting the state information of the target voxel to be an update state.

In some embodiments, voxel determination module 510 is further configured to, prior to said determining a corresponding voxel of a pixel in a preset voxel space based on image coordinates of the pixel in the current depth image:

Corresponding to the above mesh generation method, an embodiment of the present disclosure further provides an electronic device 600, as shown in fig. 6, which is a schematic structural diagram of the electronic device 600 provided in the embodiment of the present disclosure, and includes:

a processor 61, a memory 62, and a bus 63; the memory 62 is used for storing execution instructions and includes a memory 621 and an external memory 622; the memory 621 is also referred to as an internal memory, and is used for temporarily storing the operation data in the processor 61 and the data exchanged with the external memory 622 such as a hard disk, the processor 61 exchanges data with the external memory 622 through the memory 621, and when the electronic device 600 operates, the processor 61 communicates with the memory 62 through the bus 63, so that the processor 61 executes the following instructions:

acquiring a current depth image and structured data corresponding to a previous depth image used for grid generation before the current depth image; determining a corresponding voxel of a pixel in a preset voxel space based on the image coordinate of the pixel in the current depth image; determining structured data increment information corresponding to the neighborhood voxels of the voxels based on the neighborhood voxels of the voxels and the structured data corresponding to the pre-depth image; and generating a target grid corresponding to the current depth image based on the structured data increment information corresponding to the neighborhood voxels of the voxels.

The embodiments of the present disclosure also provide a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the computer program performs the steps of the grid generation method in the above-mentioned method embodiments. The storage medium may be a volatile or non-volatile computer-readable storage medium.

The embodiments of the present disclosure also provide a computer program product, which includes a computer readable storage medium storing a program code, where the program code includes instructions for executing the steps of the grid generating method in the above method embodiments, and in particular, the computer program product may be implemented by hardware, software, or a combination thereof. In an alternative embodiment, the computer program product is embodied in a computer storage medium, and in another alternative embodiment, the computer program product is embodied in a Software product, such as a Software Development Kit (SDK), or the like.

The disclosure relates to the field of augmented reality, and aims to detect or identify relevant features, states and attributes of a target object by means of various visual correlation algorithms by acquiring image information of the target object in a real environment, so as to obtain an AR effect combining virtual and reality matched with specific applications. For example, the target object may relate to a face, a limb, a gesture, an action, etc. associated with a human body, or a marker, a marker associated with an object, or a sand table, a display area, a display item, etc. associated with a venue or a place. The vision-related algorithms may involve visual localization, SLAM, three-dimensional reconstruction, image registration, background segmentation, key point extraction and tracking of objects, pose or depth detection of objects, and the like. The specific application can not only relate to interactive scenes such as navigation, explanation, reconstruction, virtual effect superposition display and the like related to real scenes or articles, but also relate to special effect treatment related to people, such as interactive scenes such as makeup beautification, limb beautification, special effect display, virtual model display and the like.

The detection or identification processing of the relevant characteristics, states and attributes of the target object can be realized through the convolutional neural network. The convolutional neural network is a network model obtained by performing model training based on a deep learning framework.

It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the system and the apparatus described above may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again. In the several embodiments provided in the present disclosure, it should be understood that the disclosed system, apparatus, and method may be implemented in other ways. The above-described embodiments of the apparatus are merely illustrative, and for example, the division of the units is only one logical division, and there may be other divisions when actually implemented, and for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection of devices or units through some communication interfaces, and may be in an electrical, mechanical or other form.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, functional units in the embodiments of the present disclosure may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit.

The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a non-volatile computer-readable storage medium executable by a processor. Based on such understanding, the technical solution of the present disclosure may be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for causing an electronic device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present disclosure. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.

Finally, it should be noted that: the above-mentioned embodiments are merely specific embodiments of the present disclosure, which are used for illustrating the technical solutions of the present disclosure and not for limiting the same, and the scope of the present disclosure is not limited thereto, and although the present disclosure is described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: any person skilled in the art can modify or easily conceive of the technical solutions described in the foregoing embodiments or equivalent technical features thereof within the technical scope of the present disclosure; such modifications, changes or substitutions do not depart from the spirit and scope of the embodiments of the present disclosure, and should be construed as being included therein. Therefore, the protection scope of the present disclosure shall be subject to the protection scope of the claims.

Claims

1. A mesh generation method, comprising:

2. The method of generating a mesh according to claim 1, wherein the determining incremental structured data information corresponding to a neighborhood voxel of the voxel based on the neighborhood voxel of the voxel and the structured data corresponding to the pre-depth image comprises:

3. The mesh generation method of claim 2, wherein the structured data corresponding to the front depth image comprises key values of voxels corresponding to the front depth image;

4. The mesh generation method of claim 3, wherein the structured data delta information comprises state information of the corresponding voxel and a truncated symbolic distance function delta value of the corresponding voxel;

5. The mesh generation method of claim 3, wherein the structured data delta information comprises state information of the corresponding voxel and a truncated symbolic distance function delta value of the corresponding voxel;

6. The mesh generation method of claim 4, wherein determining the incremental structured data information corresponding to the neighborhood voxels of the voxels based on the key values of the neighborhood voxels and the structured data corresponding to the front depth image further comprises:

7. The mesh generation method according to any one of claims 2 to 6, wherein the bias information includes first bias information and second bias information of the corresponding voxel;

8. The grid generation method according to any one of claims 1 to 7, wherein the generating a target grid corresponding to a current depth image based on the structured data increment information corresponding to the neighborhood voxels of the voxels comprises:

9. The mesh generation method of claim 8, wherein the structured data delta information comprises state information of a corresponding voxel and a truncated symbolic distance function delta value of the corresponding voxel;

10. The mesh generation method of claim 8, further comprising:

11. The mesh generation method of claim 10, wherein the structured data delta information comprises state information of the corresponding voxel, a truncated symbolic distance function delta value of the corresponding voxel;

the mesh generation method further includes:

and in response to the situation that the truncated sign distance function value of the voxel corresponding to the depth image is smaller than a preset value after the structured data increment information of the voxel corresponding to the depth image is removed from the target structured data, setting the state information of the voxel corresponding to the depth image to be in a deleted state.

12. The mesh generation method of any one of claims 1 to 11, further comprising, before generating the target mesh corresponding to the current depth image based on the structured data increment information corresponding to the neighborhood voxels of the voxels:

projecting the target voxel onto the current depth image;

13. The mesh generation method of claim 12, wherein the structured data delta information comprises state information of the corresponding voxel and a truncated symbolic distance function delta value of the corresponding voxel;

and setting the state information of the target voxel to be an update state.

14. The mesh generation method of any one of claims 1 to 13, further comprising, before the determining, based on image coordinates of a pixel in the current depth image, a corresponding voxel of the pixel in a preset voxel space, further:

15. A mesh generation apparatus, comprising:

16. An electronic device, comprising: a processor, a memory and a bus, the memory storing machine-readable instructions executable by the processor, the processor and the memory communicating over the bus when the electronic device is running, the machine-readable instructions when executed by the processor performing the steps of the mesh generation method of any of claims 1 to 14.

17. A computer-readable storage medium, characterized in that the computer-readable storage medium has stored thereon a computer program which, when being executed by a processor, carries out the steps of the mesh generation method according to any one of claims 1 to 14.