CN117078748A - Method, device and storage medium for positioning virtual object in mixed reality scene - Google Patents
Method, device and storage medium for positioning virtual object in mixed reality scene Download PDFInfo
- Publication number
- CN117078748A CN117078748A CN202311121113.2A CN202311121113A CN117078748A CN 117078748 A CN117078748 A CN 117078748A CN 202311121113 A CN202311121113 A CN 202311121113A CN 117078748 A CN117078748 A CN 117078748A
- Authority
- CN
- China
- Prior art keywords
- virtual object
- space model
- depth information
- position coordinate
- coordinate
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 52
- 230000009466 transformation Effects 0.000 claims description 34
- 239000011159 matrix material Substances 0.000 claims description 33
- 230000006870 function Effects 0.000 claims description 15
- 238000004422 calculation algorithm Methods 0.000 claims description 9
- 238000005457 optimization Methods 0.000 claims description 9
- 238000004088 simulation Methods 0.000 claims description 9
- 238000004364 calculation method Methods 0.000 claims description 7
- 238000006243 chemical reaction Methods 0.000 claims description 5
- 238000010586 diagram Methods 0.000 abstract description 28
- 230000035515 penetration Effects 0.000 abstract description 7
- 238000004590 computer program Methods 0.000 description 13
- 238000013461 design Methods 0.000 description 6
- 230000000694 effects Effects 0.000 description 6
- 238000012545 processing Methods 0.000 description 5
- 230000008901 benefit Effects 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 4
- 230000004807 localization Effects 0.000 description 3
- 230000000149 penetrating effect Effects 0.000 description 3
- 238000004891 communication Methods 0.000 description 2
- 230000014509 gene expression Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 230000002194 synthesizing effect Effects 0.000 description 2
- 230000015572 biosynthetic process Effects 0.000 description 1
- 238000011960 computer-aided design Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000007599 discharging Methods 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 230000005055 memory storage Effects 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 238000009877 rendering Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 238000003786 synthesis reaction Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
- G06T19/006—Mixed reality
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2200/00—Indexing scheme for image data processing or generation, in general
- G06T2200/04—Indexing scheme for image data processing or generation, in general involving 3D image data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10004—Still image; Photographic image
- G06T2207/10012—Stereo images
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Computer Graphics (AREA)
- Computer Hardware Design (AREA)
- General Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Processing Or Creating Images (AREA)
Abstract
The embodiment of the disclosure discloses a virtual object positioning method, a device and a storage medium in a mixed reality scene, wherein the method comprises the following steps: determining a second position coordinate of the virtual object in the virtual space model and a third position coordinate in the real space model based on the first position coordinate of the virtual object in the house type graph; collecting first depth information of the virtual object at a position corresponding to the second position coordinate; collecting second depth information of the virtual object at a position corresponding to the third position coordinate; calculating an error between the first depth information and the second depth information; and calculating target position coordinates of the virtual object in the real space model based on the first depth information, the third position coordinates and the error. According to the technical scheme, the relative spatial position of the virtual object in the real space model is optimized to be consistent with the relative spatial position of the virtual object in the virtual space model, so that the problem of virtual object mold penetration caused by inconsistent house type diagrams and the real space model is effectively solved.
Description
Technical Field
The disclosure relates to the technical field of mixed reality, and in particular relates to a method, a device and a storage medium for positioning a virtual object in a mixed reality scene.
Background
Synthesizing virtual objects into real scenes/pictures, and completing realistic virtual-real fusion is a problem to be solved in the field of immersive reality mixed reality. In the existing technology for synthesizing virtual objects into real scenes/pictures, firstly, a vector drawing software (Adobe illustrator, abbreviated as AI) technology is utilized to carry out automatic and stylized design in a house type picture corresponding to a real scene, then the position of the virtual object in the house type picture is transformed into the position in a real space model, and the problem that the position of the virtual object in the house type picture is directly transformed into the real space model due to the fact that misalignment exists possibly exists in the house type picture and the real space model is caused.
Disclosure of Invention
One technical problem to be solved by the embodiments of the present disclosure is: provided are a method, a device and a storage medium for positioning a virtual object in a mixed reality scene.
According to an aspect of an embodiment of the present disclosure, there is provided a method for positioning a virtual object in a mixed reality scene, the method including:
Determining a second position coordinate of the virtual object in the virtual space model and a third position coordinate in the real space model based on a first position coordinate of the virtual object in the house type graph;
collecting first depth information of the virtual object in the virtual space model at a position corresponding to the second position coordinate;
acquiring second depth information of the virtual object in the real space model at a position corresponding to the third position coordinate;
calculating an error between the first depth information and the second depth information;
calculating a target position coordinate of the virtual object in the real space model based on the third position coordinate and the error, and calculating a target position coordinate of the virtual object in the real space model based on the first depth information, the third position coordinate and the error.
In one embodiment of the present disclosure, the calculating, based on the first depth information, the third position coordinates, and the error, target position coordinates of the virtual object in the real space model includes:
iteratively updating the position of the virtual object in the real space model through a gradient descent optimization algorithm, and updating the third position coordinate by using the updated position coordinate;
Repeating the operation step of collecting the second depth information of the virtual object in the real space model at the position corresponding to the third position coordinate according to the third position coordinate updated each time, until the error converges to the minimum value;
and determining the position coordinates of the corresponding virtual object in the real space model when the error converges to the minimum value as the target position coordinates.
In yet another embodiment of the present disclosure, the determining, based on the first position coordinate of the virtual object in the house type graph, the second position coordinate of the virtual object in the virtual space model and the third position coordinate in the real space model includes:
calculating a value of a preset first coordinate system transformation matrix multiplied by the first position coordinate to obtain the second position coordinate;
calculating a value of a preset second coordinate system transformation matrix multiplied by the first position coordinate to obtain the third position coordinate; or, calculating a preset third coordinate system transformation matrix to multiply the second position coordinate left to obtain the third position coordinate;
the second coordinate system transformation matrix is obtained by multiplying the first coordinate system transformation matrix and the third coordinate system transformation matrix.
In yet another embodiment of the present disclosure, the first position coordinate is a coordinate of a center point of a virtual object in the house type graph;
the second position coordinates are coordinates of a center point of a cuboid bounding box of the virtual object in the virtual space model;
and the third position coordinates are coordinates of a center point of a cuboid bounding box of the virtual object in the real space model.
In yet another embodiment of the present disclosure, the acquiring, at a position corresponding to the second position coordinate, first depth information of the virtual object in the virtual space model includes:
and acquiring a first depth map by using a point simulation depth camera corresponding to the second position coordinate, wherein the first depth map is used for representing the depth information of the virtual object in the virtual space model.
In yet another embodiment of the present disclosure, the acquiring second depth information of the virtual object in the real space model at the position corresponding to the third position coordinate includes:
and acquiring a second depth map by using a point simulation depth camera corresponding to the third position coordinate, wherein the second depth map is used for representing the depth information of the virtual object in the real space model.
In yet another embodiment of the present disclosure, the calculating an error between the first depth information and the second depth information includes:
an absolute value of a difference between the first depth map and the second depth map is calculated by setting a loss function.
According to still another aspect of the embodiments of the present disclosure, there is provided a virtual object positioning apparatus in a mixed reality scene, the apparatus including:
the coordinate conversion module is used for determining a second position coordinate of the virtual object in the virtual space model and a third position coordinate of the virtual object in the real space model based on a first position coordinate of the virtual object in the house type graph;
the first acquisition module is used for acquiring first depth information of the virtual object in the virtual space model at a position corresponding to the second position coordinate;
the second acquisition module is used for acquiring second depth information of the virtual object in the real space model at a position corresponding to the third position coordinate;
a calculation module for calculating an error between the first depth information and the second depth information;
and the target position determining module is used for calculating target position coordinates of the virtual object in the real space model based on the first depth information, the third position coordinates and the error.
In one embodiment of the present disclosure, the target location determination module includes:
the iteration sub-module is used for iteratively updating the position of the virtual object in the real space model through a gradient descent optimization algorithm, and updating the third position coordinate by using the updated position coordinate; repeating the operation step of collecting the second depth information of the virtual object in the real space model at the position corresponding to the third position coordinate according to the third position coordinate updated each time, until the error converges to the minimum value;
and the target position determining sub-module is used for determining the position coordinates of the corresponding virtual object in the real space model when the error converges to the minimum value as the target position coordinates.
In yet another embodiment of the present disclosure, the coordinate conversion module includes:
the first computing sub-module is used for computing a value of the first position coordinate multiplied by a preset first coordinate system transformation matrix to obtain the second position coordinate;
the second calculation sub-module is used for calculating the value of the first position coordinate multiplied by the preset second coordinate system transformation matrix to obtain the third position coordinate; or, calculating a preset third coordinate system transformation matrix to multiply the second position coordinate left to obtain the third position coordinate;
The second coordinate system transformation matrix is obtained by multiplying the first coordinate system transformation matrix and the third coordinate system transformation matrix.
In yet another embodiment of the present disclosure, the first position coordinate is a coordinate of a center point of a virtual object in the house type graph;
the second position coordinates are coordinates of a center point of a cuboid bounding box of the virtual object in the virtual space model;
and the third position coordinates are coordinates of a center point of a cuboid bounding box of the virtual object in the real space model.
In yet another embodiment of the disclosure, the first acquisition module is specifically configured to acquire a first depth map at a point simulation depth camera corresponding to the second position coordinate, where the first depth map is used to represent depth information of the virtual object in the virtual space model.
In yet another embodiment of the present disclosure, the second acquisition module is specifically configured to acquire a second depth map at a point simulation depth camera corresponding to the third position coordinate, where the second depth map is used to represent depth information of the virtual object in the real space model.
In a further embodiment of the disclosure, the calculating module is specifically configured to calculate an absolute value of a difference between the first depth map and the second depth map by setting a loss function.
According to still another aspect of the embodiments of the present disclosure, there is provided an electronic device including:
a memory for storing a computer program;
and the processor is used for executing the computer program stored in the memory, and when the computer program is executed, the virtual object positioning method in the mixed reality scene is realized.
According to still another aspect of the embodiments of the present disclosure, there is provided a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the above-described virtual object positioning method in a mixed reality scenario.
According to yet another aspect of embodiments of the present disclosure, there is provided a computer program product comprising a computer program/instruction which, when executed by a processor, implements the above-described method of positioning a virtual object in a mixed reality scenario.
Based on the method, the device and the storage medium for positioning the virtual object in the mixed reality scene provided by the embodiment of the disclosure, when the virtual object needs to be synthesized into the real scene, the virtual object can be designed in the house type graph first, and then the second position coordinate of the virtual object in the virtual space model and the third position coordinate of the virtual object in the real space model are determined based on the first position coordinate of the virtual object in the house type graph; then, first depth information of the virtual object in the virtual space model is acquired at a position corresponding to the second position coordinate, and second depth information of the virtual object in the real space model is acquired at a position corresponding to the third position coordinate; and calculating to obtain target position coordinates of the virtual object in the real space model according to the error between the first depth information and the second depth information and the initial third position coordinates of the virtual object in the real space model. According to the technical scheme, the relative spatial position of the virtual object in the real space model and the relative spatial position of the virtual object in the virtual space model are optimized to be consistent with each other through iterative updating based on the relative spatial position (first depth information) of the virtual object in the virtual space model and the initial relative spatial position (second depth information) of the virtual object in the real space model, and the problem of virtual object mold penetration caused by inconsistent house type diagrams and the real space model is effectively solved.
The technical scheme of the present disclosure is described in further detail below through the accompanying drawings and examples.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the disclosure and together with the description, serve to explain the principles of the disclosure.
The disclosure may be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings in which:
FIG. 1 is a flow chart of one embodiment of a method of virtual object positioning in a mixed reality scenario of the present disclosure;
FIG. 2A is a flow chart of the present disclosure for determining a target position of a virtual object in a real space model;
FIG. 2B is a schematic drawing of the present disclosure for directly converting the position of a virtual object in a house type graph into a real space model;
FIG. 2C is a schematic diagram of the present disclosure of a virtual object at a target depth of a virtual space model, an initial depth of a real space model, and an optimized depth;
fig. 2D is an initial position schematic diagram and an optimized position schematic diagram of a virtual object in a real space model according to the present disclosure;
FIG. 3 is a schematic diagram illustrating the structure of one embodiment of a virtual object positioning device in a mixed reality scenario of the present disclosure;
FIG. 4 is a schematic structural diagram of a virtual object positioning apparatus in a mixed reality scenario of the present disclosure;
Fig. 5 is a block diagram of an electronic device provided in an exemplary embodiment of the present disclosure.
Detailed Description
Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that: the relative arrangement of the components and steps, numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present disclosure unless it is specifically stated otherwise.
Meanwhile, it should be understood that the sizes of the respective parts shown in the drawings are not drawn in actual scale for convenience of description.
The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the disclosure, its application, or uses.
Techniques, methods, and apparatus known to one of ordinary skill in the relevant art may not be discussed in detail, but are intended to be part of the specification where appropriate.
It should be noted that: like reference numerals and letters denote like items in the following figures, and thus once an item is defined in one figure, no further discussion thereof is necessary in subsequent figures.
Embodiments of the present disclosure may be applicable to electronic devices such as computer systems/servers that are operable with numerous other general purpose or special purpose computing system environments or configurations. Examples of well known computing systems, environments, and/or configurations that may be suitable for use with electronic devices, e.g., computer systems/servers, include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, hand-held or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, minicomputer systems, mainframe computer systems, and distributed cloud computing technology environments that include any of the above systems, and the like.
An electronic device, such as a computer system/server, may be described in the general context of computer-system-executable instructions, such as program modules, being executed by a computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc., that perform particular tasks or implement particular abstract data types. The computer system/server may be implemented in a distributed cloud computing environment in which tasks are performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules may be located in both local and remote computing system storage media including memory storage devices.
Exemplary embodiments of the invention
FIG. 1 is a flow chart of one embodiment of a method of virtual object positioning in a mixed reality scenario of the present disclosure; the method for positioning the virtual object in the mixed reality scene can be applied to electronic equipment (such as a computer system and a server), and as shown in fig. 1, the method for positioning the virtual object in the mixed reality scene comprises the following steps:
in step 101, based on the first position coordinates of the virtual object in the house type graph, the second position coordinates of the virtual object in the virtual space model and the third position coordinates in the real space model are determined.
In an embodiment, the virtual furniture placement design can be performed in the house type graph through an AI technology, so as to obtain a first position coordinate P_floor of each virtual furniture in the house type graph.
In one embodiment, the virtual space model is determined based on the house type graph, and specifically, the house type graph can be converted into the virtual space model through some design instructions and design algorithms, such as calculation-aided design (Computer Aided Design, abbreviated as CAD).
In one embodiment, the first position coordinates may be converted to the second position coordinates by equation (1):
p_vr1=t1 p_floorplan formula (1)
In the formula (1), P_floorplan represents a first position coordinate of a virtual object in a house type diagram; t1 is a known first coordinate system transformation matrix, and P_v1 represents second position coordinates of the virtual object in the virtual space model.
In one embodiment, in calculating the third position coordinate, the first position coordinate may be converted into the third position coordinate by equation (2):
p_vr2=t2 p_floorplan formula (2)
In the formula (2), P_floorplan represents a first position coordinate of a virtual object in a house type diagram; t2 is a known second coordinate system transformation matrix, and P_vr2 represents a third position coordinate of the virtual object in the virtual space model.
For the calculation of the third position coordinates, the second position coordinates can also be converted into third position coordinates by the expression (3):
p_vr2=t3 p_vr1 formula (3)
In the formula (3), P_v1 represents a second position coordinate of the virtual object in the house type graph; t3 is a known third coordinate system transformation matrix, and p_vr2 represents a third position coordinate of the virtual object in the virtual space model.
The second coordinate system transformation matrix T2 is obtained by multiplying the first coordinate system transformation matrix T1 by the third coordinate system transformation matrix T3.
In order to implement the embodiments of the present disclosure, a first coordinate system transformation matrix, a second coordinate system transformation matrix, and a third coordinate system transformation matrix may be preset for implementing coordinate system transformation.
In an embodiment, since the space occupied by the virtual object is not a point, the first position coordinate, the second position coordinate, and the third position coordinate defined in this embodiment are the coordinates of the center point of the box where the virtual object is located in the house type map, the virtual space model, and the real space model, respectively.
In one embodiment, in the house type graph, the box in which the virtual object is located may be a rectangular frame; in the virtual space model and the real space model, the box where the virtual object is located is a bounding box cuboid.
In an embodiment, since the house type graph and the real space model have a misalignment problem, a mold penetration problem may exist in the real space model according to the third position coordinate obtained by converting the first position coordinate, see fig. 2B, where the left side graph shows a furniture design in the house type graph, and the right side graph shows an effect graph showing that the furniture position based on the left side graph is directly converted to the furniture position in the real space model, and it can be seen that a part of the furniture indicated by reference numerals 21 and 22 is penetrated into the wall.
In step 102, first depth information of the virtual object in the virtual space model is collected at a position corresponding to the second position coordinate.
In an embodiment, the first depth information is used to represent a relative position of the virtual object in the virtual space model, and in particular the first depth information may be represented by a depth map, which indicates a target depth map of the virtual object in the space model.
In an embodiment, the first depth map may be acquired at a point corresponding to the second position coordinates by simulating the depth camera. The depth map may be a 360 degree depth panorama.
In step 103, second depth information of the virtual object in the real space model is acquired at a position corresponding to the third position coordinate.
In an embodiment, the second depth information is used to represent a relative position of the virtual object in the real space model, and the second depth information may be represented by a depth map.
In an embodiment, the second depth map may be acquired by using a point simulation depth camera corresponding to the third position coordinate.
In step 104, an error between the first depth information and the second depth information is calculated.
In an embodiment, the absolute value of the difference between the first depth map and the second depth map may be calculated by setting a loss function, resulting in an error between the first depth information and the second depth information. The set loss function may be a mean square error (Mean Square Error, abbreviated MSE) loss function, or an average absolute error (Mean Absolute Error, abbreviated MAE) loss function, or a Huber loss function.
In step 105, target position coordinates of the virtual object in the real space model are calculated based on the first depth information, the third position coordinates and the error.
In an embodiment, by optimizing the position of the virtual object in the real space model, the relative spatial position of the virtual object in the real space model is identical to the relative spatial position in the virtual space model, and when the relative positions of the virtual object in the real space model and the virtual space model are identical, the position of the virtual object is the target position coordinate.
In an embodiment, see fig. 2C, where the target depth of the virtual object in the virtual space model, the initial depth of the real space model and the optimized depth are illustrated, the left side diagram of fig. 2C is the target depth diagram (i.e. the first depth diagram collected at the position of the virtual object in the virtual space model), the middle diagram of fig. 2C is the initial depth diagram in the real space model (i.e. the second depth diagram collected at the initial position (third position coordinates) of the virtual object in the real space model), and the right side diagram of fig. 2C is the optimized depth diagram collected after the virtual object is moved to the target position coordinates, and it is known that the relative positions of the virtual objects in the real space model and the virtual space model achieve a consistent effect through the position optimization by the left side depth diagram and the right side depth diagram.
In an embodiment, referring to fig. 2D, an initial position schematic diagram and an optimized position effect diagram of a virtual object in a real space model are shown, a television at a position indicated by a reference numeral 23 in a left side diagram of fig. 2D has a problem of penetrating into a wall, an air conditioner at a position indicated by a reference numeral 24 also has a part of the problem of penetrating into the wall, and the relative positions of the television and the air conditioner in an indoor space are optimized respectively to obtain a right side diagram of fig. 2D, so that the problem of penetrating into the mold of the television and the air conditioner is solved.
In an embodiment, a specific implementation manner of calculating the target position coordinates of the virtual object in the real space model may be referred to the embodiment shown in fig. 2A, which is not described in detail herein.
Step 101-105 above, when the virtual object needs to be synthesized into the real scene, the virtual object may be designed in the house type graph first, and then the second position coordinate of the virtual object in the virtual space model and the third position coordinate of the virtual object in the real space model are determined based on the first position coordinate of the virtual object in the house type graph; then, first depth information of the virtual object in the virtual space model is acquired at a position corresponding to the second position coordinate, and second depth information of the virtual object in the real space model is acquired at a position corresponding to the third position coordinate; and calculating to obtain target position coordinates of the virtual object in the real space model according to the error between the first depth information and the second depth information and the initial third position coordinates of the virtual object in the real space model. According to the technical scheme, the relative spatial position of the virtual object in the real space model and the relative spatial position of the virtual object in the virtual space model are optimized to be consistent with each other through iterative updating based on the relative spatial position (first depth information) of the virtual object in the virtual space model and the initial relative spatial position (second depth information) of the virtual object in the real space model, and the problem of virtual object mold penetration caused by inconsistent house type diagrams and the real space model is effectively solved.
To better illustrate the scheme of virtual object positioning in the mixed reality scenario of the present disclosure, another embodiment is described below.
Fig. 2A is a flowchart of determining a target position of a virtual object in a real space model according to the present disclosure, fig. 2B is a schematic drawing of directly converting a position of the virtual object in a house type graph into the real space model according to the present disclosure, fig. 2C is a schematic drawing of target depth of the virtual object in the virtual space model, initial depth and optimized depth of the real space model according to the present disclosure, and fig. 2D is a schematic drawing of initial position and optimized position of the virtual object in the real space model according to the present disclosure; the present embodiment is exemplified by how to implement positioning of a virtual object in a mixed reality scene, as shown in fig. 2A, and includes the following steps:
in step 201, the position of the virtual object in the real space model is iteratively updated by a gradient descent optimization algorithm, and the third position coordinates are updated using the coordinates of the updated position.
In one embodiment, before iteratively updating the position of the virtual object in the real space model, the Depth map of the virtual object in the virtual space model and the error |depth of the Depth map in the real space model need to be calculated by setting a loss function first vr -Depth floorplan I (wherein, depth) vr Depth for rendering a Depth map at a position of a simulated Depth camera in a real space model floorplan A depth map rendered for simulating a depth camera in a virtual spatial model of the house type map).
In an embodiment, after calculating the error of the depth map of the virtual object in the virtual space model and the depth map in the real space model, the position of the simulated depth camera in the real space model (i.e. the position of the virtual object in the real space model) can be optimized by using a gradient descent optimization algorithm, and the coordinates of the position updated by each iteration are used to update the third position coordinates.
In step 202, according to the third position coordinate updated in each iteration, the operation step of collecting the second depth information of the virtual object in the real space model at the position corresponding to the third position coordinate is repeatedly performed until the error converges to the minimum value.
In an embodiment, after updating the third position coordinate by using the position coordinate of the position updated each time through the gradient descent optimization algorithm, the operation step of collecting the second depth information of the virtual object in the real space model at the position corresponding to the third position coordinate may be executed again, after the second depth information is calculated, the error between the first depth information and the second depth information is calculated again until the error converges to the minimum value, and it may be determined that the last updated third position coordinate is the position coordinate of the virtual object in the real space model, that is, the target position coordinate.
In step 203, the position coordinates of the corresponding virtual object in the real space model when the error converges to the minimum value are determined as the target position coordinates.
In an embodiment, when the error converges to the minimum value, it may be determined that the relative position of the virtual object in the real space model is consistent with the relative position in the virtual space model, and the virtual object does not have a mold penetration in the virtual space model, so that the virtual object also avoids the mold penetration problem in the real space model after optimizing the position.
Through the steps 201 to 203, the optimal position of the virtual object in the real space model can be automatically repositioned through the gradient descent optimization algorithm, so that the phenomenon of mold penetration/collision can be avoided, the virtual furniture is still at the reasonable placement position designed by the AI in the house type graph, and the synthesis effect of the virtual object in the real scene is optimized.
Corresponding to the embodiment of the method for positioning the virtual object in the mixed reality scene, the disclosure also provides a corresponding embodiment of the device for positioning the virtual object in the mixed reality scene.
Fig. 3 is a schematic structural diagram of an embodiment of a virtual object positioning apparatus in a mixed reality scenario of the present disclosure, where the apparatus is applied to an electronic device (e.g., a computer system, a server), as shown in fig. 3, and the apparatus includes:
A coordinate conversion module 31, configured to determine, based on a first position coordinate of a virtual object in a house type graph, a second position coordinate of the virtual object in a virtual space model and a third position coordinate in a real space model;
a first acquisition module 32, configured to acquire first depth information of the virtual object in the virtual space model at a position corresponding to the second position coordinate;
a second acquisition module 33, configured to acquire second depth information of the virtual object in the real space model at a position corresponding to the third position coordinate;
a calculation module 34 for calculating an error between the first depth information and the second depth information;
a target position determining module 35, configured to calculate target position coordinates of the virtual object in the real space model based on the first depth information, the third position coordinates and the error.
Fig. 4 is a schematic structural diagram of still another embodiment of a virtual object positioning device in a mixed reality scenario of the disclosure, as shown in fig. 4, on the basis of the embodiment shown in fig. 3, in an embodiment, the target position determining module 35 includes:
an iteration sub-module 351, configured to iteratively update, by using a gradient descent optimization algorithm, a position of the virtual object in the real space model, and update the third position coordinate using the updated position coordinate until the error converges to a minimum value;
And a target position determining sub-module 352, configured to determine, as the target position coordinate, a position coordinate of the virtual object in the real space model corresponding to the error converged to a minimum value.
In one embodiment, the coordinate conversion module 31 includes:
a first calculation sub-module 311, configured to calculate a value obtained by multiplying the first position coordinate by a preset first coordinate system transformation matrix, to obtain the second position coordinate;
a second calculation sub-module 312, configured to calculate a value obtained by multiplying the first position coordinate by a preset second coordinate system transformation matrix to obtain the third position coordinate; or, calculating a preset third coordinate system transformation matrix to multiply the second position coordinate left to obtain the third position coordinate;
the second coordinate system transformation matrix is obtained by multiplying the first coordinate system transformation matrix and the third coordinate system transformation matrix.
In an embodiment, the first position coordinate is a coordinate of a center point of a virtual object in the house type graph;
the second position coordinates are coordinates of a center point of a cuboid bounding box of the virtual object in the virtual space model;
and the third position coordinates are coordinates of a center point of a cuboid bounding box of the virtual object in the real space model.
In an embodiment, the first acquisition module 32 is specifically configured to acquire a first depth map at a point simulation depth camera corresponding to the second position coordinate, where the first depth map is used to represent depth information of the virtual object in the virtual space model.
In an embodiment, the second acquisition module 33 is specifically configured to acquire a second depth map from the point simulation depth camera corresponding to the third position coordinate, where the second depth map is used to represent depth information of the virtual object in the real space model.
In an embodiment, the calculating module 34 is specifically configured to calculate an absolute value of a difference between the first depth map and the second depth map by setting a loss function.
The implementation process of the functions and roles of each unit in the above device is specifically shown in the implementation process of the corresponding steps in the above method, and will not be described herein again.
For the device embodiments, reference is made to the description of the method embodiments for the relevant points, since they essentially correspond to the method embodiments. The apparatus embodiments described above are merely illustrative, wherein elements illustrated as separate elements may or may not be physically separate, and elements shown as elements may or may not be physical elements, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the objectives of the disclosed solution. Those of ordinary skill in the art will understand and implement the present invention without undue burden.
An electronic device according to an embodiment of the present disclosure, in which an apparatus for implementing a method of an embodiment of the present disclosure may be integrated, is described below with reference to fig. 5. Fig. 5 is a block diagram of an electronic device provided in an exemplary embodiment of the present disclosure, as shown in fig. 5, the electronic device including one or more processors 51, memory 52 of one or more computer-readable storage media, and a computer program stored on the memory and executable on the processors. The virtual object positioning method in the mixed reality scenario described above can be implemented when the program of the memory 52 is executed.
In particular, in practice, the electronic device may further comprise input means 53, output means 54, etc., which are interconnected by a bus system and/or other form of connection mechanism (not shown). It will be appreciated by those skilled in the art that the structure of the electronic device shown in fig. 5 is not limiting of the electronic device and may include more or fewer components than shown, or certain components, or a different arrangement of components. Wherein:
the processor 51 may be a Central Processing Unit (CPU) or other form of processing unit having data processing and/or instruction execution capabilities, performing various functions and processing data by running or executing software programs and/or modules stored in the memory 52 and invoking data stored in the memory 52 to thereby monitor the electronic device as a whole.
Memory 52 may include one or more computer program products that may include various forms of computer-readable storage media, such as volatile memory and/or non-volatile memory. Volatile memory can include, for example, random Access Memory (RAM) and/or cache memory (cache) and the like. The non-volatile memory may include, for example, read Only Memory (ROM), hard disk, flash memory, and the like. One or more computer program instructions may be stored on a computer readable storage medium and the processor 51 may execute the program instructions to implement the virtual object localization method and/or other desired functions in the mixed reality scenario of the various embodiments of the present disclosure above. Various contents such as an input signal, a signal component, a noise component, and the like may also be stored in the computer-readable storage medium.
The input device 53 may be used to receive input numeric or character information and to generate keyboard, mouse, joystick, optical or trackball signal inputs associated with user settings and function control.
The output device 54 may output various information to the outside, including the determined distance information, direction information, and the like. The output device 54 may include, for example, a display, speakers, a printer, and a communication network and remote output devices connected thereto, etc.
The electronic device may further include a power supply for supplying power to the respective components, and may be logically connected to the processor 51 through a power management system, so that functions of managing charging, discharging, power consumption management, and the like are implemented through the power management system. The power supply may also include one or more of any of a direct current or alternating current power supply, a recharging system, a power failure detection circuit, a power converter or inverter, a power status indicator, and the like.
Of course, only some of the components of the electronic device relevant to the present disclosure are shown in fig. 5 for simplicity, components such as buses, input/output interfaces, etc. being omitted. In addition, the electronic device may include any other suitable components depending on the particular application.
In addition to the methods and apparatus described above, embodiments of the present disclosure may also be a computer program product comprising computer program instructions which, when executed by a processor, cause the processor to perform the steps in a virtual object localization method in a mixed reality scenario according to various embodiments of the present disclosure described in the "exemplary methods" section of this specification.
The computer program product may write program code for performing the operations of embodiments of the present disclosure in any combination of one or more programming languages, including an object oriented programming language such as Java, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device, partly on a remote computing device, or entirely on the remote computing device or server.
Furthermore, embodiments of the present disclosure may also be a computer-readable storage medium, having stored thereon computer program instructions, which when executed by a processor, cause the processor to perform the steps in a virtual object localization method in a mixed reality scenario according to various embodiments of the present disclosure described in the above "exemplary methods" section of this specification.
The computer readable storage medium may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may include, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium would include the following: an electrical connection having one or more wires, a portable disk, a hard disk, random Access Memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
The basic principles of the present disclosure have been described above in connection with specific embodiments, however, it should be noted that the advantages, benefits, effects, etc. mentioned in the present disclosure are merely examples and not limiting, and these advantages, benefits, effects, etc. are not to be considered as necessarily possessed by the various embodiments of the present disclosure. Furthermore, the specific details disclosed herein are for purposes of illustration and understanding only, and are not intended to be limiting, since the disclosure is not necessarily limited to practice with the specific details described.
In this specification, each embodiment is described in a progressive manner, and each embodiment is mainly described in a different manner from other embodiments, so that the same or similar parts between the embodiments are mutually referred to. For system embodiments, the description is relatively simple as it essentially corresponds to method embodiments, and reference should be made to the description of method embodiments for relevant points.
Those of ordinary skill in the art will appreciate that: all or part of the steps for implementing the above method embodiments may be implemented by hardware associated with program instructions, where the foregoing program may be stored in a computer readable storage medium, and when executed, the program performs steps including the above method embodiments; and the aforementioned storage medium includes: various media that can store program code, such as ROM, RAM, magnetic or optical disks.
The methods and apparatus of the present disclosure may be implemented in a number of ways. For example, the methods and apparatus of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, firmware. The above-described sequence of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the sequence specifically described above unless specifically stated otherwise. Furthermore, in some embodiments, the present disclosure may also be implemented as programs recorded in a recording medium, the programs including machine-readable instructions for implementing the methods according to the present disclosure. Thus, the present disclosure also covers a recording medium storing a program for executing the method according to the present disclosure.
The description of the present disclosure has been presented for purposes of illustration and description, and is not intended to be exhaustive or limited to the disclosure in the form disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art. The embodiments were chosen and described in order to best explain the principles of the disclosure and the practical application, and to enable others of ordinary skill in the art to understand the disclosure for various embodiments with various modifications as are suited to the particular use contemplated.
Claims (10)
1. A method for locating a virtual object in a mixed reality scene, the method comprising:
determining a second position coordinate of the virtual object in the virtual space model and a third position coordinate in the real space model based on a first position coordinate of the virtual object in the house type graph;
collecting first depth information of the virtual object in the virtual space model at a position corresponding to the second position coordinate;
acquiring second depth information of the virtual object in the real space model at a position corresponding to the third position coordinate;
calculating an error between the first depth information and the second depth information;
and calculating target position coordinates of the virtual object in the real space model based on the first depth information, the third position coordinates and the error.
2. The method of claim 1, wherein the calculating target position coordinates of the virtual object in the real space model based on the first depth information, the third position coordinates, and the error comprises:
iteratively updating the position of the virtual object in the real space model through a gradient descent optimization algorithm, and updating the third position coordinate by using the updated position coordinate;
repeating the operation step of collecting the second depth information of the virtual object in the real space model at the position corresponding to the third position coordinate according to the third position coordinate updated each time, until the error converges to the minimum value;
and determining the position coordinates of the corresponding virtual object in the real space model when the error converges to the minimum value as the target position coordinates.
3. The method of claim 1, wherein determining the second position coordinates of the virtual object in the virtual space model and the third position coordinates in the real space model based on the first position coordinates of the virtual object in the house type graph comprises:
Calculating a value of a preset first coordinate system transformation matrix multiplied by the first position coordinate to obtain the second position coordinate;
calculating a value of a preset second coordinate system transformation matrix multiplied by the first position coordinate to obtain the third position coordinate; or, calculating a preset third coordinate system transformation matrix to multiply the second position coordinate left to obtain the third position coordinate;
the second coordinate system transformation matrix is obtained by multiplying the first coordinate system transformation matrix and the third coordinate system transformation matrix.
4. The method of claim 1, wherein the first location coordinates are coordinates of a center point of the virtual object in the house type graph;
the second position coordinates are coordinates of a center point of a cuboid bounding box of the virtual object in the virtual space model;
and the third position coordinates are coordinates of a center point of a cuboid bounding box of the virtual object in the real space model.
5. The method of claim 1, wherein the acquiring the first depth information of the virtual object in the virtual space model at the location corresponding to the second location coordinate comprises:
And acquiring a first depth map by using a point simulation depth camera corresponding to the second position coordinate, wherein the first depth map is used for representing the depth information of the virtual object in the virtual space model.
6. The method of claim 5, wherein the acquiring second depth information of the virtual object in the real space model at the position corresponding to the third position coordinate comprises:
and acquiring a second depth map by using a point simulation depth camera corresponding to the third position coordinate, wherein the second depth map is used for representing the depth information of the virtual object in the real space model.
7. The method of claim 6, wherein the calculating an error between the first depth information and the second depth information comprises:
an absolute value of a difference between the first depth map and the second depth map is calculated by setting a loss function.
8. A virtual object positioning device in a mixed reality scene, the device comprising:
the coordinate conversion module is used for determining a second position coordinate of the virtual object in the virtual space model and a third position coordinate of the virtual object in the real space model based on a first position coordinate of the virtual object in the house type graph;
The first acquisition module is used for acquiring first depth information of the virtual object in the virtual space model at a position corresponding to the second position coordinate;
the second acquisition module is used for acquiring second depth information of the virtual object in the real space model at a position corresponding to the third position coordinate;
a calculation module for calculating an error between the first depth information and the second depth information;
and the target position determining module is used for calculating target position coordinates of the virtual object in the real space model based on the first depth information, the third position coordinates and the error.
9. An electronic device, comprising:
a memory for storing a computer product;
a processor for executing a computer product stored in said memory, which, when executed, implements the method of any of the preceding claims 1-7.
10. A computer readable storage medium having stored thereon computer instructions, which when executed by a processor, implement the method of any of the preceding claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311121113.2A CN117078748A (en) | 2023-08-31 | 2023-08-31 | Method, device and storage medium for positioning virtual object in mixed reality scene |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311121113.2A CN117078748A (en) | 2023-08-31 | 2023-08-31 | Method, device and storage medium for positioning virtual object in mixed reality scene |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117078748A true CN117078748A (en) | 2023-11-17 |
Family
ID=88705868
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311121113.2A Pending CN117078748A (en) | 2023-08-31 | 2023-08-31 | Method, device and storage medium for positioning virtual object in mixed reality scene |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117078748A (en) |
-
2023
- 2023-08-31 CN CN202311121113.2A patent/CN117078748A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106651987B (en) | Paths planning method and device | |
| CN111199062B (en) | Simulation method and system based on industrial development software and electronic equipment | |
| US20150029198A1 (en) | Motion control of active deformable objects | |
| CN111868738B (en) | Cross-device monitoring computer vision system | |
| CN116310062A (en) | Three-dimensional scene construction method and device, storage medium and electronic equipment | |
| KR20190063046A (en) | Method and apparatus for creating virtual reality contents for virtual reality-based interior consulting service | |
| CN112950759B (en) | Three-dimensional house model construction method and device based on house panoramic image | |
| CN112465971B (en) | Method and device for guiding point positions in model, storage medium and electronic equipment | |
| CN113129362B (en) | Method and device for acquiring three-dimensional coordinate data | |
| CN113989376A (en) | Method and device for acquiring indoor depth information and readable storage medium | |
| JP2021192244A (en) | Positional information generating method, related apparatus, and computer program product | |
| CN114241029B (en) | Image three-dimensional reconstruction method and device | |
| CN117078748A (en) | Method, device and storage medium for positioning virtual object in mixed reality scene | |
| CN115619989B (en) | Fusion effect diagram generation method and device, electronic equipment and storage medium | |
| CN115063564B (en) | Method, device and medium for displaying object label in two-dimensional display image | |
| CN113538562B (en) | Indoor area determination method and device, electronic equipment and storage medium | |
| CN114307158A (en) | Three-dimensional virtual scene data generation method and device, storage medium and terminal | |
| CN114170381A (en) | Three-dimensional path display method and device, readable storage medium and electronic equipment | |
| CN111986300A (en) | Rendering point determination method and device for house decoration, storage medium and electronic equipment | |
| CN113450438A (en) | Virtual character driving method and device based on motion capture and computer equipment | |
| CN110889157A (en) | Wind power plant layout visualization display method and related device | |
| CN115761122B (en) | Method, device, equipment and medium for realizing three-dimensional auxiliary ruler | |
| CN115761114B (en) | Video generation method, device and computer readable storage medium | |
| WO2024150339A1 (en) | Learning device, three-dimensional reconstruction device, learning method, three-dimensional reconstruction method, and program | |
| CN115481476B (en) | Data display method, device and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |