WO2024060952A1 - 用于渲染虚拟对象的方法、装置、设备和介质 - Google Patents

用于渲染虚拟对象的方法、装置、设备和介质 Download PDF

Info

Publication number
WO2024060952A1
WO2024060952A1 PCT/CN2023/115909 CN2023115909W WO2024060952A1 WO 2024060952 A1 WO2024060952 A1 WO 2024060952A1 CN 2023115909 W CN2023115909 W CN 2023115909W WO 2024060952 A1 WO2024060952 A1 WO 2024060952A1
Authority
WO
WIPO (PCT)
Prior art keywords
environment
rendering
environment image
images
image
Prior art date
Application number
PCT/CN2023/115909
Other languages
English (en)
French (fr)
Inventor
高星
刘慧琳
Original Assignee
北京字跳网络技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 北京字跳网络技术有限公司 filed Critical 北京字跳网络技术有限公司
Publication of WO2024060952A1 publication Critical patent/WO2024060952A1/zh

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/00—3D [Three Dimensional] image rendering

Definitions

  • Exemplary implementations of the present disclosure relate generally to rendering virtual objects, and in particular to methods, apparatus, devices, and computer-readable storage media for rendering virtual objects based on real-time selected environment images in augmented reality (AR) applications.
  • AR augmented reality
  • a large number of augmented reality applications have been developed. Users can use image acquisition devices in devices such as mobile terminals to shoot scenes in the real environment, and can add virtual objects to the captured videos of the real environment. For example, virtual objects can be placed at desired locations, or virtual characters can be added that can move, etc. Since the lighting in the real environment may continue to change, how to set lighting rendering parameters for virtual objects so that the lighting effects of the rendered virtual objects are consistent with the surrounding real environment has become an urgent problem to be solved.
  • a method for rendering virtual objects in an augmented reality application is provided.
  • multiple environment images of the real environment are acquired, and the multiple environment images include images of the real environment collected at multiple acquisition time points respectively.
  • an environment image matching the current time point when the augmented reality application is used is selected from the plurality of environment images as the rendering environment image.
  • an apparatus for rendering virtual objects in an augmented reality application includes: an acquisition module configured to acquire multiple environment images of the real environment, the multiple environment images including images of the real environment collected at multiple acquisition time points respectively; and a selection module configured to acquire based on the multiple acquisitions. A time point that selects an environment image that matches a current time point when the augmented reality application is used from a plurality of environment images as a rendering environment image; and a presentation module configured to present the rendering environment image in the augmented reality application. Rendered virtual object.
  • an electronic device in a third aspect of the present disclosure, includes: at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions when executed by the at least one processing unit, cause the device to perform A method according to the first aspect of the present disclosure.
  • a computer-readable storage medium having a computer program stored thereon.
  • the computer program when executed by a processor, causes the processor to implement the method according to the first aspect of the present disclosure.
  • FIG1 illustrates an example of an application environment in which implementations of the present disclosure may be used
  • FIG. 2 illustrates a block diagram for rendering virtual objects in an augmented reality application in accordance with some implementations of the present disclosure
  • FIG. 3 shows a block diagram of multiple environmental images collected at multiple acquisition time points respectively, according to some implementations of the present disclosure
  • FIG. 4 illustrates acquisition at multiple acquisition locations in accordance with some implementations of the present disclosure.
  • FIG. 5 illustrates a block diagram of a process for selecting a rendering environment image based on a comparison of a device location of an end device running an augmented reality application and a capture location, in accordance with some implementations of the present disclosure
  • FIG. 6 illustrates a block diagram of a process for selecting an environment image for rendering based on occlusion relationships, in accordance with some implementations of the present disclosure
  • FIG. 7 illustrates a block diagram of a process for converting an environment image into a standard environment image in accordance with some implementations of the present disclosure
  • FIG. 8 illustrates a block diagram of a process for mapping pixels in accordance with some implementations of the present disclosure
  • Figure 9 shows a block diagram of a spherical panoramic image in accordance with some implementations of the present disclosure.
  • FIG. 10 illustrates a block diagram of a process for generating new rendering parameters based on multiple environment images in accordance with some implementations of the present disclosure
  • 11A and 11B respectively illustrate block diagrams of rendering virtual objects in augmented reality applications according to some implementations of the present disclosure
  • FIG12 illustrates a flow chart of a method for rendering a virtual object according to some implementations of the present disclosure
  • FIG. 13 illustrates a block diagram of an apparatus for rendering virtual objects in accordance with some implementations of the present disclosure.
  • Figure 14 illustrates a block diagram of a device capable of implementing various implementations of the present disclosure.
  • the term “including” and similar terms shall be used. It is understood as open inclusion, i.e., “including but not limited to”.
  • the term “based on” should be understood as “based at least in part on”.
  • the term “an implementation” or “the implementation” should be understood as “at least one implementation”.
  • the term “some implementations” should be understood as “at least some implementations”.
  • the following may also include other explicit and implicit definitions.
  • the term “model” can represent the association relationship between various data. For example, the above-mentioned association relationship can be obtained based on a variety of technical solutions currently known and/or to be developed in the future.
  • a prompt message is sent to the user to clearly remind the user that the operation requested will require the acquisition and use of the user's personal information. Therefore, users can autonomously choose whether to provide personal information to software or hardware such as electronic devices, applications, servers or storage media that perform the operations of the technical solution of the present disclosure based on the prompt information.
  • the method of sending prompt information to the user can be, for example, a pop-up window, and the prompt information can be presented in the form of text in the pop-up window.
  • the pop-up window may also host a selection control for the user to select "agree” or "disagree” to provide personal information to the electronic device.
  • FIG. 1 illustrates how an implementation of the present disclosure may Take an example of an application environment in which 100 is used.
  • the AR application can be run at the terminal device, and the user can hold the terminal device, and use the augmented reality application 110 running at the terminal device to capture scenes in the real environment, and can add virtual objects 120 to the captured video of the real environment. (For example, add a cube sculpture).
  • the device position of the terminal device in the real environment is the same as the user's position.
  • FIG. 1 shows a virtual object 120 rendered using preset lighting rendering data.
  • the lighting effect of the rendered virtual object 120 is inconsistent with the surrounding real environment.
  • the preset lighting rendering data is set for the day scene, and the lighting effect of the virtual object 120 obtained by using the lighting rendering data is relatively bright.
  • the lighting of the real environment is dim.
  • the too bright virtual object 120 is not in harmony with the surrounding dim environment. This causes the visual effects of the augmented reality application 110 to be unrealistic.
  • the real environment may have complex lighting information.
  • lighting can be roughly divided into direct lighting and ambient lighting, and there may be multiple direct light sources in the real environment.
  • the image of the real environment taken can be processed based on the real-time lighting estimation technology, that is, the lighting analysis can be performed on the image of the real environment to obtain the real lighting.
  • this technical solution can only obtain the overall lighting intensity of the real environment or the most significant direct lighting, but the accuracy is not satisfactory.
  • the real-time lighting estimation technology generates a large computing resource overhead, and therefore it is difficult to implement in portable computing devices such as mobile terminals. At this point, how to determine the peripheral lighting information of the virtual object and render it in a more convenient and effective way has become a problem to be solved.
  • a method for rendering virtual objects in augmented reality applications is proposed.
  • multiple environment images of a real environment can be obtained, and these environment images can be A candidate rendering environment image for rendering virtual objects.
  • the plurality of environment images may be images of the real environment collected at multiple collection time points respectively.
  • multiple environmental images may be collected respectively at different time points during the day, such as during the day and at night.
  • AR applications can be run at a variety of terminal devices.
  • AR applications can be run on conventional mobile terminal devices (including, but not limited to, mobile phones, mobile tablet computing devices, mobile notebook computing devices, etc.).
  • AR applications can be run on wearable terminal devices (including but not limited to glasses devices with computing functions, helmet devices, etc.).
  • the AR application may be run on a computing device with separate display functions and computing functions (eg, a portable computing device is used to run the AR application, and a glasses device in communication with the portable computing device is used to display an interface of the AR application).
  • FIG. 2 illustrates a block diagram 200 for rendering virtual objects in an augmented reality application in accordance with some implementations of the present disclosure.
  • FIG. 2 illustrates a block diagram 200 for rendering virtual objects in an augmented reality application in accordance with some implementations of the present disclosure.
  • multiple environmental images collected at multiple acquisition times can be acquired.
  • the collection time point 210 of the environment image 220 may be daytime,..., and the collection time point 212 of the environment image 222 may be nighttime.
  • the current point in time 244 when the user uses the augmented reality application 240 may be determined. Further, based on the comparison between the current time point 244 and the multiple acquisition time points 210,...,212, the environment image 222 matching the current time point 244 can be selected from the multiple environment images 220,...,222 as the rendered environment image 230. Specifically, if the user uses the augmented reality application 240 during the day, the environment image 220 collected during the day may be selected as the rendered environment image 230 and rendered. At this time, the rendered virtual object 250 will present a bright lighting effect during the day. For another example, if the user uses the augmented reality application 240 at night, the environment image 222 collected at night may be selected as the rendered environment image 230 . At this time, the rendered virtual object 250 will exhibit a dim lighting effect at night.
  • the current time point 244 is at night, so the environment image 222 at night can be selected as the rendered environment image 230 .
  • the virtual object 250 may be rendered using the rendering environment image 230 and the rendered virtual object 250 may be presented in the augmented reality application 240 .
  • the nighttime lighting effect as shown in Figure 2 can be presented.
  • the rendering environment image 230 for rendering the virtual object 250 may be selected in real time based on the current point in time 244 and the rendering may be performed.
  • the illumination information of the rendered environment image 230 can be consistent with the illumination information of the real environment at the current time point 244, so that the virtual object 250 obtained by rendering can be consistent with the surrounding real environment.
  • FIG. 3 illustrates a block diagram 300 of multiple environmental images acquired at multiple acquisition time points, respectively, in accordance with some implementations of the present disclosure.
  • multiple environmental images can be collected at different time points in a day.
  • environmental images 220 can be collected at time points T1 at predetermined time intervals (eg, 2 hours, 4 hours, etc.),...
  • the environment image 222 is collected at time point TN.
  • a plurality of environment images may be stored directly at the terminal device for running the augmented reality application 240.
  • a plurality of environment images may be stored at a server providing the AR service in order to obtain a desired environment image via a network.
  • VPS Visual Positioning System
  • a large number of images of the real environment need to be collected in advance at different time points and different locations.
  • no additional steps are required to collect the multiple environments described above. Images 220, ..., 222, instead, the respective environment images collected in advance for VPS purposes may be directly used as environment images for rendering purposes. In this way, the workload of data collection is not increased, but already collected environment images can be reused to serve the purpose of rendering virtual objects 250 .
  • a set of environment images matching the current time point 244 may be first found from a plurality of environment images based on the current time point 244 . Further, an environment image that is more consistent with the location of the device can be found in the set of environment images as the rendered environment image 230 . Specifically, a plurality of environment images collected respectively at a plurality of collection locations in a real environment may be acquired.
  • FIG. 4 illustrates a block diagram 400 of multiple environmental images collected at multiple acquisition locations, respectively, in accordance with some implementations of the present disclosure.
  • the real environment 410 may include multiple acquisition locations 420, 430, . . . , and 440, and images of the environment may be acquired at each acquisition location.
  • environment image 220 may be acquired at acquisition location 420
  • other environment images may be acquired at acquisition locations 430 and 440, respectively, and so on.
  • multiple environmental images can be collected at multiple collection positions respectively; ...; at time point TN, multiple environmental images can be collected at multiple collection positions respectively. It will be understood that the present disclosure does not limit the number of environmental images collected at each time point, nor does it limit whether the collection positions selected at each time point are the same, but multiple environmental images collected for VPS purposes can be directly used. The above multiple environmental images can be managed according to the collection time and collection position.
  • a user's device location in the real environment 410 may be determined, and a rendering environment that matches the device location may be selected from a plurality of environment images based on a comparison of the device location with a plurality of acquisition locations.
  • Image 230. Further details of selecting a rendering environment image 230 are described with reference to FIG. 5 , which illustrates a method for selecting rendering based on a comparison of a device location 510 of an end device running an augmented reality application and a capture location in accordance with some implementations of the present disclosure.
  • Block diagram 500 of the process of environment imaging 230 is described with reference to FIG. 5 , which illustrates a method for selecting rendering based on a comparison of a device location 510 of an end device running an augmented reality application and a capture location in accordance with some implementations of the present disclosure.
  • the distances between the device location 510 and each collection location 420, 430, ..., and 440 can be obtained respectively.
  • the environmental image collected at the collection location closest to the device location 510 may be selected based on a comparison of the respective distances.
  • the distances between the device position 510 and the collection positions 420 and 440 are 520 and 530 respectively. Based on the comparison, it can be seen that the distance 520 is greater than the distance 530.
  • the environment image collected at the collection position 440 can be selected as the rendering Environmental images 230.
  • virtual objects can be rendered using an environment image that most closely approximates the surrounding environment seen by the user at the device location 510 250.
  • the lighting effect of the virtual object 250 can be made to better match the ambient lighting seen at the device location 510, thereby improving the visual effect of the augmented reality application 240.
  • the occlusion relationship may be further considered.
  • the spatial structure of the real environment 410 may be determined based on multiple environment images.
  • the spatial structure of the real environment 410 can be obtained based on currently known and/or three-dimensional reconstruction technologies that will be developed in the future, which will not be described again in this article.
  • the spatial structure can be used to determine whether there is an obstacle between the acquisition location of the rendering environment image and the device location 510 .
  • the spatial structure indicates the presence of an obstacle 610 in the real environment 410 and that the obstacle 610 is located between the device location 510 and the collection location 440 .
  • the obstacle 610 may cause the user at the device position 510 to not directly see the collection position 440. Therefore, if the environment image collected at the collection position 440 is directly used as the rendered environment image 230, it may cause the resulting The lighting information of the virtual object 250 does not correspond to the user's real peripheral lighting. At this time, even if the acquisition location 440 is closest to the device location 510, the environment image at the acquisition location 440 cannot be selected.
  • an environment image acquired at another acquisition location 420 adjacent to the device location 510 may be selected from a plurality of environment images. For example, since there is no obstacle between the collection position 420 and the device position 510 , the environment image at the collection position 420 that is relatively close to the device position 510 may be selected as the rendered environment image 230 . With exemplary implementations of the present disclosure, an environment image that better matches the user's peripheral illumination information can be selected by considering the occlusion relationship. In this manner, incongruous issues that arise when selecting rendering environment image 230 based on distance alone can be avoided.
  • panoramic image acquisition equipment is installed on a pan-tilt device, and engineers can hold the pan-tilt and/or fix the pan-tilt on a bracket or other equipment and collect it.
  • engineers need to move the pan/tilt to collect environmental images at different acquisition locations. At this time, it is not always possible to ensure that the acquisition angle of the gimbal is consistent.
  • the rendering effect of the virtual object 250 depends on the pixel distribution used for the rendering environment image 230, so different acquisition angles will directly lead to different rendering lighting effects of the virtual object 250.
  • the collection angle of each collected panoramic image needs to be standardized, that is, "reset" to a standard angle (for example, (0,0,0)).
  • standardization processing can be performed on each collected environment image. Specifically, the collection angle associated with each environment image can be obtained, and the environment image can be converted to a standard environment image at a standard angle based on the collection angle. In this way, each environment image can be processed in a unified manner at a unified standard angle and an accurate rendering lighting effect can be obtained.
  • FIG7 shows a block diagram 700 of a process for converting an environment image into a standard environment image according to some implementations of the present disclosure.
  • the purpose of the conversion process is to zero the angle of each panoramic image, that is, to convert the Euler angle of the panoramic image to a predefined standard angle 720 (0,0,0) while keeping the position of the panoramic image unchanged.
  • the panoramic image 3 acquisition device works at the acquisition angle 710, the environment image 220 can be obtained.
  • the acquisition angle 710 may be defined based on a variety of coordinate systems, for example, the coordinate system 712 may be used as a reference coordinate system. According to an exemplary implementation of the present disclosure, the acquisition angle 710 may be represented based on Euler angles (roll, pitch, heading). In the VPS technology, the acquisition position (x, y, z) and acquisition angle (roll, pitch, heading) of the environment image may be determined based on known algorithms, and thus will not be described in detail.
  • each pixel in the environment image 220 may be processed one by one to convert the environment image 220 into the standard environment image 730.
  • the standard pixel corresponding to the pixel in the standard environment image 730 may be determined based on the acquisition angle 710, the spherical coordinates of the pixel in the environment image 220, and the standard angle 720.
  • Figure 8 illustrates a block diagram 800 for a pixel mapping process in accordance with some implementations of the present disclosure.
  • environment image 220 includes pixels 810 (eg, arbitrary pixels) that can be mapped to pixels 820 in standard environment image 730 based on a mathematical transformation.
  • the spherical coordinates of the pixel 820 in the converted standard environment image 730 can be expressed as (long new , lat new ), and the corresponding quaternion can be expressed as Q new .
  • the Euler angle of the environment image 220 can be expressed as (roll, pitch, heading), and the corresponding quaternion can be expressed as Q pano .
  • the quaternion of pixel 810 in environment image 220 may be expressed as Q new -Q pano
  • the spherical coordinate of pixel 810 may be expressed as (long old , lat old ).
  • the color of the pixel 820 in the standard environment image 730 is the color of the pixel 810 in the environment image 220 . It will be understood that the conversion between image coordinates and spherical coordinates of the panoramic image, as well as the conversion between Euler angles and quaternions, can be performed based on coordinate conversion formulas that have been proposed so far and/or will be developed in the future. No further details will be given in this article. Thus, the color of each pixel in the standard environment image 730 can be determined one by one, and the complete standard environment image 730 can be obtained.
  • the standard environment image 730 can be determined based on a simple mathematical transformation. Although the normalization process is described above only using the environment image 220 as an example, similar processing can be performed in advance for each acquired environment image so that the standard environment image of the selected rendering environment image 230 can be directly based on the selected rendering environment image 230 in the subsequent rendering process. for processing. In this way, when the standard environment image 730 is used for the subsequent rendering process, it can be ensured that the rendering lighting effect of the virtual object 250 is more consistent with the user's surrounding real environment.
  • FIG. 9 illustrates a block diagram 900 of a spherical panoramic image 910 in accordance with some implementations of the present disclosure.
  • Each pixel in the spherical panoramic image 910 can be processed one by one based on the principle of the normalization process described above, so as to obtain a corresponding standard environment image stored in a spherical format.
  • the standard environment image associated with the rendering environment image 230 can be used as an ambient light map to render the virtual object 250.
  • the ambient light map can be input to the renderer to obtain a virtual object 250 that matches the user's surrounding environment.
  • the rendering efficiency of using the panoramic image directly as the environment light map may not be ideal in the actual rendering process.
  • the standardized panoramic image can be further converted into a spherical harmonic lighting parameter vector based on the spherical harmonic lighting model, and the vector can be used to perform the rendering process.
  • each normalized standard environment image can be processed in a similar manner, and a corresponding spherical harmonic illumination parameter vector is generated for each standard environment image.
  • the corresponding spherical harmonic lighting parameter vector can be directly called, thereby performing the rendering process with higher performance. In this way, the rendering efficiency can be improved and the delay of the augmented reality application 240 in rendering virtual objects can be reduced.
  • multiple environment images may be processed in advance to extract corresponding spherical harmonic illumination parameter vectors.
  • Multiple spherical harmonic illumination parameter vectors may be stored directly at the terminal device used to run the augmented reality application 240 .
  • a plurality of spherical harmonic illumination parameter vectors may be stored at the server in order to obtain the desired spherical harmonic illumination parameter vector via the network.
  • the collection time points and Locations are collected to index multiple environment images in order to improve the search efficiency of selecting an environment image that matches the current time point and device location from a large number of environment images.
  • multiple environmental images can be indexed according to the order of collection time points and/or the distance between collection locations. This index can be used directly to search for environment images that are most similar to the current time and device location.
  • the number of acquisition positions may be small and/or the distribution of multiple acquisition positions may not evenly cover the real environment 410. In this case, even if the environment image at the acquisition position closest to the device position 510 is used, sometimes a satisfactory rendering lighting effect may not be obtained.
  • new rendering parameters can be generated based on the environment images at two or more acquisition positions near the device position 510.
  • FIG. 10 shows a block diagram 1000 of a process for generating new rendering parameters based on multiple environment images according to some implementations of the present disclosure.
  • Figure 10 describes the generation of new rendering parameters based on multiple environment images using interpolation based on spatial position as an example.
  • collection locations 420 and 440 near device location 510 may be determined based on the index described above.
  • the standard environment image 730 of the environment image 220 at the acquisition position 420 can be obtained, and further the corresponding spherical harmonic illumination parameter vector 1010 can be obtained.
  • the standard environment image 1022 of the environment image 1020 at the acquisition position 440 can be obtained, and further the corresponding spherical harmonic illumination parameter vector 1024 can be obtained.
  • the spatial position-based interpolation 1030 may be determined based on the spherical harmonic illumination parameter vectors 1010 and 1024.
  • the interpolation 1030 takes into account the illumination information at the two collection positions 420 and 440, thereby more accurately simulating the surrounding illumination information at the device position 510.
  • interpolation 1030 may be determined based on a variety of ways.
  • the interpolation between two or more vectors may be determined based on any of nearest neighbor interpolation, linear interpolation, and bilinear interpolation.
  • the interpolation 1030 may be determined based on the average of the individual vectors.
  • the interpolation 1030 may be determined based on a weighted average using the distance between the respective acquisition location and the device location 510 . Then, you can use interpolation 1030 as the new spherical harmonic lighting parameter direction Quantity and used as rendering parameters of the renderer.
  • FIG. 10 only schematically illustrates one example of generating new rendering parameters based on multiple environment images.
  • occlusion relationships may be further taken into consideration, for example, a collection location with no obstacles between the collection location and the device location may be selected based on the method described above. Assuming that there are obstacles between acquisition location 440 and device location 510, images of the environment at other acquisition locations may be selected.
  • FIG. 10 only describes the process of generating new rendering parameters using multiple environment images using space-based interpolation 1030 as an example.
  • time-based interpolation may be obtained in a similar manner.
  • another interpolation can be performed in the time dimension (for example, using linear interpolation) to simulate the peripheral illumination information at a time point that is closer to the current time point.
  • the interpolation value can be determined based on the relevant spherical harmonic lighting parameter vectors of the environment images at 8 a.m. and 12 a.m., and the interpolation value can be used as a new rendering parameter for rendering the virtual object.
  • interpolation values may be determined based on spherical harmonic illumination parameter vectors associated with multiple environmental images within a temporal and/or spatial range, respectively. In this way, interpolation can consider more environmental images in time and/or space, thereby obtaining ambient lighting information that is more consistent with the current time and/or device location.
  • the rendering lighting effect of the virtual object 250 can better match the real environment around the user.
  • FIG. 11A and 11B illustrate block diagrams 1100A and 1100B, respectively, of rendering a virtual object 250 in an augmented reality application 240 in accordance with some implementations of the present disclosure.
  • Figure 1100A schematically illustrates the effects of a user using augmented reality application 240 during the day.
  • the virtual object 250 may be rendered based on daytime environment images collected at a collection location close to the device location.
  • the surface of the virtual object 250 is consistent with the daytime lighting of the real environment, and shows a relatively bright lighting effect.
  • Figure 1100B schematically illustrates the effect of a user using the augmented reality application 240 at night.
  • the nighttime environment collected at a collection location close to the device location can be environment image to render the virtual object 250.
  • the surface of the virtual object 250 is consistent with the nighttime lighting of the real environment, and shows a relatively dim lighting effect.
  • virtual objects may represent other objects.
  • virtual objects in AR-based street view navigation applications, virtual objects can represent virtual signboards, virtual mascots, etc. of shops along the street; in AR-based game applications, virtual objects can represent virtual props, virtual characters, etc.; in AR-based shopping In applications, virtual objects can represent virtual clothes to try on, etc.
  • multiple environment images ie, panoramic images
  • multiple environment images ie, panoramic images
  • the panoramic images collected at different angles can be converted into standard environment images in the standard direction (ie, (0,0,0)). This allows standard environment images to be used directly for rendering without further coordinate transformation.
  • the spherical harmonic illumination parameter vector can be extracted from the normalized standard environment image.
  • spherical harmonic illumination parameter vectors related to multiple environmental images may be interpolated within a time and/or spatial range to obtain illumination parameters that better match the current time and/or device location. In this way, the lighting information around the current point in time and the current device position can be simulated, thereby making the rendered virtual object more consistent with the surrounding real environment.
  • Figure 12 illustrates a flow diagram of a method 1200 for rendering virtual objects in accordance with some implementations of the present disclosure.
  • multiple environment images of the real environment are acquired, and the multiple environment images include images of the real environment collected at multiple acquisition time points respectively.
  • an environment image matching the current time point when the augmented reality application is used is selected from the plurality of environment images as a rendered environment image.
  • the virtual object rendered using the rendering environment image is presented in the augmented reality application.
  • the plurality of environment images are collected at multiple collection locations in the real environment, and selecting to render the environment images further includes: determining whether the terminal device running the augmented reality application is in the real environment. the device location in the device; and based on multiple acquisition locations, select a rendering environment image that matches the device location from multiple environment images.
  • rendering the environment image that matches the device location includes: respectively determining corresponding distances between multiple acquisition locations of the multiple environment images and the device location; and selecting rendering based on comparison of the corresponding distances. environmental images.
  • selecting the rendering environment image based on the comparison of corresponding distances further includes: determining the spatial structure of the real environment based on multiple environment images; determining the relationship between the acquisition position of the rendering environment image and the device position based on the spatial structure. whether an obstacle is present; and in response to determining that no obstacle exists between the acquisition location of the rendering environment image and the location of the device, selecting the rendering environment image.
  • the method further includes: in response to determining that an obstacle exists between the acquisition location of the rendering environment image and the device location, selecting another one of the plurality of environment images whose acquisition location is adjacent to the device location.
  • the environment image is used as the rendering environment image.
  • presenting the virtual object includes: obtaining an acquisition angle associated with the rendering environment image; converting the rendering environment image to a standard environment image at a standard angle based on the acquisition angle; and using the standard environment image as Ambient lightmap to render virtual objects.
  • converting the rendering environment image to the standard environment image includes: for pixels in the rendering environment image, determining the standard environment based on the acquisition angle, the spherical coordinates of the pixel in the rendering environment image, and the standard angle.
  • the standard pixel corresponding to the pixel in the image.
  • rendering a virtual object includes: determining a spherical harmonic lighting parameter vector associated with a standard environment image based on a spherical harmonic lighting model; and rendering the virtual object based on the spherical harmonic lighting parameter vector.
  • the interpolation includes at least any one of the following: position-based interpolation, time-based interpolation.
  • Figure 13 shows a block diagram of an apparatus 1300 for rendering virtual objects in accordance with some implementations of the present disclosure.
  • the device 1300 includes: an acquisition module 1310, configured to acquire multiple environment images of a real environment, where the multiple environment images include images of the real environment collected at multiple acquisition time points respectively; a selection module 1320, configured to acquire based on A plurality of acquisition time points, selecting an environment image that matches the current time point when the augmented reality application is used from the plurality of environment images as a rendering environment image; and a presentation module 1330 configured to present in the augmented reality application A virtual object rendered using a rendering environment image.
  • an acquisition module 1310 configured to acquire multiple environment images of a real environment, where the multiple environment images include images of the real environment collected at multiple acquisition time points respectively
  • a selection module 1320 configured to acquire based on A plurality of acquisition time points, selecting an environment image that matches the current time point when the augmented reality application is used from the plurality of environment images as a rendering environment image
  • a presentation module 1330 configured to present
  • the plurality of environment images are respectively collected at a plurality of collection locations in the real environment
  • the selection module 1320 further includes: a location determination module configured to determine where the augmented reality application is running. a device position of the terminal device in the real environment; and an image selection module configured to select a rendering environment image that matches the device position from a plurality of environment images based on the plurality of acquisition positions.
  • the image selection module includes: a distance determination module configured to determine corresponding distances between multiple collection locations of the multiple environmental images and the device location respectively; and a comparison module configured with An environment image is selected for rendering based on a comparison of corresponding distances.
  • the comparison module further includes: a structure determination module configured to determine the spatial structure of the real environment based on a plurality of environment images; and a detection module configured to determine the spatial structure of the rendered environment image based on the spatial structure. whether there is an obstacle between the acquisition position and the device position; and an obstacle-based selection module configured to respond to determining that there is no obstacle between the acquisition position of the rendering environment image and the device position Object, select Render Environment Image.
  • the obstacle-based selection module is further configured to: in response to determining that an obstacle exists between the acquisition location of the rendered environment image and the device location, select the acquisition location in the plurality of environment images and the device location. Another environment image adjacent to the device location is used as the rendering environment image.
  • the rendering module 1330 includes: an angle acquisition module configured to acquire an acquisition angle associated with the rendering environment image; a conversion module configured to convert the rendering environment image based on the acquisition angle to a standard environment image at a standard angle; and a rendering module configured to use the standard environment image as an environment light map to render the virtual object.
  • the conversion module includes: a pixel determination module configured to determine a standard for a pixel in the rendering environment image based on a collection angle, a spherical coordinate of the pixel in the rendering environment image, and a standard angle.
  • the standard pixel corresponding to the pixel in the environment image.
  • the rendering module includes: a vector determination module configured to determine a spherical harmonic lighting parameter vector associated with the standard environment image based on the spherical harmonic lighting model; and a virtual object rendering module configured Used to render virtual objects based on spherical harmonic lighting parameter vectors.
  • the selection module 1320 is further configured to select another rendering environment image from multiple environment images based on at least any one of the device position and the current time; the vector determination module is further configured to determine another spherical harmonic lighting parameter vector of the other rendering environment image; and the virtual object rendering module further includes: an interpolation-based rendering module, configured to render the virtual image based on the interpolation of the spherical harmonic lighting parameter vector and another spherical harmonic lighting parameter vector.
  • the interpolation includes at least any one of the following: position-based interpolation, time-based interpolation.
  • Figure 14 illustrates a block diagram of a device 1400 capable of implementing various implementations of the present disclosure. It should be understood that the computing device 1400 shown in Figure 14 is exemplary only and should not constitute any limitation on the functionality and scope of the implementations described herein. Figure 14 shows The computing device 1400 may be used to implement the methods described above.
  • computing device 1400 is in the form of a general purpose computing device.
  • Components of computing device 1400 may include, but are not limited to, one or more processors or processing units 1410, memory 1420, storage devices 1430, one or more communication units 1440, one or more input devices 1450, and one or more output devices 1460.
  • the processing unit 1410 may be a real or virtual processor and can perform various processes according to a program stored in the memory 1420 . In a multi-processor system, multiple processing units execute computer-executable instructions in parallel to increase the parallel processing capabilities of computing device 1400 .
  • Computing device 1400 typically includes a plurality of computer storage media. Such media may be any available media that is accessible to computing device 1400, including, but not limited to, volatile and nonvolatile media, removable and non-removable media.
  • Memory 1420 may be volatile memory (e.g., registers, cache, random access memory (RAM)), nonvolatile memory (e.g., read only memory (ROM), electrically erasable programmable read only memory (EEPROM) , flash memory) or some combination thereof.
  • Storage device 1430 may be a removable or non-removable medium and may include machine-readable media such as a flash drive, a magnetic disk, or any other medium that may be capable of storing information and/or data (such as training data for training ) and can be accessed within computing device 1400.
  • machine-readable media such as a flash drive, a magnetic disk, or any other medium that may be capable of storing information and/or data (such as training data for training ) and can be accessed within computing device 1400.
  • Computing device 1400 may further include additional removable/non-removable, volatile/non-volatile storage media.
  • a disk drive may be provided for reading from or writing to a removable, non-volatile disk (eg, a "floppy disk") and for reading from or writing to a removable, non-volatile optical disk. Read or write to optical disc drives.
  • each drive may be connected to the bus (not shown) by one or more data media interfaces.
  • Memory 1420 may include a computer program product 1425 having one or more program modules configured to perform various methods or actions of various implementations of the disclosure.
  • Communication unit 1440 enables communication with other computing devices via a communication medium. Additionally, the functionality of the components of computing device 1400 may be implemented in a single computing cluster or multiple computing machines that are capable of communicating via a communication connection. Thus, computing device 1400 may use a connection to one or more other servers, network personal computers (PCs), or A logical connection to another network node to operate in a networking environment.
  • PCs network personal computers
  • Input device 1450 may be one or more input devices, such as a mouse, keyboard, trackball, etc.
  • Output device 1460 may be one or more output devices, such as a display, speakers, printer, etc.
  • the computing device 1400 may also communicate via the communication unit 1440 with one or more external devices (not shown), such as storage devices, display devices, etc., as needed, and with one or more devices that enable a user to interact with the computing device 1400 Communicate with or with any device (e.g., network card, modem, etc.) that enables computing device 1400 to communicate with one or more other computing devices. Such communication may be performed via an input/output (I/O) interface (not shown).
  • I/O input/output
  • a computer-readable storage medium is provided with computer-executable instructions stored thereon, wherein the computer-executable instructions are executed by a processor to implement the method described above.
  • a computer program product is also provided, the computer program product is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above.
  • a computer program product is provided, a computer program is stored thereon, and when the program is executed by a processor, the method described above is implemented.
  • These computer-readable program instructions may be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus, thereby producing a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, the computer-readable program instructions , resulting in an apparatus that implements the functions/actions specified in one or more blocks in the flowchart and/or block diagram.
  • These computer-readable program instructions can also be stored in a computer-readable storage medium. These instructions cause the computer, programmable data processing device and/or other equipment to work in a specific manner. Therefore, the computer-readable medium storing the instructions includes An article of manufacture that includes a device that implements the functions/acts specified in one or more blocks of the flowchart and/or block diagram Instructions in all aspects.
  • Computer-readable program instructions may be loaded onto a computer, other programmable data processing apparatus, or other equipment, causing a series of operating steps to be performed on the computer, other programmable data processing apparatus, or other equipment to produce a computer-implemented process, Thereby, instructions executed on a computer, other programmable data processing apparatus, or other equipment implement the functions/actions specified in one or more blocks of the flowcharts and/or block diagrams.
  • each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions that contains one or more executable functions for implementing the specified logical functions instruction.
  • the functions noted in the block may occur out of the order noted in the figures. For example, two consecutive blocks may actually execute substantially in parallel, or they may sometimes execute in the reverse order, depending on the functionality involved.
  • each block of the block diagram and/or flowchart illustration, and combinations of blocks in the block diagram and/or flowchart illustration can be implemented by special purpose hardware-based systems that perform the specified functions or acts. , or can be implemented using a combination of specialized hardware and computer instructions.

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Graphics (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Processing Or Creating Images (AREA)

Abstract

提供了用于渲染虚拟对象的方法、装置、设备和介质。在一种方法中，获取真实环境的多个环境图像，多个环境图像包括分别在多个采集时间点采集的真实环境的图像。基于多个采集时间点，从多个环境图像中选择与增强现实应用被使用的当前时间点相匹配的环境图像，以作为渲染环境图像。在增强现实应用中呈现利用渲染环境图像渲染的虚拟对象。利用本公开的示例性实现方式，可以基于当前时间点来实时地选择用于渲染虚拟对象的渲染环境图像并且执行渲染。以此方式，可以使得渲染环境图像的光照信息与当前时间点的真实环境的光照信息相一致。

Description

用于渲染虚拟对象的方法、装置、设备和介质
本申请要求2022年09月21日递交的、标题为“用于渲染虚拟对象的方法、装置、设备和介质”、申请号为202211154080.7的中国发明专利申请的优先权，该申请的全部内容通过引用结合在本申请中。
技术领域
本公开的示例性实现方式总体涉及渲染虚拟对象，特别地涉及在增强现实(AR)应用中基于实时选择的环境图像来渲染虚拟对象的方法、装置、设备和计算机可读存储介质。
背景技术
目前已经开发出了大量增强现实应用，用户可以利用诸如移动终端等设备中的图像采集设备来拍摄真实环境中的景物，并且可以向拍摄的真实环境的视频中添加虚拟对象。例如，可以在期望位置摆放虚拟物体，或者添加可以运动的虚拟人物等等。由于真实环境中的光照可能会不断变化，如何为虚拟对象设置光照渲染参数以使得渲染后的虚拟对象的光照效果与周围的真实环境相一致，成为一个亟待解决的问题。
发明内容
在本公开的第一方面，提供了一种用于在增强现实应用中渲染虚拟对象的方法。在该方法中，获取真实环境的多个环境图像，多个环境图像包括分别在多个采集时间点采集的真实环境的图像。基于多个采集时间点，从多个环境图像中选择与增强现实应用被使用的当前时间点相匹配的环境图像，以作为渲染环境图像。在增强现实应用中呈现利用渲染环境图像渲染的虚拟对象。
在本公开的第二方面，提供了一种用于在增强现实应用中渲染虚拟对象的装置。该装置包括：获取模块，被配置用于获取真实环境的多个环境图像，多个环境图像包括分别在多个采集时间点采集的真实环境的图像；选择模块，被配置用于基于多个采集时间点，从多个环境图像中选择与增强现实应用被使用的当前时间点相匹配的环境图像，以作为渲染环境图像；以及呈现模块，被配置用于在增强现实应用中呈现利用渲染环境图像渲染的虚拟对象。
在本公开的第三方面，提供了一种电子设备。该电子设备包括：至少一个处理单元；以及至少一个存储器，至少一个存储器被耦合到至少一个处理单元并且存储用于由至少一个处理单元执行的指令，指令在由至少一个处理单元执行时使设备执行根据本公开第一方面的方法。
在本公开的第四方面，提供了一种计算机可读存储介质，其上存储有计算机程序，计算机程序在被处理器执行时使处理器实现根据本公开第一方面的方法。
应当理解，本内容部分中所描述的内容并非旨在限定本公开的实现方式的关键特征或重要特征，也不用于限制本公开的范围。本公开的其他特征将通过以下的描述而变得容易理解。
附图说明
在下文中，结合附图并参考以下详细说明，本公开各实现方式的上述和其他特征、优点及方面将变得更加明显。在附图中，相同或相似的附图标记表示相同或相似的元素，其中：
图1示出了本公开的实现方式可以在其中使用的应用环境的示例；
图2示出了根据本公开的一些实现方式的用于在增强现实应用中渲染虚拟对象的框图；
图3示出了根据本公开的一些实现方式的分别在多个采集时间点采集的多个环境图像的框图；
图4示出了根据本公开的一些实现方式的分别在多个采集位置采 集的多个环境图像的框图；
图5示出了根据本公开的一些实现方式的用于基于运行增强现实应用的终端设备的设备位置与采集位置的比较来选择渲染环境图像的过程的框图；
图6示出了根据本公开的一些实现方式的用于基于遮挡关系来选择渲染环境图像的过程的框图；
图7示出了根据本公开的一些实现方式的用于将环境图像转换为标准环境图像的过程的框图；
图8示出了根据本公开的一些实现方式的用于映射像素的过程的框图；
图9示出了根据本公开的一些实现方式的球形全景图像的框图；
图10示出了根据本公开的一些实现方式的用于基于多个环境图像来生成新的渲染参数的过程的框图；
图11A和11B分别示出了根据本公开的一些实现方式的在增强现实应用中呈现虚拟对象的框图；
图12示出了根据本公开的一些实现方式的用于渲染虚拟对象的方法的流程图；
图13示出了根据本公开的一些实现方式的用于渲染虚拟对象的装置的框图；以及
图14示出了能够实施本公开的多个实现方式的设备的框图。
具体实施方式
下面将参照附图更详细地描述本公开的实现方式。虽然附图中示出了本公开的某些实现方式，然而应当理解的是，本公开可以通过各种形式来实现，而且不应该被解释为限于这里阐述的实现方式，相反，提供这些实现方式是为了更加透彻和完整地理解本公开。应当理解的是，本公开的附图及实现方式仅用于示例性作用，并非用于限制本公开的保护范围。
在本公开的实现方式的描述中，术语“包括”及其类似用语应当 理解为开放性包含，即“包括但不限于”。术语“基于”应当理解为“至少部分地基于”。术语“一个实现方式”或“该实现方式”应当理解为“至少一个实现方式”。术语“一些实现方式”应当理解为“至少一些实现方式”。下文还可能包括其他明确的和隐含的定义。如本文中所使用的，术语“模型”可以表示各个数据之间的关联关系。例如，可以基于目前已知的和/或将在未来开发的多种技术方案来获取上述关联关系。
可以理解的是，本技术方案所涉及的数据(包括但不限于数据本身、数据的获取或使用)应当遵循相应法律法规及相关规定的要求。
可以理解的是，在使用本公开各实施例公开的技术方案之前，均应当根据相关法律法规通过适当的方式对本公开所涉及个人信息的类型、使用范围、使用场景等告知用户并获得用户的授权。
例如，在响应于接收到用户的主动请求时，向用户发送提示信息，以明确地提示用户，其请求执行的操作将需要获取和使用到用户的个人信息。从而，使得用户可以根据提示信息来自主地选择是否向执行本公开技术方案的操作的电子设备、应用程序、服务器或存储介质等软件或硬件提供个人信息。
作为一种可选的但非限制性的实现方式，响应于接收到用户的主动请求，向用户发送提示信息的方式，例如可以是弹出窗口的方式，弹出窗口中可以以文字的方式呈现提示信息。此外，弹出窗口中还可以承载供用户选择“同意”或“不同意”向电子设备提供个人信息的选择控件。
可以理解的是，上述通知和获取用户授权过程仅是示意性的，不对本公开的实现方式构成限定，其他满足相关法律法规的方式也可应用于本公开的实现方式中。
示例环境
目前已经开发出了大量AR应用，参见图1描述根据本公开的一个示例性实现方式的AR应用环境。图1示出了本公开的实现方式可 以在其中使用的应用环境的示例100。可以在终端设备处运行AR应用，并且用户可以手持该终端设备，并且使用运行在终端设备处的增强现实应用110拍摄真实环境中的景物，并且可以向拍摄的真实环境的视频中添加虚拟对象120(例如，添加立方体雕塑)。此时，由于用户手持该终端设备，可以认为在真实环境中的终端设备的设备位置与用户位置相同。
通常而言，AR应用环境中的光照将会随着时间和设备位置移动而变化。图1示出了使用预先设置的光照渲染数据来渲染的虚拟对象120，在图1中出现了渲染后的虚拟对象120的光照效果与周边真实环境不一致。具体地，预先设置的光照渲染数据是针对日间场景设置的，并且利用该光照渲染数据获得的虚拟对象120的光照效果较为明亮。当用户在夜间使用增强现实应用110时，真实环境的光照较为昏暗。此时，在向AR场景中添加虚拟对象120时，过于明亮的虚拟对象120与周围的昏暗环境并不协调。这导致增强现实应用110的视觉效果并不真实。
将会理解，真实环境可能会存在复杂的光照信息。例如，光照大致可以分为直接光照和环境光照，并且在真实环境中可能会存在多个直接光源。根据一个技术方案，可以基于实时光照估计技术来处理拍摄的真实环境的图像，也即，可以针对真实环境的图像进行光照分析以便获得真实光照。然而，该技术方案只能获得真实环境的整体光照强度或者最显著直接光照，然而准确性并不令人满意。此外，实时光照估计技术产生较大的计算资源开销，因而难以在诸如移动终端的便携式计算设备中实现。此时，如何以更为方便并且有效的方式确定虚拟对象的周边光照信息并且进行渲染，成为一个亟待解决的问题。
渲染虚拟对象的概要过程
为了解决上述技术方案中的不足，根据本公开的一个示例性实现方式，提出了一种用于在增强现实应用中渲染虚拟对象的方法。概括而言，可以获取真实环境的多个环境图像，并且将这些环境图像作为 渲染虚拟对象的候选渲染环境图像。在此，多个环境图像可以是分别在多个采集时间点采集的真实环境的图像。例如，可以分别在一天中的不同时间点(诸如，日间和夜间)采集多个环境图像。
在本公开的上下文中，可以在多种终端设备处运行AR应用。例如，可以在常规的移动终端设备(包括但不限于，移动电话、移动平板计算设备、移动式笔记本计算设备，等等)上运行AR应用。又例如，可以在穿戴式终端设备(包括但不限于，具有计算功能的眼镜设备、头盔设备，等等)上运行AR应用。再例如，可以在显示功能与计算功能相分离的计算设备(例如，利用便携式计算设备来运行AR应用，并且利用与该便携式计算设备通信的眼镜设备来显示AR应用的界面)上运行AR应用。
参见图2描述根据本公开的一个示例性实现方式的概要，该图2示出了根据本公开的一些实现方式的用于在增强现实应用中渲染虚拟对象的框图200。如图2所示，可以获取分别在多个采集时间的采集的多个环境图像。例如，环境图像220的采集时间点210可以是日间，…，环境图像222的采集时间点212可以是夜间。
可以确定用户使用增强现实应用240的当前时间点244。进一步可以基于当前时间点244与多个采集时间点210、…、212的比较，从多个环境图像220、…、222中选择匹配于当前时间点244的环境图像222来作为渲染环境图像230。具体地，如果用户在日间使用增强现实应用240，则可以选择在日间采集的环境图像220来作为渲染环境图像230并且进行渲染。此时，渲染后的虚拟对象250将呈现日间的明亮光照效果。又例如，如果用户在夜间使用增强现实应用240，则可以选择在夜间采集的环境图像222来作为渲染环境图像230。此时，渲染后的虚拟对象250将呈现夜间的昏暗光照效果。
如图2所示，当前时间点244为夜间，因而可以选择在夜间的环境图像222来作为渲染环境图像230。可以利用渲染环境图像230来渲染虚拟对象250，并且在增强现实应用240中呈现渲染的虚拟对象250。此时，可以呈现如图2所示的夜间光照效果。
利用本公开的示例性实现方式，可以基于当前时间点244来实时地选择用于渲染虚拟对象250的渲染环境图像230并且执行渲染。以此方式，可以使得渲染环境图像230的光照信息与当前时间点244的真实环境的光照信息相一致，进而使得渲染获得的虚拟对象250可以与周边的真实环境相一致。
渲染虚拟对象的详细过程
已经参见图2描述了根据本公开的一个示例性实现方式的概要，在下文中，将详细描述用于渲染虚拟对象的更多细节。参见图3描述有关环境图像的更多信息，该图3示出了根据本公开的一些实现方式的分别在多个采集时间点采集的多个环境图像的框图300。如图3所示，可以在一天中的不同时间点采集多个环境图像，例如，可以按照预定时间间隔(例如，2小时、4小时，等等)，在时间点T1采集环境图像220，…，并且在时间点TN采集环境图像222。
根据本公开的一个示例性实现方式，可以将多个环境图像直接存储在用于运行增强现实应用240的终端设备处。备选地和/或附加地，可以多个环境图像存储在提供AR服务的服务器处，以便经由网络来获取期望的环境图像。
将会理解，尽管上文示出了在一天中的不同时间点采集环境图像的过程，备选地和/或附加地，在室外应用场景下还可以在不同季节(例如，春夏秋冬)采集不同的环境图像。以此方式，可以基于用户使用增强现实应用240的季节和时间点两者，来选择更为匹配于具体使用时间的环境图像并且执行渲染。由此，可以进一步提高虚拟对象250的渲染效果，并且使得虚拟对象250的光照信息更加匹配于周边真实环境。
在增强现实应用中通常基于视觉定位***(VisualPositioning System，缩写VPS)来进行定位。为了提高VPS的定位精度，需要在不同时间点和不同位置预先采集真实环境的大量图像。根据本公开的一个示例性实现方式，并不需要额外步骤来采集上文描述的多个环境 图像220、…、222，而是可以将预先采集的用于VPS目的的各个环境图像来直接作为用于渲染目的的环境图像。以此方式，并不会增加数据采集的工作负荷，而是可以重用已经采集的环境图像来服务于渲染虚拟对象250的目的。
将会理解，增强现实应用240中的虚拟对象250的光照信息将会随着用户在真实环境中的位置而变化，因而在渲染虚拟对象250时还可以考虑设备位置。根据本公开的一个示例性实现方式，可以首先基于当前时间点244来从多个环境图像中找到与当前时间点244相匹配的一组环境图像。进一步，可以在该组环境图像中找到更加符合设备位置的环境图像，来作为渲染环境图像230。具体地，可以获取分别在真实环境中的多个采集位置处采集的多个环境图像。
参见图4描述有关采集位置的更多信息。图4示出了根据本公开的一些实现方式的分别在多个采集位置采集的多个环境图像的框图400。如图4所示，真实环境410可以包括多个采集位置420、430、…、以及440，并且可以在每个采集位置处采集环境图像。例如，环境图像220可以是在采集位置420处采集的，并且其他环境图像可以是分别在采集位置430和440处采集的，等等。
根据本公开的一个示例性实现方式，在时间点T1，可以分别在多个采集位置处采集多个环境图像；…；在时间点TN，可以分别在多个采集位置采集多个环境图像。将会理解，本公开并不限制在每个时间点所采集环境图像的数量，也并不限制在每个时间点所选取的采集位置是否相同，而是可以直接使用出于VPS目的而采集的多个环境图像。可以按照采集时间和采集位置来管理上述多个环境图像。
根据本公开的一个示例性实现方式，可以确定用户在真实环境410中的设备位置，并且基于设备位置与多个采集位置的比较，来从多个环境图像中选择与设备位置相匹配的渲染环境图像230。参见图5描述选择渲染环境图像230的更多细节，该图5示出了根据本公开的一些实现方式的用于基于运行增强现实应用的终端设备的设备位置510与采集位置的比较来选择渲染环境图像230的过程的框图500。
如图5所示，假设用户位于真实环境410中的设备位置510，可以分别获取设备位置510与各个采集位置420、430、…、以及440之间的距离。进一步，可以基于各个距离的比较来选择在最靠近设备位置510的采集位置处采集的环境图像。在图5中，假设设备位置510与采集位置420和440之间的距离分别为520和530，基于比较可知距离520大于距离530，此时可以选择在采集位置440处采集的环境图像来作为渲染环境图像230。
利用本公开的示例性实现方式，通过选择在最为靠近设备位置510的采集位置440处采集的环境图像，可以利用最为近似于用户在设备位置510所看到的周边环境的环境图像来渲染虚拟对象250。以此方式，可以使得虚拟对象250的光照效果更加匹配于在设备位置510处看到的环境光照，进而提高增强现实应用240的视觉效果。
将会理解，在真实环境410中可能会存在诸如墙体、植物等物体的遮挡，这些遮挡将会影响真实环境410中的光照效果。根据本公开的一个示例性实现方式，在选择渲染环境图像230时，可以进一步考虑遮挡关系。
图6示出了根据本公开的一些实现方式的基于遮挡关系来选择渲染环境图像230的过程的框图600。具体地，可以基于多个环境图像来确定真实环境410的空间结构。可以基于目前已知的和/或将在未来开发的三维重建技术来获取真实环境410的空间结构，在本文中不再赘述。进一步，可以利用该空间结构来确定渲染环境图像的采集位置与设备位置510之间是否存在障碍物。如图6所示，空间结构指示真实环境410中存在障碍物610，并且该障碍物610位于设备位置510和采集位置440之间。
将会理解，障碍物610可以导致设备位置510处的用户并不能直接看到采集位置440，因而如果直接将在采集位置440处采集的环境图像作为渲染环境图像230，则可能会导致渲染所得的虚拟对象250的光照信息并不符合用户的真实周边光照。此时，即使采集位置440最为靠近设备位置510，也不能选择在采集位置440处的环境图像。
根据本公开的一个示例性实现方式，可以从多个环境图像中选择在与设备位置510相邻的另一采集位置420处采集的环境图像。例如，由于在采集位置420和设备位置510之间不存在障碍物，可以选择与设备位置510较为靠近的采集位置420处的环境图像来作为渲染环境图像230。利用本公开的示例性实现方式，通过考虑遮挡关系可以选择更加匹配于用户的周边光照信息的环境图像。以此方式，可以避免单独基于距离来选择渲染环境图像230时出现的不协调问题。
将会理解，在采集环境图像的过程中，工程师可以使用全景图像采集设备来在不同时间点和不同位置处采集多个环境图像全景图像。通常而言，全景图像采集设备被安装在云台设备上，并且工程师可以手持云台和/或将云台固定在支架等设备处并且进行采集。在采集过程中，工程师需要移动云台以便在不同采集位置采集环境图像。此时，并不能始终确保云台的采集角度是一致的。
将会理解，采集角度不同可以导致周边景物在采集的全景图像中像素分布有所不同。虚拟对象250的渲染效果依赖于用作渲染环境图像230的像素分布，因而不同的采集角度将会直接导致虚拟对象250的不同渲染光照效果。此时，需要将采集到的各个全景图像的采集角度进行标准化处理，也即将其“归零”至标准角度(例如，(0,0,0))。
根据本公开的一个示例性实现方式，可以针对采集的各个环境图像执行标准化处理。具体地，可以获取与各个环境图像相关联的采集角度，并且基于采集角度来将环境图像转换至标准角度下的标准环境图像。以此方式，可以在统一的标准角度下，以统一方式处理各个环境图像并且获得准确的渲染光照效果。
图7示出了根据本公开的一些实现方式的用于将环境图像转换为标准环境图像的过程的框图700。转换过程的目的在于对各个全景图像进行角度归零，也即在保持全景图像的位置不变的情况下，将全景图像的欧拉角转换至预定义的标准角度720(0,0,0)。如图7所示，当全景图3采集设备以采集角度710工作时，可以获得环境图像220。
根据本公开的一个示例性实现方式，可以基于多种坐标系来定义采集角度710，例如，可以基于坐标系712来作为参考坐标系。根据本公开的一个示例性实现方式，可以基于欧拉角(roll,pitch,heading)来表示采集角度710。在VPS技术中可以基于已知算法来确定环境图像的采集位置(x,y,z)和采集角度(roll,pitch,heading)，因而不再赘述。
进一步，可以逐一处理环境图像220中的各个像素，以便将环境图像220转换至标准环境图像730。具体地，可以基于采集角度710、像素在环境图像220中的球面坐标、以及标准角度720，确定标准环境图像730中的与像素相对应的标准像素。在下文中，将参见图8描述标准化过程的更多细节，该图8示出了根据本公开的一些实现方式的用于像素映射过程的框图800。
如图8所示，环境图像220包括像素810(例如，任意像素)，可以基于数学变换来将像素810映射至标准环境图像730中的像素820。转换后的标准环境图像730中的像素820的球面坐标可以表示为(longnew,latnew)，对应的四元数可以表示为Qnew。对于转化前的环境图像220而言，该环境图像220的欧拉角可以表示为(roll,pitch,heading)，并对应的四元数可以表示为Qpano。环境图像220中的像素810的四元数可以表示为Qnew-Qpano，并且像素810的球面坐标可以表示为(longold,latold)。
基于映射关系可知，标准环境图像730中的像素820的颜色为环境图像220中的像素810的颜色。将会理解，可以基于目前已经提出的和/或将在未来开发的坐标转换公式，执行全景图像的图像坐标和球面坐标之间的转换、以及欧拉角和四元数之间的转换，在本文中将不再赘述。由此，可以逐一地确定标准环境图像730中的每个像素的颜色，进而获得完整的标准环境图像730。
利用本公开的示例性实现方式，可以基于简单的数学变换即可确定标准环境图像730。尽管上文仅以环境图像220作为示例描述了标准化过程，可以预先针对获取的每个环境图像进行类似的处理，以便在后续渲染过程中直接基于所选择渲染环境图像230的标准环境图像 进行处理。以此方式，当利用标准环境图像730进行后续渲染过程时，可以确保虚拟对象250的渲染光照效果更为匹配于用户的周边真实环境。
将会理解，尽管上文以矩形格式全景图像作为环境图像的示例描述了针对环境图像的处理，备选地和/或附加地，还可以以图9所示球形格式存储全景图像。该图9示出了根据本公开的一些实现方式的球形全景图像910的框图900。可以基于上文描述的标准化过程的原理来逐一处理球形全景图像910中的每个像素，以便获得相应的以球形格式存储的标准环境图像。
根据本公开的一个示例性实现方式，在已经选择最为匹配于当前时间和设备位置510的渲染环境图像230之后，可以将该渲染环境图像230的相关标准环境图像作为环境光贴图，以渲染虚拟对象250。利用本公开的示例性实现方式，可以向渲染器输入环境光贴图，进而获得与用户的周边环境相匹配的虚拟对象250。
将会理解，在实际渲染过程中直接使用全景图像作为环境光贴图的渲染效率可能并不理想。此时，可基于球谐光照模型，来将标准化后的全景图像进一步转化为球谐光照参数向量，并且利用该向量来执行渲染过程。具体地，可以以类似方式来处理标准化后的每个标准环境图像，并且为每个标准环境图像生成相应的球谐光照参数向量。利用本公开的示例性实现方式，在后期渲染过程中，可以直接调用相应的球谐光照参数向量，从而以更高的性能执行渲染过程。以此方式，可以提高渲染效率进而降低增强现实应用240在渲染虚拟对象方面的延迟。
根据本公开的一个示例性实现方式，可以预先对多个环境图像进行处理，以便提取相应的球谐光照参数向量。可以将多个球谐光照参数向量直接存储在用于运行增强现实应用240的终端设备处。备选地和/或附加地，可以将多个球谐光照参数向量存储在服务器处，以便经由网络来获取期望的球谐光照参数向量。
根据本公开的一个示例性实现方式，可以分别按照采集时间点和 采集位置来为多个环境图像建立索引，以便提高从大量环境图像中选择匹配于当前时间点和设备位置的环境图像的搜索效率。例如，可以按照采集时间点的先后顺序和/或采集位置之间的距离来索引多个环境图像。可以直接基于该索引来搜索与当前时间和设备位置最为类似的环境图像。
将会理解，采集位置的数量可能较少和/或多个采集位置的分布可能并不能均匀地覆盖真实环境410。此时，即使使用与设备位置510最为靠近的采集位置处的环境图像，有时可能也不能获得满意的渲染光照效果。为了进一步提高渲染光照效果，可以基于设备位置510附近的两个或者更多采集位置处的环境图像，来生成新的渲染参数。在下文中，参见图10描述更多细节，该图10示出了根据本公开的一些实现方式的用于基于多个环境图像来生成新的渲染参数的过程的框图1000。
图10以基于空间位置的插值为示例描述了基于多个环境图像来生成新的渲染参数。如图10所示，可以基于上文描述的索引来确定设备位置510附近的采集位置420和440。例如，可以获取采集位置420处的环境图像220的标准环境图像730，进一步可以获取相应的球谐光照参数向量1010。类似地，可以获取采集位置440处的环境图像1020的标准环境图像1022，进一步可以获取相应的球谐光照参数向量1024。此时，可以基于球谐光照参数向量1010和1024来确定基于空间位置的插值1030。相对于球谐光照参数向量1010和1024而言，该插值1030考虑了两个采集位置420和440处的光照信息，进而可以更为准确地模拟设备位置510处的周边光照信息。
根据本公开的一个示例性实现方式，可以基于多种方式来确定插值1030。例如，可以基于最近邻插值、线性插值以及双线性插值中的任一项来确定两个或者更多向量之间的插值。在一个简单示例中，可以基于各个向量的平均来确定插值1030。备选地和/或附加地，可以利用各个采集位置与设备位置510之间的距离，基于加权平均的方式来确定插值1030。继而，可以使用插值1030作为新的球谐光照参数向 量，并且作为渲染器的渲染参数。
将会理解，图10仅仅示意性示出了基于多个环境图像来生成新的渲染参数的一个示例。备选地和/或附加地，可以进一步考虑遮挡关系，例如，可以基于上文描述的方法来选择在采集位置和设备位置之间不存在障碍物的采集位置。假设在采集位置440和设备位置510之间存在障碍物，则可以选择其他采集位置处的环境图像。
将会理解，图10仅以基于空间的插值1030为示例描述了利用多个环境图像来生成新的渲染参数的过程，备选地和/或附加地，可以以类似方式来获取基于时间的插值。例如，对于不同时间点的球谐光照参数向量，可在时间维度上再进行一次插值(例如，使用线性插值)，以便模拟与当时间点更为接近的时间点的周边光照信息。假设目前仅存在上午8点和12点的环境图像，然而当前时间点为上午10点。此时，可以基于上午8点和12点的环境图像的相关球谐光照参数向量来确定插值，可以将该插值作为渲染虚拟对象的新的渲染参数。
利用本公开的示例性实现方式，可以分别在时间和/或空间范围内，基于多个环境图像相关的球谐光照参数向量确定插值。以此方式，插值可以考虑在时间和/或空间范围内的更多环境图像，进而获得更为匹配于当前时间和/或设备位置的周边光照信息。当利用插值执行渲染时，虚拟对象250的渲染光照效果可以更加匹配于用户周边的真实环境。
上文已经描述了用于在增强现实应用240中渲染虚拟对象250的具体过程。在下文中，参见图11A和11B提供具体渲染效果。该图11A和11B分别示出了根据本公开的一些实现方式的在增强现实应用240中呈现虚拟对象250的框图1100A和1100B。具体地，图1100A示意性示出了用户在日间使用增强现实应用240的效果。此时，可以基于在靠近设备位置的采集位置采集的日间环境图像，来渲染虚拟对象250。如图11A所示，此时虚拟对象250的表面与真实环境的日间光照相一致，并且表现出较为明亮的光照效果。
进一步，图1100B示意性示出了用户在夜间使用增强现实应用240的效果。此时，可以基于在靠近设备位置的采集位置采集的夜间环 境图像，来渲染虚拟对象250。如图11B所示，此时虚拟对象250的表面与真实环境的夜间光照相一致，并且表现出较为昏暗的光照效果。
将会理解，尽管上文以立方体作为虚拟对象的具体示例描述了渲染过程。根据本公开的一个示例性实现方式，虚拟对象可以表示其他物体。例如，在基于AR的街景导航应用中，虚拟对象可以表示沿街店铺的虚拟招牌、虚拟吉祥物等；在基于AR的游戏应用中，虚拟对象可以表示虚拟道具、虚拟人物等；在基于AR的购物应用中，虚拟对象可以表示试穿的虚拟衣物等。
利用本公开的示例性实现方式，可以重用出于VPS目的而采集的多个环境图像(也即全景图像)，并且在无需额外数据采集开销的情况下，改善虚拟对象渲染的视觉效果。进一步，通过将采集到的全景图像进行标准化处理，可以将在不同角度采集的全景图像转换至标准方向(即，(0,0,0))下的标准环境图像。由此，可以在无需进一步坐标转换的情况下直接将标准环境图像用于渲染。
备选地和/或附加地，为了进一步提高渲染性能，可以从标准化处理后的标准环境图像提取球谐光照参数向量。备选地和/或附加地，可以在时间和/或空间范围内对多个环境图像相关的球谐光照参数向量进行插值处理，以便获得更加匹配于当前时间和/或设备位置的光照参数。以此方式，可以模拟当前时间点和当前设备位置周围的光照信息，进而使得渲染的虚拟对象与周边真实环境更加一致。
示例过程
图12示出了根据本公开的一些实现方式的用于渲染虚拟对象的方法1200的流程图。具体地，在框1210处，获取真实环境的多个环境图像，多个环境图像包括分别在多个采集时间点采集的真实环境的图像。在框1220处，基于多个采集时间点，从多个环境图像中选择与增强现实应用被使用的当前时间点相匹配的环境图像，以作为渲染环境图像。在框1230处，在增强现实应用中呈现利用渲染环境图像渲染的虚拟对象。
根据本公开的一个示例性实现方式，多个环境图像是分别在真实环境中的多个采集位置处采集的，以及选择渲染环境图像进一步包括：确定运行所述增强现实应用的终端设备在真实环境中的设备位置；以及基于多个采集位置，从多个环境图像中选择与设备位置相匹配的渲染环境图像。
根据本公开的一个示例性实现方式，与设备位置相匹配的渲染环境图像包括：分别确定多个环境图像的多个采集位置与设备位置之间的相应距离；以及基于相应距离的比较来选择渲染环境图像。
根据本公开的一个示例性实现方式，基于相应距离的比较来选择渲染环境图像进一步包括：基于多个环境图像确定真实环境的空间结构；基于空间结构确定渲染环境图像的采集位置与设备位置之间是否存在障碍物；以及响应于确定渲染环境图像的采集位置与设备位置之间不存在障碍物，选择渲染环境图像。
根据本公开的一个示例性实现方式，该方法进一步包括：响应于确定渲染环境图像的采集位置与设备位置之间存在障碍物，选择多个环境图像中的采集位置与设备位置相邻的另一环境图像以作为渲染环境图像。
根据本公开的一个示例性实现方式，呈现虚拟对象包括：获取与渲染环境图像相关联的采集角度；基于采集角度，将渲染环境图像转换至标准角度下的标准环境图像；以及将标准环境图像作为环境光贴图以渲染虚拟对象。
根据本公开的一个示例性实现方式，将渲染环境图像转换至标准环境图像包括：针对渲染环境图像中的像素，基于采集角度、像素在渲染环境图像中的球面坐标、以及标准角度，确定标准环境图像中的与像素相对应的标准像素。
根据本公开的一个示例性实现方式，渲染虚拟对象包括：基于球谐光照模型来确定与标准环境图像相关联的球谐光照参数向量；以及基于球谐光照参数向量来渲染虚拟对象。
根据本公开的一个示例性实现方式，利用球谐光照参数向量来渲 染虚拟对象包括：基于设备位置和当前时间中的至少任一项，从多个环境图像中选择另一渲染环境图像；确定另一渲染环境图像的另一球谐光照参数向量；以及基于球谐光照参数向量和另一球谐光照参数向量的插值，渲染虚拟图像。
根据本公开的一个示例性实现方式，插值包括以下至少任一项：基于位置的插值、基于时间的插值。
示例装置和设备
图13示出了根据本公开的一些实现方式的用于渲染虚拟对象的装置1300的框图。该装置1300包括：获取模块1310，被配置用于获取真实环境的多个环境图像，多个环境图像包括分别在多个采集时间点采集的真实环境的图像；选择模块1320，被配置用于基于多个采集时间点，从多个环境图像中选择与增强现实应用被使用的当前时间点相匹配的环境图像，以作为渲染环境图像；以及呈现模块1330，被配置用于在增强现实应用中呈现利用渲染环境图像渲染的虚拟对象。
根据本公开的一个示例性实现方式，多个环境图像是分别在真实环境中的多个采集位置处采集的，以及选择模块1320进一步包括：位置确定模块，被配置用于确定运行增强现实应用的终端设备在真实环境中的设备位置；以及图像选择模块，被配置用于基于多个采集位置，从多个环境图像中选择与设备位置相匹配的渲染环境图像。
根据本公开的一个示例性实现方式，图像选择模块包括：距离确定模块，被配置用于分别确定多个环境图像的多个采集位置与设备位置之间的相应距离；以及比较模块，被配置用于基于相应距离的比较来选择渲染环境图像。
根据本公开的一个示例性实现方式，比较模块进一步包括：结构确定模块，被配置用于基于多个环境图像确定真实环境的空间结构；检测模块，被配置用于基于空间结构确定渲染环境图像的采集位置与设备位置之间是否存在障碍物；以及基于障碍物的选择模块，被配置用于响应于确定渲染环境图像的采集位置与设备位置之间不存在障碍 物，选择渲染环境图像。
根据本公开的一个示例性实现方式，基于障碍物的选择模块进一步被配置用于：响应于确定渲染环境图像的采集位置与设备位置之间存在障碍物，选择多个环境图像中的采集位置与设备位置相邻的另一环境图像以作为渲染环境图像。
根据本公开的一个示例性实现方式，呈现模块1330包括：角度获取模块，被配置用于获取与渲染环境图像相关联的采集角度；转换模块，被配置用于基于采集角度，将渲染环境图像转换至标准角度下的标准环境图像；以及渲染模块，被配置用于将标准环境图像作为环境光贴图以渲染虚拟对象。
根据本公开的一个示例性实现方式，转换模块包括：像素确定模块，被配置用于针对渲染环境图像中的像素，基于采集角度、像素在渲染环境图像中的球面坐标、以及标准角度，确定标准环境图像中的与像素相对应的标准像素。
根据本公开的一个示例性实现方式，渲染模块包括：向量确定模块，被配置用于基于球谐光照模型来确定与标准环境图像相关联的球谐光照参数向量；以及虚拟对象渲染模块，被配置用于基于球谐光照参数向量来渲染虚拟对象。
根据本公开的一个示例性实现方式，选择模块1320进一步被配置用于基于设备位置和当前时间中的至少任一项，从多个环境图像中选择另一渲染环境图像；向量确定模块进一步被配置用于确定另一渲染环境图像的另一球谐光照参数向量；以及虚拟对象渲染模块进一步包括：基于插值的渲染模块，被配置用于基于球谐光照参数向量和另一球谐光照参数向量的插值，渲染虚拟图像。
根据本公开的一个示例性实现方式，插值包括以下至少任一项：基于位置的插值、基于时间的插值。
图14示出了能够实施本公开的多个实现方式的设备1400的框图。应当理解，图14所示出的计算设备1400仅仅是示例性的，而不应当构成对本文所描述的实现方式的功能和范围的任何限制。图14所示出 的计算设备1400可以用于实现上文描述的方法。
如图14所示，计算设备1400是通用计算设备的形式。计算设备1400的组件可以包括但不限于一个或多个处理器或处理单元1410、存储器1420、存储设备1430、一个或多个通信单元1440、一个或多个输入设备1450以及一个或多个输出设备1460。处理单元1410可以是实际或虚拟处理器并且能够根据存储器1420中存储的程序来执行各种处理。在多处理器***中，多个处理单元并行执行计算机可执行指令，以提高计算设备1400的并行处理能力。
计算设备1400通常包括多个计算机存储介质。这样的介质可以是计算设备1400可访问的任何可以获得的介质，包括但不限于易失性和非易失性介质、可拆卸和不可拆卸介质。存储器1420可以是易失性存储器(例如寄存器、高速缓存、随机访问存储器(RAM))、非易失性存储器(例如，只读存储器(ROM)、电可擦除可编程只读存储器(EEPROM)、闪存)或它们的某种组合。存储设备1430可以是可拆卸或不可拆卸的介质，并且可以包括机器可读介质，诸如闪存驱动、磁盘或者任何其他介质，其可以能够用于存储信息和/或数据(例如用于训练的训练数据)并且可以在计算设备1400内被访问。
计算设备1400可以进一步包括另外的可拆卸/不可拆卸、易失性/非易失性存储介质。尽管未在图14中示出，可以提供用于从可拆卸、非易失性磁盘(例如“软盘”)进行读取或写入的磁盘驱动和用于从可拆卸、非易失性光盘进行读取或写入的光盘驱动。在这些情况中，每个驱动可以由一个或多个数据介质接口被连接至总线(未示出)。存储器1420可以包括计算机程序产品1425，其具有一个或多个程序模块，这些程序模块被配置为执行本公开的各种实现方式的各种方法或动作。
通信单元1440实现通过通信介质与其他计算设备进行通信。附加地，计算设备1400的组件的功能可以以单个计算集群或多个计算机器来实现，这些计算机器能够通过通信连接进行通信。因此，计算设备1400可以使用与一个或多个其他服务器、网络个人计算机(PC)或者 另一个网络节点的逻辑连接来在联网环境中进行操作。
输入设备1450可以是一个或多个输入设备，例如鼠标、键盘、追踪球等。输出设备1460可以是一个或多个输出设备，例如显示器、扬声器、打印机等。计算设备1400还可以根据需要通过通信单元1440与一个或多个外部设备(未示出)进行通信，外部设备诸如存储设备、显示设备等，与一个或多个使得用户与计算设备1400交互的设备进行通信，或者与使得计算设备1400与一个或多个其他计算设备通信的任何设备(例如，网卡、调制解调器等)进行通信。这样的通信可以经由输入/输出(I/O)接口(未示出)来执行。
根据本公开的示例性实现方式，提供了一种计算机可读存储介质，其上存储有计算机可执行指令，其中计算机可执行指令被处理器执行以实现上文描述的方法。根据本公开的示例性实现方式，还提供了一种计算机程序产品，计算机程序产品被有形地存储在非瞬态计算机可读介质上并且包括计算机可执行指令，而计算机可执行指令被处理器执行以实现上文描述的方法。根据本公开的示例性实现方式，提供了一种计算机程序产品，其上存储有计算机程序，程序被处理器执行时实现上文描述的方法。
这里参照根据本公开实现的方法、装置、设备和计算机程序产品的流程图和/或框图描述了本公开的各个方面。应当理解，流程图和/或框图的每个方框以及流程图和/或框图中各方框的组合，都可以由计算机可读程序指令实现。
这些计算机可读程序指令可以提供给通用计算机、专用计算机或其他可编程数据处理装置的处理单元，从而生产出一种机器，使得这些指令在通过计算机或其他可编程数据处理装置的处理单元执行时，产生了实现流程图和/或框图中的一个或多个方框中规定的功能/动作的装置。也可以把这些计算机可读程序指令存储在计算机可读存储介质中，这些指令使得计算机、可编程数据处理装置和/或其他设备以特定方式工作，从而，存储有指令的计算机可读介质则包括一个制造品，其包括实现流程图和/或框图中的一个或多个方框中规定的功能/动作的 各个方面的指令。
可以把计算机可读程序指令加载到计算机、其他可编程数据处理装置、或其他设备上，使得在计算机、其他可编程数据处理装置或其他设备上执行一系列操作步骤，以产生计算机实现的过程，从而使得在计算机、其他可编程数据处理装置、或其他设备上执行的指令实现流程图和/或框图中的一个或多个方框中规定的功能/动作。
附图中的流程图和框图显示了根据本公开的多个实现的***、方法和计算机程序产品的可能实现的体系架构、功能和操作。在这点上，流程图或框图中的每个方框可以代表一个模块、程序段或指令的一部分，模块、程序段或指令的一部分包含一个或多个用于实现规定的逻辑功能的可执行指令。在有些作为替换的实现中，方框中所标注的功能也可以以不同于附图中所标注的顺序发生。例如，两个连续的方框实际上可以基本并行地执行，它们有时也可以按相反的顺序执行，这依所涉及的功能而定。也要注意的是，框图和/或流程图中的每个方框、以及框图和/或流程图中的方框的组合，可以用执行规定的功能或动作的专用的基于硬件的***来实现，或者可以用专用硬件与计算机指令的组合来实现。
以上已经描述了本公开的各实现，上述说明是示例性的，并非穷尽性的，并且也不限于所公开的各实现。在不偏离所说明的各实现的范围和精神的情况下，对于本技术领域的普通技术人员来说许多修改和变更都是显而易见的。本文中所用术语的选择，旨在最好地解释各实现的原理、实际应用或对市场中的技术的改进，或者使本技术领域的其他普通技术人员能理解本文公开的各个实现方式。

Claims (13)

  1. 一种用于在增强现实应用中渲染虚拟对象的方法，包括：
    获取真实环境的多个环境图像，所述多个环境图像包括分别在多个采集时间点采集的所述真实环境的图像；
    基于所述多个采集时间点，从所述多个环境图像中选择与所述增强现实应用被使用的当前时间点相匹配的环境图像，以作为渲染环境图像；以及
    在所述增强现实应用中呈现利用所述渲染环境图像渲染的所述虚拟对象。
  2. 根据权利要求1的所述方法，其中所述多个环境图像是分别在所述真实环境中的多个采集位置处采集的，以及选择所述渲染环境图像进一步包括：
    确定运行所述增强现实应用的终端设备在所述真实环境中的设备位置；以及
    基于所述多个采集位置，从所述多个环境图像中选择与所述设备位置相匹配的所述渲染环境图像。
  3. 根据权利要求2的所述方法，其中确定与所述设备位置相匹配的所述渲染环境图像包括：
    分别确定所述多个环境图像的多个采集位置与所述设备位置之间的相应距离；以及
    基于所述相应距离的比较来选择所述渲染环境图像。
  4. 根据权利要求3的所述方法，其中基于所述相应距离的所述比较来选择所述渲染环境图像进一步包括：
    基于所述多个环境图像确定所述真实环境的空间结构；
    基于所述空间结构确定所述渲染环境图像的采集位置与所述设备位置之间是否存在障碍物；以及
    响应于确定所述渲染环境图像的所述采集位置与所述设备位置之间不存在障碍物，选择所述渲染环境图像。
  5. 根据权利要求4的所述方法，进一步包括：响应于确定所述渲染环境图像的所述采集位置与所述设备位置之间存在障碍物，选择所述多个环境图像中的采集位置与所述设备位置相邻的另一环境图像以作为所述渲染环境图像。
  6. 根据权利要求2的所述方法，其中呈现所述虚拟对象包括：
    获取与所述渲染环境图像相关联的采集角度；
    基于所述采集角度，将所述渲染环境图像转换至标准角度下的标准环境图像；以及
    将所述标准环境图像作为环境光贴图以渲染所述虚拟对象。
  7. 根据权利要求6的所述方法，其中将所述渲染环境图像转换至所述标准环境图像包括：针对所述渲染环境图像中的像素，
    基于所述采集角度、所述像素在所述渲染环境图像中的球面坐标、以及所述标准角度，确定所述标准环境图像中的与所述像素相对应的标准像素。
  8. 根据权利要求6的所述方法，其中渲染所述虚拟对象包括：
    基于球谐光照模型来确定与所述标准环境图像相关联的球谐光照参数向量；以及
    基于所述球谐光照参数向量来渲染所述虚拟对象。
  9. 根据权利要求8的所述方法，其中利用所述球谐光照参数向量来渲染所述虚拟对象包括：
    基于所述设备位置和所述当前时间中的至少任一项，从所述多个环境图像中选择另一渲染环境图像；
    确定所述另一渲染环境图像的另一球谐光照参数向量；以及
    基于所述球谐光照参数向量和所述另一球谐光照参数向量的插值，渲染所述虚拟图像。
  10. 根据权利要求9的所述方法，其中所述插值包括以下至少任一项：基于位置的插值、基于时间的插值。
  11. 一种用于在增强现实应用中渲染虚拟对象的装置，包括：
    获取模块，被配置用于获取真实环境的多个环境图像，所述多个 环境图像包括分别在多个采集时间点采集的所述真实环境的图像；
    选择模块，被配置用于基于所述多个采集时间点，从所述多个环境图像中选择与所述增强现实应用被使用的当前时间点相匹配的环境图像，以作为渲染环境图像；以及
    呈现模块，被配置用于在所述增强现实应用中呈现利用所述渲染环境图像渲染的所述虚拟对象。
  12. 一种电子设备，包括：
    至少一个处理单元；以及
    至少一个存储器，所述至少一个存储器被耦合到所述至少一个处理单元并且存储用于由所述至少一个处理单元执行的指令，所述指令在由所述至少一个处理单元执行时使所述电子设备执行根据权利要求1至10中任一项所述的方法。
  13. 一种计算机可读存储介质，其上存储有计算机程序，所述计算机程序在被处理器执行时使所述处理器实现根据权利要求1至10中任一项所述的方法。
PCT/CN2023/115909 2022-09-21 2023-08-30 用于渲染虚拟对象的方法、装置、设备和介质 WO2024060952A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202211154080.7 2022-09-21
CN202211154080.7A CN115457179A (zh) 2022-09-21 2022-09-21 用于渲染虚拟对象的方法、装置、设备和介质

Publications (1)

Publication Number Publication Date
WO2024060952A1 true WO2024060952A1 (zh) 2024-03-28

Family

ID=84306618

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2023/115909 WO2024060952A1 (zh) 2022-09-21 2023-08-30 用于渲染虚拟对象的方法、装置、设备和介质

Country Status (2)

Country Link
CN (1) CN115457179A (zh)
WO (1) WO2024060952A1 (zh)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113379884B (zh) * 2021-07-05 2023-11-17 北京百度网讯科技有限公司 地图渲染方法、装置、电子设备、存储介质以及车辆
CN115457179A (zh) * 2022-09-21 2022-12-09 北京字跳网络技术有限公司 用于渲染虚拟对象的方法、装置、设备和介质

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10777010B1 (en) * 2018-03-16 2020-09-15 Amazon Technologies, Inc. Dynamic environment mapping for augmented reality
US11195324B1 (en) * 2018-08-14 2021-12-07 Certainteed Llc Systems and methods for visualization of building structures
CN114979457A (zh) * 2021-02-26 2022-08-30 华为技术有限公司 一种图像处理方法及相关装置
CN115457179A (zh) * 2022-09-21 2022-12-09 北京字跳网络技术有限公司 用于渲染虚拟对象的方法、装置、设备和介质

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107705353B (zh) * 2017-11-06 2020-02-11 太平洋未来科技（深圳）有限公司 应用于增强现实的虚拟对象光影效果的渲染方法和装置
GB2598608B (en) * 2020-09-04 2024-05-08 Sony Interactive Entertainment Inc Content generation system and method
RU2757563C1 (ru) * 2021-02-19 2021-10-18 Самсунг Электроникс Ко., Лтд. Способ визуализации 3d портрета человека с измененным освещением и вычислительное устройство для него
CN114549723A (zh) * 2021-03-30 2022-05-27 完美世界(北京)软件科技发展有限公司 游戏场景中光照信息的渲染方法、装置及设备

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10777010B1 (en) * 2018-03-16 2020-09-15 Amazon Technologies, Inc. Dynamic environment mapping for augmented reality
US11195324B1 (en) * 2018-08-14 2021-12-07 Certainteed Llc Systems and methods for visualization of building structures
CN114979457A (zh) * 2021-02-26 2022-08-30 华为技术有限公司 一种图像处理方法及相关装置
CN115457179A (zh) * 2022-09-21 2022-12-09 北京字跳网络技术有限公司 用于渲染虚拟对象的方法、装置、设备和介质

Also Published As

Publication number Publication date
CN115457179A (zh) 2022-12-09

Similar Documents

Publication Publication Date Title
WO2024060952A1 (zh) 用于渲染虚拟对象的方法、装置、设备和介质
KR102051889B1 (ko) 스마트 글래스에서 ２ｄ 데이터를 기반으로 ３ｄ 증강현실을 구현하는 방법 및 시스템
EP3798801A1 (en) Image processing method and apparatus, storage medium, and computer device
CN108895981B (zh) 一种三维测量方法、装置、服务器和存储介质
CN108805979B (zh) 一种动态模型三维重建方法、装置、设备和存储介质
US20200082571A1 (en) Method and apparatus for calibrating relative parameters of collector, device and storage medium
WO2020228643A1 (zh) 交互控制方法、装置、电子设备及存储介质
WO2017167032A1 (zh) 目标对象展示方法和装置
CN109191554B (zh) 一种超分辨图像重建方法、装置、终端和存储介质
CN108388649B (zh) 处理音视频的方法、***、设备及存储介质
CN111882634A (zh) 一种图像渲染方法、装置、设备及存储介质
CN112766027A (zh) 图像处理方法、装置、设备及存储介质
CN111597628B (zh) 一种模型标记方法、装置、存储介质及电子设备
CN116563493A (zh) 基于三维重建的模型训练方法、三维重建方法及装置
CN114782901A (zh) 基于视觉变动分析的沙盘投影方法、装置、设备及介质
US20220375164A1 (en) Method and apparatus for three dimensional reconstruction, electronic device and storage medium
CN114004972A (zh) 一种图像语义分割方法、装置、设备和存储介质
CN112734629B (zh) 一种正射影像生成方法、装置、设备及存储介质
CN113129362A (zh) 一种三维坐标数据的获取方法及装置
CN113178017A (zh) Ar数据展示方法、装置、电子设备及存储介质
CN112288878A (zh) 增强现实预览方法及预览装置、电子设备及存储介质
CN115512046B (zh) 模型外点位的全景图展示方法和装置、设备、介质
CN115272575B (zh) 图像生成方法及装置、存储介质和电子设备
CN112465692A (zh) 图像处理方法、装置、设备及存储介质
US11910068B2 (en) Panoramic render of 3D video

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23867243

Country of ref document: EP

Kind code of ref document: A1