WO2022166868A1 - 漫游视图的生成方法、装置、设备和存储介质 - Google Patents

漫游视图的生成方法、装置、设备和存储介质 Download PDF

Info

Publication number
WO2022166868A1
WO2022166868A1 PCT/CN2022/074910 CN2022074910W WO2022166868A1 WO 2022166868 A1 WO2022166868 A1 WO 2022166868A1 CN 2022074910 W CN2022074910 W CN 2022074910W WO 2022166868 A1 WO2022166868 A1 WO 2022166868A1
Authority
WO
WIPO (PCT)
Prior art keywords
depth
panoramic
image
roaming
initial
Prior art date
Application number
PCT/CN2022/074910
Other languages
English (en)
French (fr)
Inventor
焦少慧
刘鑫
王悦
张永杰
Original Assignee
北京字节跳动网络技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 北京字节跳动网络技术有限公司 filed Critical 北京字节跳动网络技术有限公司
Priority to US18/276,139 priority Critical patent/US20240037856A1/en
Publication of WO2022166868A1 publication Critical patent/WO2022166868A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G06T19/003—Navigation within 3D models or images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/00—3D [Three Dimensional] image rendering
    • G06T15/10—Geometric effects
    • G06T15/20—Perspective computation
    • G06T15/205—Image-based rendering
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4038—Image mosaicing, e.g. composing plane images from plane sub-images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G06T7/55—Depth or shape recovery from multiple images

Definitions

  • the embodiments of the present application relate to the technical field of image processing, for example, to a method, apparatus, device, and storage medium for generating a roaming view.
  • VR Virtual Reality
  • VR technology is applied in more and more business scenarios.
  • virtual scene roaming needs to be realized.
  • virtual scene roaming uses 360-degree panoramic images.
  • the user can only view the 360-degree panoramic image by changing the viewing angle at a fixed position, that is, only three-degree-of-freedom roaming can be realized.
  • the displayed walkthrough view tends to be distorted and distorted, resulting in an unrealistic look.
  • the present application provides a method, apparatus, device and storage medium for generating a roaming view.
  • an embodiment of the present application provides a method for generating a roaming view, including:
  • the initial 3D model and the repaired 3D model are fused, and the fusion result is rendered to obtain a current roaming view.
  • an apparatus for generating a roaming view including:
  • an acquisition module configured to acquire an initial three-dimensional model in the same spatial region and a repaired three-dimensional model corresponding to the initial three-dimensional model, wherein the repaired three-dimensional model is obtained after repairing the spatial information in the initial three-dimensional model;
  • a determination module configured to respectively determine a first intersection point set of the roaming ray corresponding to the current roaming parameter and the initial 3D model, and a second intersection point set of the roaming parameter and the repaired 3D model, wherein the current roaming parameter Including the roaming position and roaming perspective after moving;
  • a processing module configured to fuse the initial 3D model and the repaired 3D model according to the depth difference between the corresponding intersections in the first intersection set and the second intersection set, and render the result after fusion to obtain Current roaming view.
  • an embodiment of the present application provides a device for generating a roaming view, including a memory and a processor, where the memory stores a computer program, and when the processor executes the computer program, the first aspect of the embodiment of the present application provides The steps of the generation method of the walkthrough view.
  • an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, implements the steps of the method for generating a roaming view provided in the first aspect of the embodiment of the present application .
  • FIG. 1 is a schematic flowchart of a method for generating a roaming view according to an embodiment of the present application
  • FIG. 2 is a schematic flowchart of an acquisition process of an initial three-dimensional model and a repaired three-dimensional model provided by an embodiment of the present application;
  • FIG. 3 is a schematic flowchart of a process of generating a panoramic depth image according to an embodiment of the present application
  • FIG. 4 is a schematic flowchart of a process of generating a panoramic depth inpainting image according to an embodiment of the present application
  • FIG. 5 is a schematic flowchart of a generation process of a panoramic color restoration image provided by an embodiment of the present application
  • FIG. 6 is a schematic diagram of a principle of a generation process of a panoramic depth restoration image and a panoramic color restoration image provided by an embodiment of the present application;
  • FIG. 7 is a schematic structural diagram of an apparatus for generating a roaming view according to an embodiment of the present application.
  • FIG. 8 is a schematic structural diagram of a device for generating a roaming view according to an embodiment of the present application.
  • the term “including” and variations thereof are open-ended inclusions, ie, "including but not limited to”.
  • the term “based on” is “based at least in part on.”
  • the term “one embodiment” means “at least one embodiment”; the term “another embodiment” means “at least one additional embodiment”; the term “some embodiments” means “at least some embodiments”. Relevant definitions of other terms will be given in the description below.
  • the technical solutions provided by the embodiments of the present application can provide a 6-DOF roaming mode that can change both the position and the viewing angle.
  • Three degrees of freedom refers to the degree of freedom with three rotation angles, that is, it only has the ability to rotate on the three axes of X, Y, and Z, and does not have the ability to move on the three axes of X, Y, and Z.
  • Six degrees of freedom It refers to the degrees of freedom with 3 rotation angles, and also includes 3 degrees of freedom related to the up and down, front and rear, left and right, that is, the six degrees of freedom not only have the ability to rotate on the X, Y, and Z axes, It also has the ability to move on the X, Y, and Z axes.
  • the execution body of the following method embodiments may be a device for generating a roaming view, and the device may be implemented as part of a device (hereinafter referred to as an electronic device) for generating a roaming view through software, hardware, or a combination of software and hardware. all.
  • the electronic device may be a client, including but not limited to a smart phone, a tablet computer, an e-book reader, a vehicle terminal, and the like.
  • the electronic device may also be an independent server or a server cluster, and the specific form of the electronic device is not limited in this embodiment of the present application.
  • the following method embodiments are described by taking the execution subject being an electronic device as an example.
  • FIG. 1 is a schematic flowchart of a method for generating a roaming view according to an embodiment of the present application. This embodiment relates to a process of how an electronic device generates a roaming view. As shown in Figure 1, the method may include:
  • the repaired three-dimensional model is obtained after repairing the spatial information in the initial three-dimensional model.
  • the initial three-dimensional model reflects the panoramic spatial information under the spatial region, which may include RGB (red, green and blue) color information and depth information corresponding to the RGB color information. Since viewing the same spatial area from different positions and different perspectives, the panoramic spatial information that can be viewed will change. Therefore, the initial 3D model needs to be filled and repaired with spatial information to form a corresponding repaired 3D model.
  • Both the above-mentioned initial 3D model and the repaired 3D model can be represented by a 3D point cloud or a 3D mesh.
  • the initial three-dimensional model and the repaired three-dimensional model in the same space area can be pre-generated and stored in corresponding positions.
  • the electronic device acquires the original three-dimensional model and the repaired three-dimensional model under the space area from the corresponding storage location.
  • the current roaming parameters include a roaming position and a roaming angle of view after moving.
  • the roaming view angle may include a field of view angle and a line-of-sight orientation.
  • the user can set the current roaming parameters.
  • the user can input the current roaming parameters through the parameter input box in the current display interface, and can also adjust the position of the virtual sensor and the shooting angle to realize the space area.
  • roaming
  • the virtual sensor can be implemented by a roaming control, that is, a roaming control can be inserted in the current display interface, and the user can operate the roaming control to change the position of the virtual sensor and the shooting angle, that is, the user can change the position of the virtual sensor according to actual needs. Roaming parameters under this spatial area.
  • the electronic device can determine the intersection points of multiple roaming rays corresponding to the current roaming parameters and the initial three-dimensional model based on the current roaming parameters, thereby obtaining a first intersection point set; and determine the multiple roaming rays corresponding to the current roaming parameters.
  • the intersection of the roaming ray and the repaired 3D model is obtained to obtain a second set of intersections. It can be understood that each intersection in the first set of intersections has depth information under the spatial region, and each intersection in the second set of intersections also has depth information under the spatial region.
  • each intersection in the first intersection set has depth information under this spatial region
  • each intersection in the second intersection set also has depth information under this spatial region
  • the intersection in the first intersection set must be the same as The intersections in the second intersection set will form a front-to-back occlusion relationship due to the difference in depth values.
  • the electronic device needs to fuse the initial 3D model and the repaired 3D model based on the depth difference between the corresponding intersections in the first intersection set and the second intersection set. That is, it is determined which points in the first intersection set are not occluded and which are occluded by the corresponding points in the second intersection set, and which points in the second intersection set are not occluded and which are the corresponding points in the first intersection set. occluded, so as to obtain the fusion result of the two 3D models. Next, the fusion result is rendered or drawn, so as to obtain the current roaming view under the current roaming parameters.
  • the above-mentioned process of S103 may be: calculating the depth difference between the first intersection in the first intersection set and the corresponding second intersection in the second intersection set one by one; All the first intersection points that are equal to zero and all second intersection points whose depth difference is greater than zero are taken as the result of fusion of the initial three-dimensional model and the repaired three-dimensional model.
  • the corresponding first intersection point set and second intersection point set are calculated one by one. Depth difference between intersection points. For all the first intersections whose depth difference is less than or equal to zero are not occluded by the corresponding second intersections, and for all second intersections whose depth differences are greater than zero are not occluded by the corresponding first intersections, that is, under the current roaming parameters, they are not blocked by the corresponding second intersections.
  • the occluded points include all the first intersections with the calculated depth difference less than or equal to zero, and all the second intersections with the depth difference greater than zero. Therefore, these unoccluded points can be used as the initial 3D model and the restored 3D model after fusion. the result of.
  • an initial 3D model and a repaired 3D model corresponding to the initial 3D model in the same space area are obtained, and the roaming ray corresponding to the current roaming parameter and the first 3D model of the initial 3D model are respectively determined.
  • the 3D information not limited to the spherical surface can be obtained during the roaming process, and the 3D information includes depth information.
  • the second set of intersections corresponds to the depth difference between the intersections to generate the current roaming view.
  • a six-degree-of-freedom roaming mode that can change both the position and the viewing angle is realized, avoiding the situation that the panoramic image can only be viewed at a fixed position in the related art.
  • the initial 3D model and the repaired 3D model can form an accurate occlusion relationship based on the depth information during the fusion process. Therefore, through the solution described in the embodiment of the present application, the displayed The walkthrough view of , will not be distorted.
  • the user can change the current roaming parameters based on actual needs.
  • the initial 3D model and the repaired 3D model corresponding to the initial 3D model in the same space area can be generated in advance.
  • the foregoing S101 may include:
  • the panoramic color image refers to a 360-degree panoramic image with color information, and the pixel value of each pixel included in it is represented by three components of R, G, and B, and each component is between (0, 255).
  • the spatial area can be captured by a panoramic capture device including at least two cameras, where the sum of the angles of view of all cameras is greater than or equal to the spherical angle of view of 360 degrees.
  • the processing software modifies the combination of images captured by different cameras, so that the images captured by different cameras can be smoothly combined, thereby generating a panoramic color image, that is, stitching the multi-view color images captured into a panoramic color image.
  • a panoramic depth image refers to a 360-degree panoramic image with depth information, and the pixel value of each pixel it includes represents depth information.
  • the depth information here refers to the distance between the plane where the camera that collects the image is located and the surface of the object corresponding to the pixel. the distance.
  • the electronic device can obtain the RGB color information and corresponding depth information of each pixel, so that the electronic device can obtain the RGB color information and corresponding depth information of each pixel based on the , to obtain the three-dimensional information representation in space, thereby generating the initial three-dimensional model.
  • the initial 3D model can be represented by a 3D point cloud or a 3D network.
  • the panoramic color restoration image refers to an image after color information restoration is performed on a panoramic color image
  • the panoramic depth restoration image refers to an image after depth information restoration is performed on a panoramic depth image. Since viewing the same space area from different positions and different perspectives, the panoramic spatial information that can be viewed will change. Therefore, it is also necessary to repair the color information of the panoramic color image to obtain the panoramic color repaired image, and to perform depth analysis on the panoramic depth image. Information inpainting to obtain panoramic depth inpainted images.
  • the electronic device After having the panoramic color restoration image and the panoramic depth restoration image, the electronic device can obtain the RGB color information of each pixel and the corresponding depth information. In this way, the electronic device can obtain the RGB color information of each pixel and the corresponding depth information.
  • the depth information is obtained to obtain the three-dimensional information representation in space, so as to generate the repaired three-dimensional model corresponding to the original three-dimensional model.
  • the repaired 3D model may be represented by a 3D point cloud or a 3D network.
  • the initial 3D model is generated based on the panoramic color image and the panoramic depth image in the same spatial region
  • the restored 3D model corresponding to the initial 3D model is generated based on the panoramic color restored image and the panoramic depth restored image, so that the obtained initial Both the 3D model and the repaired 3D model contain spatial depth information, so that during the roaming process, the current roaming view can be generated based on the depth difference between the corresponding intersection points in the first intersection set and the second intersection set.
  • the method further includes: respectively generating the panoramic color image, the panoramic depth image, and the panoramic color restoration image and the panoramic depth inpainted image.
  • the generating process of the panoramic color image may be: acquiring multiple color images of different shooting angles of view in the same spatial region, wherein the sum of the angles of view of the different shooting angles is greater than or equal to 360 degrees. Next, a transformation matrix between multiple color images is obtained, matching of overlapping feature points in multiple color images is performed based on the transformation matrix between multiple color images, and multiple color images are stitched based on the matching result, thereby obtaining a panorama Color image.
  • the generation process of the panoramic depth image may include:
  • the sum of the viewing angles of different shooting angles is greater than or equal to 360 degrees.
  • a depth camera such as a Time of Flight (TOF) camera
  • a color camera can be set up on a dedicated panorama pan/tilt, and the depth camera and the color camera can be used to shoot the same spatial area synchronously, and adjust continuously.
  • the angle of view is captured, resulting in multiple color images and multiple depth images.
  • TOF Time of Flight
  • a panoramic depth image is obtained by stitching multiple depth images.
  • the process of splicing multiple depth images may be as follows: obtaining transformation matrices between multiple depth images, matching the coincident feature points in multiple depth images based on the transformation matrices between multiple depth images, and matching Multiple depth images are stitched to obtain a panoramic depth image.
  • the above-mentioned process of S302 may be: adopting the same splicing method as that used to generate the panoramic color image, splicing a plurality of depth images to obtain a panoramic depth image.
  • the stitching method of multiple color images can be directly used to perform multiple depth images. stitching, thereby improving the generation efficiency of panoramic depth images.
  • the depth camera will have over-exposure and under-exposure on smooth and bright, frosted or transparent surfaces, resulting in a large number of holes in the collected depth images; at the same time, relatively color
  • the depth acquisition range of the depth camera (including the acquisition viewing angle range and the acquisition depth range) is also limited. For areas that are relatively too far or too close, the depth camera cannot collect the corresponding depth information.
  • the method further includes: respectively performing depth padding and depth enhancement on the plurality of depth images.
  • the three-dimensional information may include a depth boundary, a normal vector, and a straight line that can reflect a spatial perspective relationship.
  • the above depth boundary can be understood as the outline of an object in a color image, such as the outline of a human face.
  • the above normal vector can represent the plane in the color image.
  • the above-mentioned spatial straight lines may be road lines, building edge lines, interior wall corner lines, skirting lines, etc. existing in the color image.
  • the process of generating a panoramic depth image may include: inputting the panoramic color image into a first pre-trained neural network to obtain a panoramic depth image corresponding to the panoramic color image.
  • the first pre-trained neural network is obtained by training the sample panoramic color image and the sample panoramic depth image corresponding to the sample panoramic color image.
  • the prediction of panoramic depth images can be achieved through the first pre-trained neural network. Therefore, a large amount of training data needs to be used to train the first pre-trained neural network.
  • training can be performed by using a large number of sample panoramic color images and sample panoramic depth images corresponding to the sample panoramic color images.
  • the sample panoramic color image is used as the input of the first pre-trained neural network
  • the sample panoramic depth image is used as the expected output of the first pre-trained neural network
  • the preset output and expected output of the first pre-trained neural network are used to calculate the preset The loss value of the loss function, and the parameters of the first pre-trained neural network are adjusted in combination with the loss value until a preset convergence condition is reached, thereby obtaining the trained first pre-trained neural network.
  • the first pretrained neural network can be constructed by a convolutional neural network or an encoder-decoder network.
  • the panoramic color image is input into the first pre-trained neural network, and the panoramic depth image corresponding to the panoramic color image can be predicted through the first pre-trained neural network.
  • a panoramic depth image can be obtained by stitching a plurality of depth images from different shooting perspectives in the same spatial area, or the corresponding panoramic color images in the same spatial area can be predicted by the first pre-trained neural network.
  • the panorama depth image can be generated in a variety of ways, which improves the universality of the scheme.
  • the stitching method of multiple color images can be directly used to stitch multiple depth images, thereby improving the generating efficiency of the panoramic depth image.
  • the generation process of the panoramic depth restoration image may include:
  • one side of the depth discontinuity is the depth foreground, and the other side is the depth background.
  • the depth foreground can be understood as the picture with the depth discontinuity close to the position of the lens
  • the depth background can be understood as the picture with the depth discontinuity far away from the lens position.
  • a threshold can be preset based on actual needs. When the difference between pixel values between adjacent pixels is greater than the threshold, it is considered that the depth value has undergone a large jump. At this time, it can be considered that this part of the pixel
  • the edges formed by the dots are depth discontinuities. Exemplarily, assuming that the set threshold is 20, if the depth difference between adjacent pixels is 100, it can be considered that the edge formed by the part of the pixels is a depth discontinuity.
  • S402. Perform depth expansion on the depth foreground and the depth background respectively to obtain a panoramic depth restoration image corresponding to the panoramic depth image.
  • the depth information of the panoramic depth image needs to be repaired.
  • the depth foreground and the depth background on both sides of the depth discontinuity can be separately expanded.
  • the depth foreground is inflated with a specific structuring element
  • the depth background is inflated with the specific structuring element, thereby realizing the restoration of depth information at depth discontinuities.
  • the generation process of the panoramic color restoration image may include:
  • the electronic device may perform binarization processing on the panoramic depth restoration image, so as to distinguish the first area that has undergone depth restoration and the second area that has not been depth restored in the panoramic depth restoration image. It is used as a reference for color information restoration of panoramic color images.
  • the electronic device can perform the first region based on the first region that has been deeply repaired and the second region that has not been deeply repaired as shown in the binarization mask map. Inpainting of color information, resulting in a panoramic color inpainted image.
  • the texture information of the first area may also be repaired based on the color information in the first area.
  • panoramic color inpainting images can be generated by means of artificial intelligence.
  • the above-mentioned process of S502 may be: inputting the binarized mask map and the panoramic color image into the second pre-trained neural network, through the second pre-training neural network The pre-trained neural network performs color restoration on the panoramic color image to obtain a panoramic color restoration image corresponding to the panoramic color image.
  • the second pre-training neural network is obtained by training the sample binarized mask map, the sample panoramic color image, and the sample panoramic color restoration image corresponding to the sample panoramic color image.
  • training can be performed by using a large number of sample binarized mask images, sample panoramic color images, and sample panoramic color restoration images corresponding to the sample panoramic color images.
  • sample binarized mask map and the sample panoramic color image are used as the input of the second pre-training neural network
  • sample panoramic color restoration image is used as the expected output of the second pre-training neural network.
  • the second pre-trained neural network may be constructed by using a convolutional neural network or an encoder-decoder network, which is not limited in this embodiment.
  • the generation process of the panoramic depth restoration image and the panoramic color restoration image is introduced with the process shown in FIG. 6 , for example:
  • depth expansion is performed on both sides of the depth discontinuity to repair the missing depth information at the depth discontinuity in the panoramic depth image.
  • the color information of the panoramic color image is repaired in combination with the area where the panoramic depth image is deeply repaired, and the missing color information in the panoramic color image is also repaired, so as to prepare for the generation of the subsequent roaming view.
  • FIG. 7 is a schematic structural diagram of an apparatus for generating a roaming view according to an embodiment of the present application.
  • the apparatus may include: an acquisition module 701, a determination module 702 and a processing module 703;
  • the acquisition module 701 is configured to acquire an initial three-dimensional model in the same spatial region and a repaired three-dimensional model corresponding to the initial three-dimensional model, wherein the repaired three-dimensional model is obtained by repairing the spatial information in the original three-dimensional model of;
  • the determining module 702 is configured to respectively determine the first intersection point set of the roaming ray corresponding to the current roaming parameter and the initial 3D model, and the second intersection point set of the roaming parameter and the repaired 3D model, wherein the current roaming parameter Including the roaming position and roaming perspective after moving;
  • the processing module 703 is configured to fuse the initial 3D model and the repaired 3D model according to the depth difference between the corresponding intersections in the first intersection set and the second intersection set, and render the result after fusion to obtain Current roaming view.
  • the device for generating a roaming view acquires an initial 3D model and a repaired 3D model corresponding to the initial 3D model in the same spatial area, and determines the roaming ray corresponding to the current roaming parameters and the first three-dimensional model of the initial 3D model, respectively. an intersection point set, and a second intersection point set of the roaming ray and the repaired 3D model, according to the depth difference between the corresponding intersection points in the first intersection point set and the second intersection point set, the initial 3D model and the The repaired three-dimensional model is fused, and the fused result is rendered to obtain the current roaming view.
  • the 3D information not limited to the spherical surface can be obtained during the roaming process, and the 3D information includes depth information.
  • the second set of intersections corresponds to the depth difference between the intersections to generate the current roaming view.
  • a six-degree-of-freedom roaming mode that can change both the position and the viewing angle is realized, avoiding the situation that the panoramic image can only be viewed at a fixed position in the related art.
  • the initial 3D model and the repaired 3D model can form an accurate occlusion relationship based on the depth information during the fusion process. Therefore, through the solution described in the embodiment of the present application, the displayed The walkthrough view of , will not be distorted.
  • the obtaining module 701 may include: a first generating unit and a second generating unit;
  • the first generating unit is set to generate the initial three-dimensional model according to the panoramic color image and the panoramic depth image in the same spatial area;
  • the second generating unit is configured to generate a restored three-dimensional model corresponding to the initial three-dimensional model according to the panoramic color restored image corresponding to the panoramic color image and the panoramic depth restored image corresponding to the panoramic depth image.
  • the obtaining module 701 may further include: a third generating unit;
  • the third generating unit is configured to generate the panoramic color image, the panoramic depth image, the all the The panoramic color restoration image and the panoramic depth restoration image.
  • the third generating unit includes: a first panoramic depth image generating subunit
  • the first panoramic depth image generating subunit is configured to acquire multiple depth images with different shooting angles of view in the same spatial region; and stitch the multiple depth images to obtain the panoramic depth image.
  • the splicing of the multiple depth images to obtain the panoramic depth image may include: splicing the multiple depth images in the same splicing manner as generating the panoramic color image, The panoramic depth image is obtained.
  • the first panoramic depth image generating subunit is further configured to perform depth repair and depth enhancement on the multiple depth images respectively before splicing the multiple depth images to obtain the panoramic depth image.
  • the third generating unit further includes: a second panoramic depth image generating subunit;
  • the second panoramic depth image generation subunit is configured to input the panoramic color image into a first pre-trained neural network to obtain a panoramic depth image corresponding to the panoramic color image, wherein the first pre-trained neural network It is obtained by training the sample panoramic color image and the sample panoramic depth image corresponding to the sample panoramic color image.
  • the third generating unit further includes: a panorama depth inpainting image generating subunit;
  • the panorama depth inpainting image generation subunit is set to determine depth discontinuities existing in the panorama depth image; respectively perform depth expansion on the depth foreground and the depth background to obtain the panorama depth corresponding to the panorama depth image Repair the image, wherein one side of the depth discontinuity is the depth foreground, and the other side is the depth background.
  • the third generation unit further includes: a panoramic color restoration image generation subunit;
  • the panorama color restoration image generation subunit is set to perform binarization processing on the panorama depth restoration image to obtain a binarized mask map; according to the binarization mask map and the panoramic color image, determine the The panorama color restoration image corresponding to the panorama color image.
  • the panorama color restoration image generation subunit is set to input the binarized mask map and the panorama color image into the second pre-training neural network, and the second pre-training neural network
  • the network performs color restoration on the panoramic color image to obtain a panoramic color restoration image corresponding to the panoramic color image, wherein the second pre-training neural network is obtained by using the sample binarized mask map, the sample panoramic color image and the It is obtained by training the sample panoramic color inpainting image corresponding to the sample panoramic color image.
  • the processing module 703 is configured to calculate the depth difference between the first intersection in the first intersection set and the corresponding second intersection in the second intersection set one by one; All the first intersection points and all the second intersection points whose depth difference is greater than zero are taken as the result of fusion of the initial three-dimensional model and the repaired three-dimensional model.
  • FIG. 8 it shows a schematic structural diagram of an electronic device 800 suitable for implementing an embodiment of the present disclosure.
  • the electronic devices in the embodiments of the present disclosure may include, but are not limited to, such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablets), PMPs (portable multimedia players), vehicle-mounted terminals (eg, mobile terminals such as in-vehicle navigation terminals), etc., and stationary terminals such as digital TVs, desktop computers, and the like.
  • the electronic device shown in FIG. 8 is only an example, and should not impose any limitation on the function and scope of use of the embodiments of the present disclosure.
  • an electronic device 800 may include a processing device (eg, a central processing unit, a graphics processor, etc.) 801, which may be loaded into random access according to a program stored in a read only memory (ROM) 802 or from a storage device 806
  • a program in a memory (RAM) 803 executes various appropriate actions and processes.
  • RAM 803 executes various programs and data necessary for the operation of the electronic device 800 are also stored.
  • the processing device 801, the ROM 802, and the RAM 803 are connected to each other through a bus 804.
  • An input/output (I/O) interface 805 is also connected to bus 804 .
  • the following devices may be connected to the I/O interface 805: input devices 806 including, for example, a touch screen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; including, for example, a liquid crystal display (LCD), speakers, vibration An output device 809 such as a computer; a storage device 806 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 809 .
  • Communication means 809 may allow electronic device 800 to communicate wirelessly or by wire with other devices to exchange data.
  • FIG. 8 shows an electronic device 800 having various means, it should be understood that not all of the illustrated means are required to be implemented or available. More or fewer devices may alternatively be implemented or provided.
  • embodiments of the present disclosure include a computer program product comprising a computer program carried on a non-transitory computer readable medium, the computer program comprising program code arranged to perform the method illustrated in the flowchart.
  • the computer program may be downloaded and installed from the network via the communication device 809, or from the storage device 806, or from the ROM 802.
  • the processing device 801 the above-mentioned functions defined in the methods of the embodiments of the present disclosure are executed.
  • the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the above two.
  • the computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or a combination of any of the above. More specific examples of computer readable storage media may include, but are not limited to, electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable Programmable read only memory (EPROM or flash memory), fiber optics, portable compact disk read only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.
  • a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.
  • a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave with computer-readable program code embodied thereon. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing.
  • a computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device .
  • Program code embodied on a computer readable medium may be transmitted using any suitable medium including, but not limited to, electrical wire, optical fiber cable, RF (radio frequency), etc., or any suitable combination of the foregoing.
  • clients and servers can communicate using any currently known or future developed network protocols, such as HyperText Transfer Protocol (HTTP), and can communicate with digital data in any form or medium.
  • Communication eg, a communication network
  • Examples of communication networks include local area networks (“LAN”), wide area networks (“WAN”), the Internet (eg, the Internet), and peer-to-peer networks (eg, ad hoc peer-to-peer networks), as well as any currently known or future development network of.
  • LAN local area networks
  • WAN wide area networks
  • the Internet eg, the Internet
  • peer-to-peer networks eg, ad hoc peer-to-peer networks
  • the above-mentioned computer-readable medium may be included in the above-mentioned electronic device; or may exist alone without being assembled into the electronic device.
  • the above-mentioned computer-readable medium carries one or more programs, and when the above-mentioned one or more programs are executed by the electronic device, the electronic device: acquires at least two Internet Protocol addresses; A node evaluation request for an Internet Protocol address, wherein the node evaluation device selects an Internet Protocol address from the at least two Internet Protocol addresses and returns it; receives the Internet Protocol address returned by the node evaluation device; wherein the obtained The Internet Protocol address indicates an edge node in the content distribution network.
  • the above computer-readable medium carries one or more programs, when the one or more programs are executed by the electronic device, the electronic device: receives a node evaluation request including at least two Internet Protocol addresses; From the at least two Internet Protocol addresses, the Internet Protocol address is selected; the selected Internet Protocol address is returned; wherein, the received Internet Protocol address indicates an edge node in the content distribution network.
  • Computer program code for performing operations of the present disclosure may be written in one or more programming languages, including but not limited to object-oriented programming languages—such as Java, Smalltalk, C++, and This includes conventional procedural programming languages - such as the "C" language or similar programming languages.
  • the program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server.
  • the remote computer may be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (eg, using an Internet service provider through Internet connection).
  • LAN local area network
  • WAN wide area network
  • each block in the flowchart or block diagrams may represent a module, segment, or portion of code that contains one or more logical functions for implementing the specified functions executable instructions.
  • the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved.
  • each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations can be implemented in dedicated hardware-based systems that perform the specified functions or operations , or can be implemented in a combination of dedicated hardware and computer instructions.
  • the units involved in the embodiments of the present disclosure may be implemented in a software manner, and may also be implemented in a hardware manner.
  • the name of the unit does not constitute a limitation of the unit itself under certain circumstances, for example, the first obtaining unit may also be described as "a unit that obtains at least two Internet Protocol addresses".
  • exemplary types of hardware logic components include: Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), Systems on Chips (SOCs), Complex Programmable Logical Devices (CPLDs) and more.
  • FPGAs Field Programmable Gate Arrays
  • ASICs Application Specific Integrated Circuits
  • ASSPs Application Specific Standard Products
  • SOCs Systems on Chips
  • CPLDs Complex Programmable Logical Devices
  • a machine-readable medium may be a tangible medium that may contain or store a program for use by or in connection with the instruction execution system, apparatus or device.
  • the machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium.
  • Machine-readable media may include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or devices, or any suitable combination of the foregoing.
  • machine-readable storage media would include one or more wire-based electrical connections, portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), fiber optics, compact disk read only memory (CD-ROM), optical storage, magnetic storage, or any suitable combination of the foregoing.
  • RAM random access memory
  • ROM read only memory
  • EPROM or flash memory erasable programmable read only memory
  • CD-ROM compact disk read only memory
  • magnetic storage or any suitable combination of the foregoing.
  • a device for generating a roaming view including a memory and a processor, where the memory stores a computer program, and the processor implements the following steps when executing the computer program:
  • the initial 3D model and the repaired 3D model are fused, and the fusion result is rendered to obtain a current roaming view.
  • a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:
  • the initial 3D model and the repaired 3D model are fused, and the fusion result is rendered to obtain a current roaming view.
  • the apparatus, device, and storage medium for generating a roaming view provided in the foregoing embodiments can execute the method for generating a roaming view provided in any embodiment of the present application, and have corresponding functional modules and beneficial effects for executing the method.
  • the apparatus, device, and storage medium for generating a roaming view provided in the foregoing embodiments can execute the method for generating a roaming view provided in any embodiment of the present application, and have corresponding functional modules and beneficial effects for executing the method.
  • a method for generating a roaming view including:
  • the initial 3D model and the repaired 3D model are fused, and the fusion result is rendered to obtain a current roaming view.
  • the above method for generating a roaming view further comprising: generating an initial three-dimensional model according to a panoramic color image and a panoramic depth image in the same spatial region; The panoramic color restoration image and the panoramic depth restoration image corresponding to the panoramic depth image are generated, and the restoration 3D model corresponding to the initial 3D model is generated.
  • the above method for generating a roaming view is provided, further comprising: respectively generating the panoramic color image, the panoramic depth image, the panoramic color restoration image, and the panoramic depth restoration image.
  • the above method for generating a roaming view further comprising: acquiring multiple depth images from different shooting angles of view in the same spatial region; splicing the multiple depth images to obtain the panoramic depth image.
  • the above method for generating a roaming view is provided, further comprising: stitching the plurality of depth images in the same stitching manner as generating the panoramic color image to obtain the Panoramic depth image.
  • the above method for generating a roaming view is provided, further comprising: performing depth inpainting and depth enhancement on the plurality of depth images, respectively.
  • the above method for generating a roaming view is provided, further comprising: inputting the panoramic color image into a first pre-trained neural network to obtain a panoramic depth corresponding to the panoramic color image image, wherein the first pre-trained neural network is obtained by training a sample panoramic color image and a sample panoramic depth image corresponding to the sample panoramic color image.
  • the above method for generating a roaming view is provided, further comprising: determining depth discontinuities existing in the panoramic depth image; Depth expansion is performed to obtain a panoramic depth restoration image corresponding to the panoramic depth image, wherein one side of the depth discontinuity is a depth foreground, and the other side is a depth background.
  • the above method for generating a roaming view further comprising: performing a binarization process on the panoramic depth inpainting image to obtain a binarized mask image;
  • the mask map and the panorama color image are transformed to determine the panorama color restoration image corresponding to the panorama color image.
  • the above method for generating a roaming view further comprising: inputting the binarized mask map and the panoramic color image into a second pre-trained neural network,
  • the second pre-trained neural network performs color restoration on the panoramic color image, and obtains a panoramic color restoration image corresponding to the panoramic color image, wherein the second pre-trained neural network is a sample binarized mask image. , a sample panoramic color image, and a sample panoramic color restoration image corresponding to the sample panoramic color image.
  • the above method for generating a roaming view further comprising: calculating one by one between a first intersection in the first intersection set and a corresponding second intersection in the second intersection set All first intersections with depth differences less than or equal to zero and all second intersections with depth differences greater than zero are used as the fusion result of the initial three-dimensional model and the repaired three-dimensional model.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Software Systems (AREA)
  • Computer Graphics (AREA)
  • Geometry (AREA)
  • Computer Hardware Design (AREA)
  • General Engineering & Computer Science (AREA)
  • Remote Sensing (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Computing Systems (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Processing Or Creating Images (AREA)

Abstract

一种漫游视图的生成方法、装置、设备和存储介质，方法包括：获取同一空间区域下的初始三维模型和初始三维模型对应的修复三维模型，修复三维模型是对初始三维模型中的空间信息进行修复后得到的；分别确定当前漫游参数对应的漫游光线与初始三维模型的第一交点集，以及漫游光线与修复三维模型的第二交点集，其中，当前漫游参数包括移动后的漫游位置和漫游视角；根据第一交点集和第二交点集中对应交点之间的深度差，对初始三维模型和修复三维模型进行融合，并渲染融合后的结果，得到当前漫游视图。

Description

漫游视图的生成方法、装置、设备和存储介质
本申请要求在2021年2月7日提交中国专利局、申请号为202110168916.8的中国专利申请的优先权，该申请的全部内容通过引用结合在本申请中。
技术领域
本申请实施例涉及图像处理技术领域，例如涉及一种漫游视图的生成方法、装置、设备和存储介质。
背景技术
随着虚拟现实(Virtual Reality，VR)技术的不断发展，VR技术被应用在越来越多的业务场景中，在VR技术的应用中，需要实现虚拟场景漫游。目前，虚拟场景漫游使用的是360度全景图像。这样，在全景漫游时，用户只能在固定的位置通过变换视角来观看360度全景图像，即仅能实现三自由度漫游。而当用户改变位置时，所显示的漫游视图往往会发生变形和扭曲，造成不真实感。
发明内容
针对相关技术中当用户改变位置时所显示的漫游视图往往会发生变形和扭曲的情况，本申请提供一种漫游视图的生成方法、装置、设备和存储介质。
第一方面，本申请实施例提供一种漫游视图的生成方法，包括：
获取同一空间区域下的初始三维模型和所述初始三维模型对应的修复三维模型，其中，所述修复三维模型是对所述初始三维模型中的空间信息进行修复后得到的；
分别确定当前漫游参数对应的漫游光线与所述初始三维模型的第一交点集，以及所述漫游光线与所述修复三维模型的第二交点集，其中，所述当前漫游参数包括移动后的漫游位置和漫游视角；
根据所述第一交点集和所述第二交点集中对应交点之间的深度差，对所述初始三维模型和所述修复三维模型进行融合，并渲染融合后的结果，得到当前漫游视图。
第二方面，本申请实施例提供一种漫游视图的生成装置，包括：
获取模块，设置为获取同一空间区域下的初始三维模型和所述初始三维模型对应的修复三维模型，其中，所述修复三维模型是对所述初始三维模型中的空间信息进行修复后得到的；
确定模块，设置为分别确定当前漫游参数对应的漫游光线与所述初始三维模型的第一交点集，以及所述漫游参数与所述修复三维模型的第二交点集，其中，所述当前漫游参数包括移动后的漫游位置和漫游视角；
处理模块，设置为根据所述第一交点集和所述第二交点集中对应交点之间的深度差，对所述初始三维模型和所述修复三维模型进行融合，并渲染融合后的结果，得到当前漫游视图。
第三方面，本申请实施例提供一种漫游视图的生成设备，包括存储器和处理器，所述存储器存储有计算机程序，所述处理器执行所述计算机程序时实现本申请实施例第一方面提供 的漫游视图的生成方法的步骤。
第四方面，本申请实施例提供一种计算机可读存储介质，其上存储有计算机程序，所述计算机程序被处理器执行时实现本申请实施例第一方面提供的漫游视图的生成方法的步骤。
附图说明
贯穿附图中，相同或相似的附图标记表示相同或相似的元素。应当理解附图是示意性的，原件和元素不一定按照比例绘制。
图1为本申请实施例提供的漫游视图的生成方法的一种流程示意图；
图2为本申请实施例提供的初始三维模型和修复三维模型的获取过程的一种流程示意图；
图3为本申请实施例提供的全景深度图像的生成过程的一种流程示意图；
图4为本申请实施例提供的全景深度修复图像的生成过程的一种流程示意图；
图5为本申请实施例提供的全景颜色修复图像的生成过程的一种流程示意图；
图6为本申请实施例提供的全景深度修复图像和全景颜色修复图像的生成过程的一种原理示意图；
图7为本申请实施例提供的漫游视图的生成装置的一种结构示意图；
图8为本申请实施例提供的漫游视图的生成设备的一种结构示意图。
具体实施方式
下面将参照附图更详细地描述本公开的实施例。虽然附图中显示了本公开的某些实施例，然而应当理解的是，本公开可以通过多种形式来实现，而且不应该被解释为限于这里阐述的实施例，相反提供这些实施例是为了更加透彻和完整地理解本公开。应当理解的是，本公开的附图及实施例仅用于示例性作用，并非用于限制本公开的保护范围。
应当理解，本公开的方法实施方式中记载的多个步骤可以按照不同的顺序执行，和/或并行执行。此外，方法实施方式可以包括附加的步骤和/或省略执行示出的步骤。本公开的范围在此方面不受限制。
本文使用的术语“包括”及其变形是开放性包括，即“包括但不限于”。术语“基于”是“至少部分地基于”。术语“一个实施例”表示“至少一个实施例”；术语“另一实施例”表示“至少一个另外的实施例”；术语“一些实施例”表示“至少一些实施例”。其他术语的相关定义将在下文描述中给出。
需要注意，本公开中提及的“第一”、“第二”等概念仅用于对不同的装置、模块或单元进行区分，并非用于限定这些装置、模块或单元所执行的功能的顺序或者相互依存关系。
需要注意，本公开中提及的“一个”、“多个”的修饰是示意性而非限制性的，本领域技术人员应当理解，除非在上下文另有明确指出，否则应该理解为“一个或多个”。
本公开实施方式中的多个装置之间所交互的消息或者信息的名称仅用于说明性的目的，而并不是用于对这些消息或信息的范围进行限制。
目前，在全景漫游时，用户只能在固定的位置通过变换视角来观看360度全景图像。而当用户改变位置时，所显示的漫游视图往往会发生变形和扭曲，造成不真实感，即相关技术中的漫游仅能实现三自由度漫游方式。为此，本申请实施例提供的技术方案，能够提供既可以改变位置，也可以改变视角的六自由度漫游方式。
为了便于本领域技术人员的理解，以下对三自由度、六自由度的概念进行介绍：
三自由度：是指有3个转动角度的自由度，即其仅具有在X、Y、Z三轴上旋转的能力，不具有在X、Y、Z三轴上移动的能力。
六自由度：是指有3个转动角度的自由度，还包括上下、前后、左右等3个位置相关的自由度，即六自由度不仅具有在X、Y、Z三轴上旋转的能力，还具有在X、Y、Z三轴上移动的能力。
为使本申请的目的、技术方案和优点更加清楚明白，下文中将结合附图对本申请的实施例进行详细说明。需要说明的是，在不冲突的情况下，本申请中的实施例及实施例中的特征可以相互任意组合。
需要说明的是，下述方法实施例的执行主体可以是漫游视图的生成装置，该装置可以通过软件、硬件或者软硬件结合的方式实现成为漫游视图的生成设备(以下简称电子设备)的部分或者全部。例如，该电子设备可以为客户端，包括但不限于智能手机、平板电脑、电子书阅读器以及车载终端等。当然，该电子设备也可以为独立的服务器或者服务器集群，本申请实施例对电子设备的具体形式不做限定。下述方法实施例以执行主体是电子设备为例进行说明。
图1为本申请实施例提供的漫游视图的生成方法的一种流程示意图。本实施例涉及的是电子设备如何生成漫游视图的过程。如图1所示，该方法可以包括：
S101、获取同一空间区域下的初始三维模型和所述初始三维模型对应的修复三维模型。
其中，所述修复三维模型是对初始三维模型中的空间信息进行修复后得到的。该初始三维模型反映了该空间区域下的全景空间信息，其可以包括RGB(红绿蓝)彩色信息和与RGB彩色信息对应的深度信息。由于从不同位置不同视角去观看同一空间区域，所能观看到的全景空间信息会发生变化，因此，还需要对该初始三维模型进行空间信息填充修补，从而形成对应的修复三维模型。上述初始三维模型和修复三维模型均可以通过三维点云或者三维网格来表示。
在实际应用中，可以预先生成同一空间区域下的初始三维模型和修复三维模型并存储在相应的位置处。当需要在该空间区域下进行漫游显示时，电子设备从该相应的存储位置中获取该空间区域下的初始三维模型和修复三维模型。
S102、分别确定当前漫游参数对应的漫游光线与所述初始三维模型的第一交点集，以及所述漫游光线与所述修复三维模型的第二交点集。
其中，所述当前漫游参数包括移动后的漫游位置和漫游视角。该漫游视角可以包括视场角以及视线朝向。
在实际应用中，用户可以对当前漫游参数进行设置，如用户可以通过当前显示界面中的参数输入框输入当前漫游参数，也可以通过调整虚拟传感器的位置以及拍摄视角，来实现在该空间区域下的漫游。例如，虚拟传感器可以通过一漫游控件来实现，即可以在当前显示界面中***漫游控件，用户可以对该漫游控件进行操作，以改变虚拟传感器的位置以及拍摄视角，即用户可以根据实际需求改变在该空间区域下的漫游参数。
在获取到当前漫游参数之后，电子设备便可以基于当前漫游参数，确定当前漫游参数对应的多条漫游光线与初始三维模型的交点，从而得到第一交点集；以及确定当前漫游参数对应的多条漫游光线与修复三维模型的交点，从而得到第二交点集。可以理解的是，第一交点 集中的每个交点具有在该空间区域下的深度信息，第二交点集中的每个交点也具有在该空间区域下的深度信息。
S103、根据所述第一交点集和所述第二交点集中对应交点之间的深度差，对所述初始三维模型和所述修复三维模型进行融合，并渲染融合后的结果，得到当前漫游视图。
其中，由于第一交点集中的每个交点具有在该空间区域下的深度信息，第二交点集中的每个交点也具有在该空间区域下的深度信息，因此，第一交点集中的交点必然与第二交点集中的交点之间因深度值的不同，会形成前后遮挡关系，即在当前漫游参数下，若第一交点集中的部分交点的深度值大于第二交点集中对应的交点的深度值，则第一交点集中的该部分交点会被第二交点集中对应的交点所遮挡，从而无法被看到；反之，若第一交点集中的部分交点的深度值小于第二交点集中对应的交点的深度值，则第二交点集中的该部分交点会被第一交点集中对应的交点所遮挡，从而无法被看到。
基于此，在得到第一交点集和第二交点集之后，电子设备需要基于第一交点集和第二交点集中对应交点之间的深度差，对初始三维模型和修复三维模型进行融合。即确定第一交点集中的哪些点没有被遮挡，哪些点被第二交点集中的对应的点遮挡了，以及确定第二交点集中的哪些点没有被遮挡，哪些点被第一交点集中对应的点遮挡了，从而得到两个三维模型的融合结果。接着，对融合结果进行渲染或者绘制，从而得到在当前漫游参数下的当前漫游视图。
可以理解的是，在漫游过程中，既可以改变漫游位置，也可以改变漫游视角，即实现了六自由度漫游方式，因此，所得到的当前漫游视图属于六自由度漫游视图。
作为一种示例地实施方式，上述S103的过程可以为：逐一计算所述第一交点集中的第一交点和所述第二交点集中对应的第二交点之间的深度差；将深度差小于或等于零的所有第一交点和深度差大于零的所有第二交点作为所述初始三维模型和所述修复三维模型融合后的结果。
其中，在得到第一交点集和第二交点集之后，基于第一交点集中每个交点的深度值以及第二交点集中每个交点的深度值，逐一计算第一交点集和第二交点集中对应交点之间的深度差。对于深度差小于或等于零的所有第一交点没有被对应的第二交点所遮挡，对于深度差大于零的所有第二交点没有被对应的第一交点所遮挡，即在当前漫游参数下，未被遮挡的点包括计算得到的深度差小于或等于零的所有第一交点，以及深度差大于零的所有第二交点，因此，可以将这些未被遮挡的所有点作为初始三维模型和修复三维模型融合后的结果。
本申请实施例提供的漫游视图的生成方法，获取同一空间区域下的初始三维模型和所述初始三维模型对应的修复三维模型，分别确定当前漫游参数对应的漫游光线与所述初始三维模型的第一交点集，以及所述漫游光线与所述修复三维模型的第二交点集，根据所述第一交点集和所述第二交点集中对应交点之间的深度差，对所述初始三维模型和所述修复三维模型进行融合，并渲染融合后的结果，得到当前漫游视图。由于同一空间区域下的初始三维模型和修复三维模型包含了空间三维信息，使得在漫游过程中可获取不限于球面的三维信息，该三维信息包含深度信息，这样，便可以基于第一交点集和第二交点集中对应交点之间的深度差，来生成当前漫游视图。实现了既可以改变位置，也可以改变视角的六自由度漫游方式，避免了相关技术中仅能在固定位置观看全景图像的情况。同时，由于漫游视图的生成过程中结合了深度信息，使得初始三维模型和修复三维模型在融合过程中基于深度信息可形成准确 的遮挡关系，因此，通过本申请实施例所述的方案，所显示的漫游视图不会发生变形失真。
在实际应用中，用户可以基于实际需求改变当前漫游参数，为了能够得到当前漫游参数下的六自由度漫游视图，可以预先生成同一空间区域下的初始三维模型和初始三维模型对应的修复三维模型。为此，在上述实施例的基础上，例如，如图2所示，上述S101可以包括：
S201、根据同一空间区域下的全景颜色图像和全景深度图像，生成初始三维模型。
其中，全景颜色图像是指具有颜色信息的360度全景图像，其包括的每个像素点的像素值由R、G、B三种分量来表示，每个分量介于(0，255)。在实际应用中，可以通过至少包括两个摄像头的全景采集设备拍摄该空间区域，其中所有摄像头镜头的视角之和大于或等于360度的球面视角，将拍摄图像传至后端处理设备后利用图像处理软件对不同摄像头拍摄的画面结合处进行修饰，使不同摄像头拍摄的画面实现平滑结合，从而生成全景颜色图像，即将拍摄的多视角颜色图像拼接为全景颜色图像。
全景深度图像是指具有深度信息的360度全景图像，其包括的每个像素点的像素值代表深度信息，这里的深度信息是指采集图像的摄像头所在平面与该像素点对应的物体表面之间的距离。
在具有全景颜色图像和全景深度图像之后，电子设备便可以得到每个像素点的RGB彩色信息以及对应的深度信息，这样，电子设备便可以基于每个像素点的RGB彩色信息以及对应的深度信息，得到空间中的三维信息表示，从而生成初始三维模型。其中，该初始三维模型可以通过三维点云或者三维网络来表示。
S202、根据所述全景颜色图像对应的全景颜色修复图像和所述全景深度图像对应的全景深度修复图像，生成所述初始三维模型对应的修复三维模型。
其中，全景颜色修复图像是指对全景颜色图像进行颜色信息修复后的图像，全景深度修复图像是指对全景深度图像进行深度信息修复后的图像。由于从不同位置不同视角去观看同一空间区域，所能观看到的全景空间信息会发生变化，因此，还需要对全景颜色图像进行颜色信息修复，得到全景颜色修复图像，以及对全景深度图像进行深度信息修复，得到全景深度修复图像。
在具有全景颜色修复图像和全景深度修复图像之后，电子设备便可以得到每个像素点的RGB彩色信息以及对应的深度信息，这样，电子设备便可以基于每个像素点的RGB彩色信息以及对应的深度信息，得到空间中的三维信息表示，从而生成初始三维模型对应的修复三维模型。其中，该修复三维模型可以通过三维点云或者三维网络来表示。
在本实施例中，基于同一空间区域下的全景颜色图像和全景深度图像生成初始三维模型，以及基于全景颜色修复图像和全景深度修复图像生成初始三维模型对应的修复三维模型，使得所得到的初始三维模型和修复三维模型均包含有空间深度信息，这样，在漫游过程中，便可以基于第一交点集和第二交点集中对应交点之间的深度差，来生成当前漫游视图。
为了能够生成初始三维模型和初始三维模型对应的修复三维模型，在上述实施例的基础上，例如，该方法还包括：分别生成所述全景颜色图像、所述全景深度图像、所述全景颜色修复图像以及所述全景深度修复图像。
例如，全景颜色图像的生成过程可以为：获取该同一空间区域下的不同拍摄视角的多个颜色图像，其中，不同拍摄视角的视角之和大于或等于360度。接着，获取多个颜色图像之间的变换矩阵，基于多个颜色图像之间的变换矩阵进行多个颜色图像中重合特征点的匹配， 并基于匹配结果对多个颜色图像进行拼接，从而得到全景颜色图像。
接下来，具体介绍全景深度图像的生成过程，作为一种示例地实施方式，如图3所示，全景深度图像的生成过程可以包括：
S301、获取所述同一空间区域下的不同拍摄视角的多个深度图像。
其中，不同拍摄视角的视角之和大于或等于360度。在实际应用中，可以将深度相机(如飞行时间(Time of flight，TOF)相机)和颜色相机设置在专用全景云台上，采用深度相机和颜色相机同步对同一空间区域进行拍摄，并不断调整拍摄视角，从而得到多个颜色图像和多个深度图像。
S302、拼接所述多个深度图像，得到所述全景深度图像。
其中，对多个颜色图像进行拼接，从而得到全景颜色图像。对多个深度图像进行拼接，从而得到全景深度图像。例如，拼接多个深度图像的过程可以为：获取多个深度图像之间的变换矩阵，基于多个深度图像之间的变换矩阵进行多个深度图像中重合特征点的匹配，并基于匹配结果对多个深度图像进行拼接，从而得到全景深度图像。
在上述实施例的基础上，例如，上述S302的过程可以为：采用与生成全景颜色图像相同的拼接方式，对多个深度图像进行拼接，得到全景深度图像。
其中，由于多个颜色图像和多个深度图像是基于同一全景云台进行同步采集得到的，因此，在生成全景深度图像时，可以直接使用多个颜色图像的拼接方式，对多个深度图像进行拼接，从而提高全景深度图像的生成效率。
目前，受限于深度相机的硬件成本，深度相机在光滑明亮、磨砂面或者透明等表面会存在过曝以及欠曝等情况，从而导致所采集的深度图像上存在大量的空洞；同时，相对彩色相机来说，深度相机的深度采集范围(包括采集视角范围和采集深度范围)也是受限的，对于相对过远或者过近的区域，深度相机无法采集到对应的深度信息。为此，例如，在上述S302之前，该方法还包括：分别对多个深度图像进行深度填补和深度增强。
例如，针对每个深度图像，预测同一空间区域下的彩色图像中的三维信息，基于该三维信息对深度图像进行深度填补和深度增强。其中，该三维信息可以包括深度边界、法向量以及能够体现空间透视关系的直线。上述深度边界可以理解为彩色图像中的物体轮廓，如人脸的轮廓。上述法向量可以体现彩色图像中的平面。上述空间直线可以为彩色图像中存在的道路线、建筑边缘线、室内墙角线以及踢脚线等。
作为另一种示例地实施方式，全景深度图像的生成过程可以包括：将所述全景颜色图像输入至第一预训练神经网络中，得到所述全景颜色图像对应的全景深度图像。
其中，第一预训练神经网络是通过样本全景颜色图像和样本全景颜色图像对应的样本全景深度图像训练得到的。
在实际应用中，可以通过第一预训练神经网络来实现全景深度图像的预测。因此，需要采用大量的训练数据来训练该第一预训练神经网络。在该第一预训练神经网络的训练过程中，可以通过大量的样本全景颜色图像和该样本全景颜色图像对应的样本全景深度图像来进行训练。例如，将样本全景颜色图像作为第一预训练神经网络的输入，将样本全景深度图像作为第一预训练神经网络的期望输出，通过第一预训练神经网络的预测输出与期望输出，计算预设的损失函数的损失值，并结合该损失值对第一预训练神经网络的参数进行调整，直至达到预设的收敛条件，从而得到训练好的第一预训练神经网络。例如，该第一预训练神经网络可 以通过卷积神经网络或者编码器-解码器网络来构建。
在得到训练好的第一预训练神经网络之后，将全景颜色图像输入至第一预训练神经网络中，通过第一预训练神经网络便可以预测出全景颜色图像对应的全景深度图像。
在本实施例中，既可以通过对同一空间区域下的不同拍摄视角的多个深度图像进行拼接，得到全景深度图像，也可以通过第一预训练神经网络预测同一空间区域下的全景颜色图像对应的全景深度图像，使得全景深度图像的生成方式多样化，提高了该方案的普适性。同时，在全景深度图像生成过程中，可以直接使用多个颜色图像的拼接方式，对多个深度图像进行拼接，从而提高全景深度图像的生成效率。
接下来，再具体介绍全景深度修复图像的生成过程，如图4所示，全景深度修复图像的生成过程可以包括：
S401、确定所述全景深度图像中存在的深度不连续处。
其中，所述深度不连续处的一侧为深度前景，另一侧为深度背景。深度前景可以理解为深度不连续处靠近镜头位置的画面，深度背景可以理解为深度不连续处远离镜头位置的画面。基于全景深度图像中像素点的深度值的变化作为寻找深度不连续处的重要线索。在实际应用中，可以基于实际需求，预先设置一个阈值，当相邻像素间的像素值之差大于该阈值时，则认为深度值发生了很大的跳变，此时可以认为由该部分像素点形成的边缘为深度不连续处。示例性的，假设所设定的阈值为20，若相邻像素间的深度差为100，则可以认为由该部分像素点形成的边缘为深度不连续处。
S402、分别对所述深度前景和所述深度背景进行深度膨胀，得到所述全景深度图像对应的全景深度修复图像。
其中，在确定出全景深度图像中的深度不连续处之后，需要对全景深度图像进行深度信息的修复，此时，可以分别对该深度不连续处两侧的深度前景和深度背景进行深度膨胀。例如，使用一个特定的结构元素对深度前景进行膨胀处理，以及使用该特定的结构元素对深度背景进行膨胀处理，从而实现了对深度不连续处的深度信息的修复。
为了能够生成修复三维模型，在对该深度不连续处进行了深度信息的修复的基础上，还需要对全景颜色图像中的相应区域进行颜色信息修复。为此，还需要生成全景颜色修复图像。接下来，具体介绍全景颜色修复图像的生成过程，如图5所示，全景颜色修复图像的生成过程可以包括：
S501、对所述全景深度修复图像进行二值化处理，得到二值化掩膜图。
其中，在得到全景深度修复图像之后，电子设备可以对全景深度修复图像进行二值化处理，以将全景深度修复图像中已进行深度修复的第一区域与未进行深度修复的第二区域区别开来，作为全景颜色图像进行颜色信息修复的参考依据。
S502、根据所述二值化掩膜图和所述全景颜色图像，确定所述全景颜色图像对应的全景颜色修复图像。
其中，在得到二值化掩膜图之后，电子设备便可以基于二值化掩膜图所示出的已进行深度修复的第一区域与未进行深度修复的第二区域，对第一区域进行颜色信息的修复，从而得到全景颜色修复图像。当然，还可以基于第一区域中的颜色信息，对第一区域的纹理信息进行修复。
在实际应用中，可以通过人工智能的方式生成全景颜色修复图像。为此，在上述实施例 的基础上，例如，上述S502的过程可以为：将所述二值化掩膜图和所述全景颜色图像输入至第二预训练神经网络中，通过所述第二预训练神经网络对所述全景颜色图像进行颜色修复，得到所述全景颜色图像对应的全景颜色修复图像。
其中，所述第二预训练神经网络是通过样本二值化掩膜图、样本全景颜色图像以及样本全景颜色图像对应的样本全景颜色修复图像训练得到的。
为了通过第二预训练神经网络来实现对全景颜色图像的信息修复。因此，需要采用大量的训练数据来训练该第二预训练神经网络。在该第二预训练神经网络的训练过程中，可以通过大量的样本二值化掩膜图、样本全景颜色图像以及样本全景颜色图像对应的样本全景颜色修复图像来进行训练。例如，将样本二值化掩膜图、样本全景颜色图像作为第二预训练神经网络的输入，将样本全景颜色修复图像作为第二预训练神经网络的期望输出，通过第二预训练神经网络的预测输出与期望输出，计算预设的损失函数的损失值，并结合该损失值对预训练神经网络的参数进行调整，直至达到预设的收敛条件，从而得到训练好的第二预训练神经网络。例如，该第二预训练神经网络可以通过卷积神经网络或者编码器-解码器网络来构建，本实施例对此不做限定。
在得到训练好的第二预训练神经网络之后，将二值化掩膜图和全景颜色图像输入至第二预训练神经网络中，通过第二预训练神经网络对全景颜色图像进行颜色信息的修复，从而得到全景颜色图像对应的全景颜色修复图像。
为了便于本领域技术人员的理解，以图6所示的过程介绍全景深度修复图像以及全景颜色修复图像的生成过程，例如：
在得到全景深度图像和全景颜色图像之后，确定该全景深度图像中存在的深度不连续处，分别对深度不连续处两侧的深度前景和深度背景进行深度膨胀，得到全景深度图像对应的全景深度修复图像。接着，对全景深度修复图像进行二值化处理，得到二值化掩膜图，将二值化掩膜图和全景颜色图像输入至第二预训练神经网络中，通过第二预训练神经网络预测全景颜色图像对应的全景颜色修复图像。
在本实施例中，通过识别全景深度图像中存在的深度不连续处，对深度不连续处的两侧进行深度膨胀，以修复全景深度图像在深度不连续处所缺失的深度信息。同时，结合全景深度图像进行深度修复的区域，对全景颜色图像进行颜色信息修复，也修复了全景颜色图像中所缺失的颜色信息，为后续漫游视图的生成做好了准备。
图7为本申请实施例提供的漫游视图的生成装置的一种结构示意图。如图7所示，该装置可以包括：获取模块701、确定模块702和处理模块703；
例如，获取模块701设置为获取同一空间区域下的初始三维模型和所述初始三维模型对应的修复三维模型，其中，所述修复三维模型是对所述初始三维模型中的空间信息进行修复后得到的；
确定模块702设置为分别确定当前漫游参数对应的漫游光线与所述初始三维模型的第一交点集，以及所述漫游参数与所述修复三维模型的第二交点集，其中，所述当前漫游参数包括移动后的漫游位置和漫游视角；
处理模块703设置为根据所述第一交点集和所述第二交点集中对应交点之间的深度差，对所述初始三维模型和所述修复三维模型进行融合，并渲染融合后的结果，得到当前漫游视图。
本申请实施例提供的漫游视图的生成装置，获取同一空间区域下的初始三维模型和所述初始三维模型对应的修复三维模型，分别确定当前漫游参数对应的漫游光线与所述初始三维模型的第一交点集，以及所述漫游光线与所述修复三维模型的第二交点集，根据所述第一交点集和所述第二交点集中对应交点之间的深度差，对所述初始三维模型和所述修复三维模型进行融合，并渲染融合后的结果，得到当前漫游视图。由于同一空间区域下的初始三维模型和修复三维模型包含了空间三维信息，使得在漫游过程中可获取不限于球面的三维信息，该三维信息包含深度信息，这样，便可以基于第一交点集和第二交点集中对应交点之间的深度差，来生成当前漫游视图。实现了既可以改变位置，也可以改变视角的六自由度漫游方式，避免了相关技术中仅能在固定位置观看全景图像的情况。同时，由于漫游视图的生成过程中结合了深度信息，使得初始三维模型和修复三维模型在融合过程中基于深度信息可形成准确的遮挡关系，因此，通过本申请实施例所述的方案，所显示的漫游视图不会发生变形失真。
在上述实施例的基础上，获取模块701可以包括：第一生成单元和第二生成单元；
例如，第一生成单元设置为根据同一空间区域下的全景颜色图像和全景深度图像，生成初始三维模型；
第二生成单元设置为根据所述全景颜色图像对应的全景颜色修复图像和所述全景深度图像对应的全景深度修复图像，生成所述初始三维模型对应的修复三维模型。
在上述实施例的基础上，获取模块701还可以包括：第三生成单元；
例如，第三生成单元设置为在所述第一生成单元根据同一空间区域下的全景颜色图像和全景深度图像，生成初始三维模型之前，分别生成所述全景颜色图像、所述全景深度图像、所述全景颜色修复图像以及所述全景深度修复图像。
在上述实施例的基础上，第三生成单元包括：第一全景深度图像生成子单元；
例如，第一全景深度图像生成子单元设置为获取所述同一空间区域下的不同拍摄视角的多个深度图像；拼接所述多个深度图像，得到所述全景深度图像。
在上述实施例的基础上，所述拼接所述多个深度图像，得到所述全景深度图像可以包括：采用与生成所述全景颜色图像相同的拼接方式，对所述多个深度图像进行拼接，得到所述全景深度图像。
在上述实施例的基础上，第一全景深度图像生成子单元还设置为在拼接所述多个深度图，得到所述全景深度图像之前，分别对所述多个深度图像进行深度修复和深度增强。
在上述实施例的基础上，第三生成单元还包括：第二全景深度图像生成子单元；
例如，第二全景深度图像生成子单元设置为将所述全景颜色图像输入至第一预训练神经网络中，得到所述全景颜色图像对应的全景深度图像，其中，所述第一预训练神经网络是通过样本全景颜色图像和所述样本全景颜色图像对应的样本全景深度图像训练得到的。
在上述实施例的基础上，第三生成单元还包括：全景深度修复图像生成子单元；
例如，全景深度修复图像生成子单元设置为确定所述全景深度图像中存在的深度不连续处；分别对所述深度前景和所述深度背景进行深度膨胀，得到所述全景深度图像对应的全景深度修复图像，其中，所述深度不连续处的一侧为深度前景，另一侧为深度背景。
在上述实施例的基础上，第三生成单元还包括：全景颜色修复图像生成子单元；
例如，全景颜色修复图像生成子单元设置为对所述全景深度修复图像进行二值化处理，得到二值化掩膜图；根据所述二值化掩膜图和所述全景颜色图像，确定所述全景颜色图像对 应的全景颜色修复图像。
在上述实施例的基础上，全景颜色修复图像生成子单元设置为将所述二值化掩膜图和所述全景颜色图像输入至第二预训练神经网络中，通过所述第二预训练神经网络对所述全景颜色图像进行颜色修复，得到所述全景颜色图像对应的全景颜色修复图像，其中，所述第二预训练神经网络是通过样本二值化掩膜图、样本全景颜色图像以及所述样本全景颜色图像对应的样本全景颜色修复图像训练得到的。
在上述实施例的基础上，处理模块703设置为逐一计算所述第一交点集中的第一交点和所述第二交点集中对应的第二交点之间的深度差；将深度差小于或等于零的所有第一交点和深度差大于零的所有第二交点作为所述初始三维模型和所述修复三维模型融合后的结果。
下面参考图8，其示出了适于用来实现本公开实施例的电子设备800的结构示意图。本公开实施例中的电子设备可以包括但不限于诸如移动电话、笔记本电脑、数字广播接收器、PDA(个人数字助理)、PAD(平板电脑)、PMP(便携式多媒体播放器)、车载终端(例如车载导航终端)等等的移动终端以及诸如数字TV、台式计算机等等的固定终端。图8示出的电子设备仅仅是一个示例，不应对本公开实施例的功能和使用范围带来任何限制。
如图8所示，电子设备800可以包括处理装置(例如中央处理器、图形处理器等)801，其可以根据存储在只读存储器(ROM)802中的程序或者从存储装置806加载到随机访问存储器(RAM)803中的程序而执行多种适当的动作和处理。在RAM803中，还存储有电子设备800操作所需的多种程序和数据。处理装置801、ROM 802以及RAM 803通过总线804彼此相连。输入/输出(I/O)接口805也连接至总线804。
通常，以下装置可以连接至I/O接口805：包括例如触摸屏、触摸板、键盘、鼠标、摄像头、麦克风、加速度计、陀螺仪等的输入装置806；包括例如液晶显示器(LCD)、扬声器、振动器等的输出装置809；包括例如磁带、硬盘等的存储装置806；以及通信装置809。通信装置809可以允许电子设备800与其他设备进行无线或有线通信以交换数据。虽然图8示出了具有多种装置的电子设备800，但是应理解的是，并不要求实施或具备所有示出的装置。可以替代地实施或具备更多或更少的装置。
例如，根据本公开的实施例，上文参考流程图描述的过程可以被实现为计算机软件程序。例如，本公开的实施例包括一种计算机程序产品，其包括承载在非暂态计算机可读介质上的计算机程序，该计算机程序包含设置为执行流程图所示的方法的程序代码。在这样的实施例中，该计算机程序可以通过通信装置809从网络上被下载和安装，或者从存储装置806被安装，或者从ROM 802被安装。在该计算机程序被处理装置801执行时，执行本公开实施例的方法中限定的上述功能。
需要说明的是，本公开上述的计算机可读介质可以是计算机可读信号介质或者计算机可读存储介质或者是上述两者的任意组合。计算机可读存储介质例如可以是——但不限于——电、磁、光、电磁、红外线、或半导体的***、装置或器件，或者任意以上的组合。计算机可读存储介质的更具体的例子可以包括但不限于：具有一个或多个导线的电连接、便携式计算机磁盘、硬盘、随机访问存储器(RAM)、只读存储器(ROM)、可擦式可编程只读存储器(EPROM或闪存)、光纤、便携式紧凑磁盘只读存储器(CD-ROM)、光存储器件、磁存储器件、或者上述的任意合适的组合。在本公开中，计算机可读存储介质可以是任何包含或存储程序的有形介质，该程序可以被指令执行***、装置或者器件使用或者与其结合使用。而 在本公开中，计算机可读信号介质可以包括在基带中或者作为载波一部分传播的数据信号，其中承载了计算机可读的程序代码。这种传播的数据信号可以采用多种形式，包括但不限于电磁信号、光信号或上述的任意合适的组合。计算机可读信号介质还可以是计算机可读存储介质以外的任何计算机可读介质，该计算机可读信号介质可以发送、传播或者传输用于由指令执行***、装置或者器件使用或者与其结合使用的程序。计算机可读介质上包含的程序代码可以用任何适当的介质传输，包括但不限于：电线、光缆、RF(射频)等等，或者上述的任意合适的组合。
在一些实施方式中，客户端、服务器可以利用诸如超文本传输协议(HyperText Transfer Protocol，HTTP)之类的任何当前已知或未来研发的网络协议进行通信，并且可以与任意形式或介质的数字数据通信(例如，通信网络)互连。通信网络的示例包括局域网(“LAN”)，广域网(“WAN”)，网际网(例如，互联网)以及端对端网络(例如，ad hoc端对端网络)，以及任何当前已知或未来研发的网络。
上述计算机可读介质可以是上述电子设备中所包含的；也可以是单独存在，而未装配入该电子设备中。
上述计算机可读介质承载有一个或者多个程序，当上述一个或者多个程序被该电子设备执行时，使得该电子设备：获取至少两个网际协议地址；向节点评价设备发送包括所述至少两个网际协议地址的节点评价请求，其中，所述节点评价设备从所述至少两个网际协议地址中，选取网际协议地址并返回；接收所述节点评价设备返回的网际协议地址；其中，所获取的网际协议地址指示内容分发网络中的边缘节点。
或者，上述计算机可读介质承载有一个或者多个程序，当上述一个或者多个程序被该电子设备执行时，使得该电子设备：接收包括至少两个网际协议地址的节点评价请求；从所述至少两个网际协议地址中，选取网际协议地址；返回选取出的网际协议地址；其中，接收到的网际协议地址指示内容分发网络中的边缘节点。
可以以一种或多种程序设计语言或其组合来编写用于执行本公开的操作的计算机程序代码，上述程序设计语言包括但不限于面向对象的程序设计语言—诸如Java、Smalltalk、C++，还包括常规的过程式程序设计语言—诸如“C”语言或类似的程序设计语言。程序代码可以完全地在用户计算机上执行、部分地在用户计算机上执行、作为一个独立的软件包执行、部分在用户计算机上部分在远程计算机上执行、或者完全在远程计算机或服务器上执行。在涉及远程计算机的情形中，远程计算机可以通过任意种类的网络——包括局域网(LAN)或广域网(WAN)—连接到用户计算机，或者，可以连接到外部计算机(例如利用因特网服务提供商来通过因特网连接)。
附图中的流程图和框图，图示了按照本公开多种实施例的***、方法和计算机程序产品的可能实现的体系架构、功能和操作。在这点上，流程图或框图中的每个方框可以代表一个模块、程序段、或代码的一部分，该模块、程序段、或代码的一部分包含一个或多个用于实现规定的逻辑功能的可执行指令。也应当注意，在有些作为替换的实现中，方框中所标注的功能也可以以不同于附图中所标注的顺序发生。例如，两个接连地表示的方框实际上可以基本并行地执行，它们有时也可以按相反的顺序执行，这依所涉及的功能而定。也要注意的是，框图和/或流程图中的每个方框、以及框图和/或流程图中的方框的组合，可以用执行规定的功能或操作的专用的基于硬件的***来实现，或者可以用专用硬件与计算机指令的组合来实现。
描述于本公开实施例中所涉及到的单元可以通过软件的方式实现，也可以通过硬件的方式来实现。其中，单元的名称在某种情况下并不构成对该单元本身的限定，例如，第一获取单元还可以被描述为“获取至少两个网际协议地址的单元”。
本文中以上描述的功能可以至少部分地由一个或多个硬件逻辑部件来执行。例如，非限制性地，可以使用的示范类型的硬件逻辑部件包括：现场可编程门阵列(FPGA)、专用集成电路(ASIC)、专用标准产品(ASSP)、片上***(SOC)、复杂可编程逻辑设备(CPLD)等等。
在本公开的上下文中，机器可读介质可以是有形的介质，其可以包含或存储以供指令执行***、装置或设备使用或与指令执行***、装置或设备结合地使用的程序。机器可读介质可以是机器可读信号介质或机器可读储存介质。机器可读介质可以包括但不限于电子的、磁性的、光学的、电磁的、红外的、或半导体***、装置或设备，或者上述内容的任何合适组合。机器可读存储介质的更具体示例会包括基于一个或多个线的电气连接、便携式计算机盘、硬盘、随机存取存储器(RAM)、只读存储器(ROM)、可擦除可编程只读存储器(EPROM或快闪存储器)、光纤、便捷式紧凑盘只读存储器(CD-ROM)、光学储存设备、磁储存设备、或上述内容的任何合适组合。
在一个实施例中，还提供了一种漫游视图的生成设备，包括存储器和处理器，所述存储器存储有计算机程序，所述处理器执行所述计算机程序时实现以下步骤：
获取同一空间区域下的初始三维模型和所述初始三维模型对应的修复三维模型，其中，所述修复三维模型是对所述初始三维模型中的空间信息进行修复后得到的；
分别确定当前漫游参数对应的漫游光线与所述初始三维模型的第一交点集，以及所述漫游光线与所述修复三维模型的第二交点集，其中，所述当前漫游参数包括移动后的漫游位置和漫游视角；
根据所述第一交点集和所述第二交点集中对应交点之间的深度差，对所述初始三维模型和所述修复三维模型进行融合，并渲染融合后的结果，得到当前漫游视图。
在一个实施例中，还提供了一种计算机可读存储介质，其上存储有计算机程序，所述计算机程序被处理器执行时实现以下步骤：
获取同一空间区域下的初始三维模型和所述初始三维模型对应的修复三维模型，其中，所述修复三维模型是对所述初始三维模型中的空间信息进行修复后得到的；
分别确定当前漫游参数对应的漫游光线与所述初始三维模型的第一交点集，以及所述漫游光线与所述修复三维模型的第二交点集，其中，所述当前漫游参数包括移动后的漫游位置和漫游视角；
根据所述第一交点集和所述第二交点集中对应交点之间的深度差，对所述初始三维模型和所述修复三维模型进行融合，并渲染融合后的结果，得到当前漫游视图。
上述实施例中提供的漫游视图的生成装置、设备以及存储介质可执行本申请任意实施例所提供的漫游视图的生成方法，具备执行该方法相应的功能模块和有益效果。未在上述实施例中详尽描述的技术细节，可参见本申请任意实施例所提供的漫游视图的生成方法。
根据本公开的一个或多个实施例，提供一种漫游视图的生成方法，包括：
获取同一空间区域下的初始三维模型和所述初始三维模型对应的修复三维模型，其中，所述修复三维模型是对所述初始三维模型中的空间信息进行修复后得到的；
分别确定当前漫游参数对应的漫游光线与所述初始三维模型的第一交点集，以及所述漫游光线与所述修复三维模型的第二交点集，其中，所述当前漫游参数包括移动后的漫游位置和漫游视角；
根据所述第一交点集和所述第二交点集中对应交点之间的深度差，对所述初始三维模型和所述修复三维模型进行融合，并渲染融合后的结果，得到当前漫游视图。
根据本公开的一个或多个实施例，提供了如上的漫游视图的生成方法，还包括：根据同一空间区域下的全景颜色图像和全景深度图像，生成初始三维模型；根据所述全景颜色图像对应的全景颜色修复图像和所述全景深度图像对应的全景深度修复图像，生成所述初始三维模型对应的修复三维模型。
根据本公开的一个或多个实施例，提供了如上的漫游视图的生成方法，还包括：分别生成所述全景颜色图像、所述全景深度图像、所述全景颜色修复图像以及所述全景深度修复图像。
根据本公开的一个或多个实施例，提供了如上的漫游视图的生成方法，还包括：获取所述同一空间区域下的不同拍摄视角的多个深度图像；拼接所述多个深度图像，得到所述全景深度图像。
根据本公开的一个或多个实施例，提供了如上的漫游视图的生成方法，还包括：采用与生成所述全景颜色图像相同的拼接方式，对所述多个深度图像进行拼接，得到所述全景深度图像。
根据本公开的一个或多个实施例，提供了如上的漫游视图的生成方法，还包括：分别对所述多个深度图像进行深度修复和深度增强。
根据本公开的一个或多个实施例，提供了如上的漫游视图的生成方法，还包括：将所述全景颜色图像输入至第一预训练神经网络中，得到所述全景颜色图像对应的全景深度图像，其中，所述第一预训练神经网络是通过样本全景颜色图像和所述样本全景颜色图像对应的样本全景深度图像训练得到的。
根据本公开的一个或多个实施例，提供了如上的漫游视图的生成方法，还包括：确定所述全景深度图像中存在的深度不连续处；分别对所述深度前景和所述深度背景进行深度膨胀，得到所述全景深度图像对应的全景深度修复图像，其中，所述深度不连续处的一侧为深度前景，另一侧为深度背景。
根据本公开的一个或多个实施例，提供了如上的漫游视图的生成方法，还包括：对所述全景深度修复图像进行二值化处理，得到二值化掩膜图；根据所述二值化掩膜图和所述全景颜色图像，确定所述全景颜色图像对应的全景颜色修复图像。
根据本公开的一个或多个实施例，提供了如上的漫游视图的生成方法，还包括：将所述二值化掩膜图和所述全景颜色图像输入至第二预训练神经网络中，通过所述第二预训练神经网络对所述全景颜色图像进行颜色修复，得到所述全景颜色图像对应的全景颜色修复图像，其中，所述第二预训练神经网络是通过样本二值化掩膜图、样本全景颜色图像以及所述样本全景颜色图像对应的样本全景颜色修复图像训练得到的。
根据本公开的一个或多个实施例，提供了如上的漫游视图的生成方法，还包括：逐一计算所述第一交点集中的第一交点和所述第二交点集中对应的第二交点之间的深度差；将深度差小于或等于零的所有第一交点和深度差大于零的所有第二交点作为所述初始三维模型和所 述修复三维模型融合后的结果。
以上描述仅为本公开的较佳实施例以及对所运用技术原理的说明。本领域技术人员应当理解，本公开中所涉及的公开范围，并不限于上述技术特征的特定组合而成的技术方案，同时也应涵盖在不脱离上述公开构思的情况下，由上述技术特征或其等同特征进行任意组合而形成的其它技术方案。例如上述特征与本公开中公开的(但不限于)具有类似功能的技术特征进行互相替换而形成的技术方案。
此外，虽然采用特定次序描绘了多种操作，但是这不应当理解为要求这些操作以所示出的特定次序或以顺序次序执行来执行。在一定环境下，多任务和并行处理可能是有利的。同样地，虽然在上面论述中包含了若干具体实现细节，但是这些不应当被解释为对本公开的范围的限制。在单独的实施例的上下文中描述的某些特征还可以组合地实现在单个实施例中。相反地，在单个实施例的上下文中描述的多种特征也可以单独地或以任何合适的子组合的方式实现在多个实施例中。

Claims (14)

  1. 一种漫游视图的生成方法，包括：
    获取同一空间区域下的初始三维模型和所述初始三维模型对应的修复三维模型，其中，所述修复三维模型是对所述初始三维模型中的空间信息进行修复后得到的；
    分别确定当前漫游参数对应的漫游光线与所述初始三维模型的第一交点集，以及所述漫游光线与所述修复三维模型的第二交点集，其中，所述当前漫游参数包括移动后的漫游位置和漫游视角；
    根据所述第一交点集和所述第二交点集中对应交点之间的深度差，对所述初始三维模型和所述修复三维模型进行融合，并渲染融合后的结果，得到当前漫游视图。
  2. 根据权利要求1所述的方法，其中，所述获取同一空间区域下的初始三维模型和所述初始三维模型对应的修复三维模型，包括：
    根据同一空间区域下的全景颜色图像和全景深度图像，生成初始三维模型；
    根据所述全景颜色图像对应的全景颜色修复图像和所述全景深度图像对应的全景深度修复图像，生成所述初始三维模型对应的修复三维模型。
  3. 根据权利要求2所述的方法，在所述根据同一空间区域下的全景颜色图像和全景深度图像，生成初始三维模型之前，还包括：
    分别生成所述全景颜色图像、所述全景深度图像、所述全景颜色修复图像以及所述全景深度修复图像。
  4. 根据权利要求3所述的方法，其中，所述生成所述全景深度图像，包括：
    获取所述同一空间区域下的不同拍摄视角的多个深度图像；
    拼接所述多个深度图像，得到所述全景深度图像。
  5. 根据权利要求4所述的方法，其中，所述拼接所述多个深度图像，得到所述全景深度图像，包括：
    采用与生成所述全景颜色图像相同的拼接方式，对所述多个深度图像进行拼接，得到所述全景深度图像。
  6. 根据权利要求5所述的方法，在所述拼接所述多个深度图，得到所述全景深度图像之前，还包括：
    分别对所述多个深度图像进行深度填补和深度增强。
  7. 根据权利要求3所述的方法，其中，所述生成所述全景深度图像，包括：
    将所述全景颜色图像输入至第一预训练神经网络中，得到所述全景颜色图像对应的全景深度图像，其中，所述第一预训练神经网络是通过样本全景颜色图像和所述样本全景颜色图像对应的样本全景深度图像训练得到的。
  8. 根据权利要求3所述的方法，其中，所述生成所述全景深度修复图像，包括：
    确定所述全景深度图像中存在的深度不连续处，其中，所述深度不连续处的第一侧为深度前景，所述深度不连续处的第二侧为深度背景；
    分别对所述深度前景和所述深度背景进行深度膨胀，得到所述全景深度图像对应的全景深度修复图像。
  9. 根据权利要求8所述的方法，其中，所述生成所述全景颜色修复图像，包括：
    对所述全景深度修复图像进行二值化处理，得到二值化掩膜图；
    根据所述二值化掩膜图和所述全景颜色图像，确定所述全景颜色图像对应的全景颜色修 复图像。
  10. 根据权利要求9所述的方法，其中，所述根据所述二值化掩膜图和所述全景颜色图像，确定所述全景颜色图像对应的全景颜色修复图像，包括：
    将所述二值化掩膜图和所述全景颜色图像输入至第二预训练神经网络中，通过所述第二预训练神经网络对所述全景颜色图像进行颜色修复，得到所述全景颜色图像对应的全景颜色修复图像，其中，所述第二预训练神经网络是通过样本二值化掩膜图、样本全景颜色图像以及所述样本全景颜色图像对应的样本全景颜色修复图像训练得到的。
  11. 根据权利要求1至10中任一项所述的方法，其中，所述根据所述第一交点集和所述第二交点集中对应交点之间的深度差，对所述初始三维模型和所述修复三维模型进行融合，包括：
    逐一计算所述第一交点集中的第一交点和所述第二交点集中对应的第二交点之间的深度差；
    将深度差小于或等于零的所有第一交点和深度差大于零的所有第二交点作为所述初始三维模型和所述修复三维模型融合后的结果。
  12. 一种漫游视图的生成装置，包括：
    获取模块，设置为获取同一空间区域下的初始三维模型和所述初始三维模型对应的修复三维模型，其中，所述修复三维模型是对所述初始三维模型中的空间信息进行修复后得到的；
    确定模块，设置为分别确定当前漫游参数对应的漫游光线与所述初始三维模型的第一交点集，以及所述漫游参数与所述修复三维模型的第二交点集，其中，所述当前漫游参数包括移动后的漫游位置和漫游视角；
    处理模块，设置为根据所述第一交点集和所述第二交点集中对应交点之间的深度差，对所述初始三维模型和所述修复三维模型进行融合，并渲染融合后的结果，得到当前漫游视图。
  13. 一种漫游视图的生成设备，包括存储器和处理器，所述存储器存储有计算机程序，所述处理器执行所述计算机程序时实现权利要求1至11中任一项所述方法的步骤。
  14. 一种计算机可读存储介质，其上存储有计算机程序，所述计算机程序被处理器执行时实现权利要求1至11中任一项所述方法的步骤。
PCT/CN2022/074910 2021-02-07 2022-01-29 漫游视图的生成方法、装置、设备和存储介质 WO2022166868A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US18/276,139 US20240037856A1 (en) 2021-02-07 2022-01-29 Walkthrough view generation method, apparatus and device, and storage medium

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202110168916.8A CN112802206B (zh) 2021-02-07 2021-02-07 漫游视图的生成方法、装置、设备和存储介质
CN202110168916.8 2021-02-07

Publications (1)

Publication Number Publication Date
WO2022166868A1 true WO2022166868A1 (zh) 2022-08-11

Family

ID=75814661

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/074910 WO2022166868A1 (zh) 2021-02-07 2022-01-29 漫游视图的生成方法、装置、设备和存储介质

Country Status (3)

Country Link
US (1) US20240037856A1 (zh)
CN (1) CN112802206B (zh)
WO (1) WO2022166868A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117201705A (zh) * 2023-11-07 2023-12-08 天津云圣智能科技有限责任公司 一种全景图像的获取方法、装置、电子设备及存储介质

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112802206B (zh) * 2021-02-07 2022-10-14 北京字节跳动网络技术有限公司 漫游视图的生成方法、装置、设备和存储介质

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2012071435A1 (en) * 2010-11-24 2012-05-31 Google Inc. Rendering and navigating photographic panoramas with depth information in a geographic information system
WO2013178069A1 (zh) * 2012-05-29 2013-12-05 腾讯科技（深圳）有限公司 基于全景图的视点间漫游方法、装置和机器可读介质
US20140285486A1 (en) * 2013-03-20 2014-09-25 Siemens Product Lifecycle Management Software Inc. Image-based 3d panorama
CN106548516A (zh) * 2015-09-23 2017-03-29 腾讯科技（深圳）有限公司 三维漫游方法和装置
CN112802206A (zh) * 2021-02-07 2021-05-14 北京字节跳动网络技术有限公司 漫游视图的生成方法、装置、设备和存储介质

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CA2818695C (en) * 2010-11-24 2018-06-26 Google, Inc. Guided navigation through geo-located panoramas
CN103049266A (zh) * 2012-12-17 2013-04-17 天津大学 Delta3D三维场景漫游的鼠标操作方法
CN108594996B (zh) * 2018-04-16 2020-12-15 微幻科技(北京)有限公司 一种虚拟漫游中自动调整视角的方法及装置
US10616483B1 (en) * 2019-02-27 2020-04-07 Hong Kong Applied Science and Technology Research Institute Company Limited Apparatus and method of generating electronic three-dimensional walkthrough environment
CN111599021A (zh) * 2020-04-30 2020-08-28 北京字节跳动网络技术有限公司 一种虚拟空间漫游指引方法、装置和电子设备
CN111798562B (zh) * 2020-06-17 2022-07-08 同济大学 一种虚拟建筑空间搭建与漫游方法

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2012071435A1 (en) * 2010-11-24 2012-05-31 Google Inc. Rendering and navigating photographic panoramas with depth information in a geographic information system
WO2013178069A1 (zh) * 2012-05-29 2013-12-05 腾讯科技（深圳）有限公司 基于全景图的视点间漫游方法、装置和机器可读介质
US20140285486A1 (en) * 2013-03-20 2014-09-25 Siemens Product Lifecycle Management Software Inc. Image-based 3d panorama
CN106548516A (zh) * 2015-09-23 2017-03-29 腾讯科技（深圳）有限公司 三维漫游方法和装置
CN112802206A (zh) * 2021-02-07 2021-05-14 北京字节跳动网络技术有限公司 漫游视图的生成方法、装置、设备和存储介质

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
YAN ZHIYUAN;WU DONGMEI;BAO YIDONG;DU ZHIJIANG: "Virtual 3d Camera's Modeling and Roaming Control", HIGH TECHNOLOGY LETTERS, vol. 23, no. 2, 15 February 2013 (2013-02-15), pages 181 - 188, XP055956475, ISSN: 1002-0470, DOI: 10.3772/j.issn.1002-0479.2013.02.011 *

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117201705A (zh) * 2023-11-07 2023-12-08 天津云圣智能科技有限责任公司 一种全景图像的获取方法、装置、电子设备及存储介质
CN117201705B (zh) * 2023-11-07 2024-02-02 天津云圣智能科技有限责任公司 一种全景图像的获取方法、装置、电子设备及存储介质

Also Published As

Publication number Publication date
US20240037856A1 (en) 2024-02-01
CN112802206B (zh) 2022-10-14
CN112802206A (zh) 2021-05-14

Similar Documents

Publication Publication Date Title
CN108665536B (zh) 三维与实景数据可视化方法、装置与计算机可读存储介质
EP2328125B1 (en) Image splicing method and device
WO2022166868A1 (zh) 漫游视图的生成方法、装置、设备和存储介质
US20120105581A1 (en) 2d to 3d image and video conversion using gps and dsm
WO2024104248A1 (zh) 虚拟全景图的渲染方法、装置、设备及存储介质
CN110728622B (zh) 鱼眼图像处理方法、装置、电子设备及计算机可读介质
CN111402404B (zh) 全景图补全方法、装置、计算机可读存储介质及电子设备
CN112801907B (zh) 深度图像的处理方法、装置、设备和存储介质
CN112733820A (zh) 障碍物信息生成方法、装置、电子设备和计算机可读介质
WO2023207379A1 (zh) 图像处理方法、装置、设备及存储介质
CN114449249B (zh) 图像投影方法、装置、存储介质以及投影设备
CN113643414A (zh) 一种三维图像生成方法、装置、电子设备及存储介质
CN113724391A (zh) 三维模型构建方法、装置、电子设备和计算机可读介质
CN115908679A (zh) 纹理映射方法、装置、设备及存储介质
CN115810101A (zh) 三维模型风格化方法、装置、电子设备及存储介质
CN113838116B (zh) 确定目标视图的方法、装置、电子设备及存储介质
CN114125411B (zh) 投影设备校正方法、装置、存储介质以及投影设备
CN111818265B (zh) 基于增强现实模型的交互方法、装置、电子设备及介质
CN115409696A (zh) 图像处理方法、装置、电子设备及存储介质
CN111862342A (zh) 增强现实的纹理处理方法、装置、电子设备及存储介质
CN115002442B (zh) 一种图像展示方法、装置、电子设备及存储介质
Hu et al. 3D map reconstruction using a monocular camera for smart cities
CN111263115B (zh) 用于呈现图像的方法、装置、电子设备和计算机可读介质
CN114419299A (zh) 虚拟物体的生成方法、装置、设备及存储介质
CN115660959B (zh) 图像的生成方法、装置、电子设备及存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22749147

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 18276139

Country of ref document: US

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM1205A DATED 14.11.2023)

122 Ep: pct application non-entry in european phase

Ref document number: 22749147

Country of ref document: EP

Kind code of ref document: A1