WO2021190649A1 - Procédé et dispositif de collecte d'images d'une scène pour générer des données de réalité virtuelle - Google Patents

Procédé et dispositif de collecte d'images d'une scène pour générer des données de réalité virtuelle Download PDF

Info

Publication number
WO2021190649A1
WO2021190649A1 PCT/CN2021/083432 CN2021083432W WO2021190649A1 WO 2021190649 A1 WO2021190649 A1 WO 2021190649A1 CN 2021083432 W CN2021083432 W CN 2021083432W WO 2021190649 A1 WO2021190649 A1 WO 2021190649A1
Authority
WO
WIPO (PCT)
Prior art keywords
imaging device
reference points
reference point
space
image
Prior art date
Application number
PCT/CN2021/083432
Other languages
English (en)
Inventor
Yonglin Yang
Yutong CHEN
Ning Tao
Original Assignee
Ke.Com (Beijing) Technology Co., Ltd.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Ke.Com (Beijing) Technology Co., Ltd. filed Critical Ke.Com (Beijing) Technology Co., Ltd.
Publication of WO2021190649A1 publication Critical patent/WO2021190649A1/fr

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/80—Camera processing pipelines; Components thereof
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/63—Control of cameras or camera modules by using electronic viewfinders
    • H04N23/633—Control of cameras or camera modules by using electronic viewfinders for displaying additional information relating to control or operation of the camera
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/64—Computer-aided capture of images, e.g. transfer from script file into camera, check of taken image quality, advice or proposal for image composition or decision on when to take image

Definitions

  • the present disclosure relates to the field of virtual reality technology and, more specifically, to collecting images for generating virtual reality data.
  • Three-dimensional (3D) virtual reality (VR) environment provides a simulated experience that is useful in various applications, such as virtual house touring, virtual map, or interior decoration.
  • the 3D VR scene may be generated based on a plurality of images taken by an imaging device, such as a smartphone or a digital camera.
  • an imaging device such as a smartphone or a digital camera.
  • a user may hold a smartphone and rotate 360 degrees to take a plurality of images of a scene.
  • the plurality of images are concatenated along a horizontal direction to form a panorama of the scene.
  • the panorama generated based on the plurality of images contains a large amount of information of the scene, and depth information of the scene may be extracted from the panorama, for example, by applying artificial intelligence algorithms.
  • the position and the orientation of the imaging device may affect the quality of the generated panorama in a significant way. Therefore, it is desired to control the position and the orientation of the imaging device, while the plurality of the images are acquired.
  • the imaging device may include sensors, such as an accelerometer and a gyroscope, which generate data for calculation of movement and orientation of the imaging device.
  • the accelerometer measures a non-gravitational axis-based linear acceleration of the movement
  • the gyroscope uses the gravity to measure a rotational velocity.
  • it may be further desired to more precisely control the imaging device in order to improve the quality of the panorama.
  • a method, computer readable medium, system and apparatus are disclosed for controlling an imaging device to collect images of a scene. Parameters including a position and an orientation of the imaging device may be effectively controlled, such that a high-quality panorama may be generated based on the images captured by the imaging device.
  • a method for capturing an image of a 3D space using an imaging device.
  • the method includes determining, based on a coordinate system with an origin in the 3D space, a plurality of first reference points with respect to the coordinate system; determining a plurality of second reference points with respect to the coordinate system; determining that one of the first reference points, the corresponding second reference point and an optical center of the imaging device are collinear; and capturing the image of the 3D space in response to the determination of the collinearity and a triggering operation for image capture.
  • the coordinate system is determined based on an initial position of the imaging device.
  • Each second reference point corresponds to one of the first reference points, and the second reference point is on a line connecting the corresponding first reference point and the origin.
  • a computer system includes a camera, a processor, and a memory having stored thereon computer-executable instructions which, when executed by the processor, cause the computer system to perform steps of the method as described above.
  • the computer program includes computer-executable instructions to cause the computer system as described above to perform steps of the method as described above.
  • a non-transitory computer-readable medium has stored thereon the computer program as described above.
  • Fig. 1 illustrates an exemplary three-dimensional (3D) virtual reality (VR) environment in accordance with an embodiment.
  • Fig. 2 illustrates a block diagram of an exemplary computer system in accordance with an embodiment.
  • Fig. 3 illustrates a process for positioning an imaging device to take images of a scene in accordance with an embodiment.
  • Fig. 4 illustrates conditions defined in a 3D space for controlling an imaging device in accordance with an embodiment.
  • Fig. 5 illustrates a process for taking a plurality of images of a 3D space by an imaging device in accordance with an embodiment.
  • Fig. 6 illustrates a plurality of first reference points determined in a 3D space in accordance with one embodiment.
  • Fig. 7 illustrates an exemplary embodiment of three collinear reference points.
  • Fig. 8A illustrates an exemplary embodiment of a user interface showing three reference points.
  • Fig. 8B illustrates an exemplary embodiment of a user interface showing three collinear reference points.
  • Fig. 9 illustrates an apparatus for positioning an imaging device to take images of a scene in accordance with an embodiment.
  • the present disclosure described herein provides a method for capturing images of a scene using an imaging device. Parameters including the position and the orientation of the imaging device may be effectively controlled, such that a high-quality panorama may be generated based on the plurality of the images captured by the imaging device.
  • An optical center of the imaging device may be positioned in a region in a 3D space of the scene. Movement of the imaging device may be controlled such that the optical center of the imaging device remains within the region while acquiring a plurality of images of the 3D space.
  • the region may be a spherical region with a radius that is sufficiently small, such as 15 centimeters or smaller.
  • the imaging device, the optical center of which is placed in the spherical region may be considered relatively fixed at a position while taking images of the scene.
  • the orientation of the imaging device may be determined based on the parameters provided by one or more motion sensors associated with the imaging device.
  • the orientation of the imaging device may be controlled by defining a shooting angle in the 3D space of the scene.
  • a first reference point may be defined with respect to the center of the spherical region with a predefined distance, such as 3 meters.
  • the first reference point and the center of the spherical region may define the shooting angle.
  • the optical center of the imaging device is placed in the spherical region, and the imaging device is oriented towards the first reference point while taking an image.
  • the shooting angle of the image device may be further controlled by defining a second reference point in the 3D space.
  • the second reference point may be collinear with the first reference point and the center of the spherical region, but do not overlap with the first reference point.
  • the optical center of the imaging device is placed in the spherical region, and the imaging device is oriented with the shooting angle defined by the first reference point and the second reference point while taking an image.
  • a plurality of images may be acquired corresponding to a plurality of predefined shooting angles.
  • the plurality of shooting angles are corresponding to a plurality of first reference points in the 3D space.
  • a panorama of the scene may be generated based on the plurality of images.
  • the two adjacent first reference points may be defined such that images taken corresponding to the adjacent first reference points include at least one object in common. At least two advantages are provided by overlapping the two images corresponding to adjacent first reference points: (1) to prevent voids or holes in a panorama; (2) to help with aligning the images for generating the panorama.
  • the imaging device may be controlled to take a plurality of images sequentially.
  • the plurality of images may be combined to generate a panorama of the scene.
  • a simulated 3D VR environment may be generated based on the panorama of the scene.
  • Fig. 1 illustrates an exemplary 3D VR environment 100, in accordance with some embodiments.
  • 3D VR environment 100 may simulate or represent a residential unit, such as an apartment or house floor. It is noted that 3D VR environment 100 may include a VR representation of any in-door space or environment.
  • 3D VR environment 100 may include one or more functional spaces, such as 110, 120, 130, 140, 150, and 160.
  • a functional space refers to an enclosed or partially enclosed space that is associated with a particular function.
  • a functional space may correspond to a room.
  • functional space 110 may correspond to a first bedroom
  • functional space 130 may correspond to a second bedroom.
  • a functional space may correspond to an enclosed or partially enclosed space within or adjacent to a room.
  • functional space 140 may correspond to a closet.
  • a function space may correspond to an area that is generally used for a specific purpose.
  • functional space 120 may correspond to a kitchen area
  • functional space 150 may correspond to a dining area
  • functional space 160 may correspond to a living room.
  • functional spaces 120, 150, and 160 may share the same room (e.g., an enclosed area) , they may be considered as different functional spaces due to their different functions.
  • Fig. 2 illustrates a block diagram of an exemplary computer system 200 configured to implement various functions disclosed herein.
  • computer system 200 may be configured as a server to create or reconstruct VR environment 100.
  • computer system 200 may be configured as terminal device to display or enrich VR environment 100.
  • computer system 200 may include a processor 210, a communication interface 220, a memory/storage 230, and a display 240.
  • Memory/storage 230 may be configured to store computer-readable instructions that, when executed by processor 210, can cause processor 210 to perform various operations disclosed herein.
  • Memory 230 may be any non-transitory type of mass storage, such as volatile or non-volatile, magnetic, semiconductor-based, tape-based, optical, removable, non-removable, or other type of storage device or tangible computer-readable medium including, but not limited to, a ROM, a flash memory, a dynamic RAM, and a static RAM.
  • Processor 210 may be configured to perform the operations in accordance with the instructions stored in memory 230.
  • Processor 210 may include any appropriate type of general-purpose or special-purpose microprocessor, digital signal processor, microcontroller, or the like.
  • Processor 210 may be configured as a separate processor module dedicated to performing one or more specific operations disclosed herein.
  • processor 210 may be configured as a shared processor module for capable of performing other operations unrelated to the one or more specific operations disclosed herein.
  • Communication interface 220 may be configured to communicate information between computer system 200 and other devices or systems.
  • communication interface 220 may include an integrated services digital network (ISDN) card, a cable modem, a satellite modem, or a modem to provide a data communication connection.
  • ISDN integrated services digital network
  • communication interface 220 may include a local area network (LAN) card to provide a data communication connection to a compatible LAN.
  • LAN local area network
  • communication interface 220 may include a high-speed network adapter such as a fiber optic network adaptor, 10G Ethernet adaptor, or the like.
  • Wireless links can also be implemented by communication interface 220.
  • communication interface 220 can send and receive electrical, electromagnetic or optical signals that carry digital data streams representing various types of information via a network.
  • the network can typically include a cellular communication network, a Wireless Local Area Network (WLAN) , a Wide Area Network (WAN) , or the like.
  • WLAN Wireless Local Area Network
  • WAN Wide Area Network
  • Communication interface 220 may also include various I/O devices such as a keyboard, a mouse, a touchpad, a touch screen, a microphone, a camera, a biosensor, etc. A user may input data to terminal a device through communication interface 220.
  • Display 240 may be integrated as part of computer system 200 or may be provided as a separate device communicatively coupled to computer system 200.
  • Display 240 may include a display device such as a Liquid Crystal Display (LCD) , a Light Emitting Diode Display (LED) , a plasma display, or any other type of display, and provide a Graphical User Interface (GUI) presented on the display for user input and data depiction.
  • display device 240 may include a VR goggle, a pair of VR glasses, or other similar devices that provide immersive VR experience.
  • VR environment 100 may be displayed on display 240.
  • display 240 may be integrated as part of communication interface 220.
  • Fig. 3 illustrates a process 300 for controlling an imaging device for acquiring an image of a scene in accordance with an embodiment.
  • the imaging device may be an electronic device with a camera module, such as a smartphone, a tablet, and a laptop.
  • the imaging device may be a camera connected with an electronic device, such as a digital camera communicating with a computer or other mobile devices through a wired or wireless connection.
  • Process 300 may be implemented by device 200 according to the computer-executable instructions stored in memory 230.
  • any system or device capable of carrying out the steps of process 300 is contemplated as being within the scope of the present disclosure.
  • the imaging device may have one or more augmented reality (AR) applications stored thereon.
  • the AR application may provide an Application Programming Interface (API) to render a virtual 3D space of the scene.
  • API Application Programming Interface
  • the AR application may be developed or deployed based on platforms known in the art, such as ARKit in an iPhone operation system (iOS) or ARCore in an Android system.
  • iOS iPhone operation system
  • ARCore ARCore in an Android system.
  • an AR experience may be controlled by using an object available on the AR platform, such as an ARSession object of ARKit in iOS.
  • the ARSession object coordinates processes including reading data from a device’s motion sensor, controlling the device’s built-in camera, and performing image analysis on captured camera images.
  • the imaging device may obtain a pose matrix and determine the position and the orientation of the imaging device.
  • the pose matrix may be a 4 ⁇ 4 matrix output from the motion sensor, which may include a 3 ⁇ 3 rotation matrix and a translation vector as known in the art.
  • a coordinate system with an origin is determined based on an initial position of the imaging device.
  • the coordinate system with the origin may be associated with a 3D space of the scene.
  • the initial position of the imaging device may be derived from a pose matrix output from the motion sensor associated with the imaging device.
  • the motion sensor may be integrated in the imaging device itself or attached to the imaging device to track the movement of the imaging device.
  • the origin of the coordinate system may be set to the optical center of the imaging device at the initial position. Alternatively, a different origin may be selected by the user from an arbitrary point in the coordinate system.
  • a first condition is generated in the coordinate system with respect to the origin of the coordinate system.
  • the first condition may be defined as a 3D region in the scene to control the position of the imaging device.
  • the imaging device is considered to be fixed as long as the optical center of the imaging device is inside the 3D region.
  • the 3D region corresponding to the first condition may be presented in the field of view of the imaging device to the user.
  • the 3D region corresponding to the first condition may be a spherical region 430 with a first radius in a 3D space 400, as illustrated in Fig. 4.
  • the center of spherical region 430 may be defined as an origin 420 of a coordinate system 410.
  • the first radius defines the maximum displacement of the imaging device while acquiring images so that the imaging device may be considered fixed.
  • the imaging device may be placed at origin 420, at position 440, or at position 490 in spherical region 430. Any position inside spherical region 430 is treated as approximately the same as origin 420 for acquiring images.
  • the first radius of spherical region 430 may be adjusted to a second radius to adapt to scenes with different geographical characteristics.
  • the radius of spherical region 430 may be reduced. In other words, the imaging device may be confined in a smaller space while acquiring images.
  • the radius of spherical region 430 may be increased. In other words, larger displacements of the imaging device are tolerated while acquiring images.
  • the adjustment of the radius of spherical region 430 may be performed automatically based on an automatic detection program running on the imaging device or devices connected to the imaging device. Alternatively, the user may manually adjust the radius of spherical region 430 corresponding to the first condition.
  • a second condition may be generated in the coordinate system with respect to the origin.
  • the second condition may control the orientation of the imaging device.
  • the orientation of the imaging device may include a pitch angle, a yaw angle, and/or a roll angle.
  • the second condition may be defined as a pitch angle, a yaw angle, and/or a roll angle of the imaging device.
  • the second condition may be compared with the current orientation of the imaging device based on the parameters output from the motion sensor.
  • the second condition may be defined as a shooting angle with respect to origin 420 in 3D space 400.
  • a first reference point 450 is defined in 3D space 400.
  • the defined shooting angle is corresponding to line 480, which connects first reference point 450 and origin 420.
  • the optical center of the imaging device is placed in spherical region 430, and the imaging device is oriented towards first reference point 450 while acquiring an image.
  • the imaging device may have an actual shooting angle that is slightly different from the defined shooting angle.
  • the actual shooting angle is corresponding to line 460 which connects position 440 and first reference point 450.
  • line 460 may be sufficiently close to line 480.
  • the shooting angle of the imaging device may be more effectively controlled by defining a second reference point 470 in 3D space 400, as illustrated in Fig. 4.
  • Second reference point 470 is collinear with first reference point 450 and origin 420.
  • the imaging device is placed in spherical region 430 with a shooting angle corresponding to line 480 while acquiring an image.
  • the imaging device acquires an image when first reference point 450, second reference point 470, and the optical center of the imaging device are collinear.
  • the actual shooting angle may be the same as the defined shooting angle.
  • the position and the orientation of the imaging device may be better controlled.
  • a plurality of shooting angles may be defined by a plurality of first reference points 450 and a plurality of corresponding second reference points 470 with respect to origin 420 in 3D space 400.
  • the plurality of first reference points 450 may be determined in 3D space 400.
  • a first reference distance may be defined between each first reference point 450 and origin 420. The position and the size of each first reference point 450 may be individually adjusted.
  • a second distance may be defined between each second reference point 470 and origin 420.
  • Each second reference point 470 is corresponding to a first reference point 450. Such that second reference point 470, corresponding first reference point 450, and origin 420 are collinear.
  • Line 480 connecting first reference point 450 and second reference point 470 defines a shooting angle.
  • a plurality of shooting angles may be defined in 3D space 400.
  • the imaging device may be aligned according to the shooting angles one by one to take a plurality of images, while the optical center of the imaging device is fixed at a desired position (e.g., inside spherical region 430) .
  • a third reference point may be presented at the center of the field of view of the imaging device, and represents the optical center of the imaging device.
  • the optical center is aligned with a defined shooting angle, when one of the first reference points 450, the corresponding second reference point 470, and the third reference point are collinear or overlap with one another when viewed through the imaging device.
  • a position and an orientation of the imaging device may be determined based on data output from the motion sensor associated with the imaging device, wherein the motion sensor may track the movement of the imaging device.
  • the motion sensor outputs data in a form of a pose matrix, which may include a rotation matrix and a translation vector.
  • the position and the orientation of the imaging device may be calculated from the pose matrix.
  • the determined position of the imaging device may be compared with the first condition.
  • the determined orientation of the imaging device may be compared with the second condition.
  • a first prompt message may be generated in response to the position of the imaging device satisfying the first condition or not. Additionally or alternatively, the region corresponding to the first condition may be presented on the imaging device in respond to the imaging device not satisfying the first condition. The region corresponding to the first condition may not be rendered on the imaging device in respond to the imaging device satisfying the first condition.
  • the first prompt message may notify the user to move the imaging device until the imaging device satisfies the first condition. In an embodiment, the user may be prompted to take an image when the imaging device satisfies the first condition.
  • the first prompt message may comprise at least a text, an image, or a sound.
  • a second prompt message may be generated in response to the orientation of the imaging device not satisfying the second condition.
  • the second condition may define at least one of a pitch angle, a yaw angle, and/or a roll angle of the imaging device. Alternatively, the second condition may define at least one shooting angle in the 3D space of the scene.
  • the orientation of the imaging device may be determined by the pose matrix output from the motion sensor. When the imaging device satisfies the first condition, the determined orientation of the imaging device is compared with the second condition.
  • the second prompt message may notify the user to move (e.g., rotate) the imaging device until the imaging device satisfies the second condition. When the imaging device satisfies the first condition and the second condition, the imaging device may be triggered to take an image.
  • Fig. 5 illustrates a process 500 for acquiring a plurality of images of a 3D space by an imaging device in accordance with an embodiment.
  • a panorama of the 3D space may be generated based on the plurality of images.
  • Each image is taken at a defined position in the 3D space.
  • two images taken at adjacent positions may share at least one common object, which may be used to align the images to generate the panorama of the scene.
  • voids or holes may be prevented in the generated panorama.
  • Process 500 may be implemented by device 200 and according to the computer-executable instructions stored in memory 230.
  • any system or device capable of carrying out the steps of process 500 is contemplated as being within the scope of the present disclosure.
  • a plurality of first reference points are determined, based on a coordinate system with an origin in the 3D space, with respect to the coordinate system.
  • the coordinate system may be determined based on an initial position of the imaging device.
  • the coordinate system may be determined in accordance with step 310 as described above with respect to Fig. 3.
  • the first reference points may be selected from the 3D space.
  • the first reference points are defined with respect to an interception between a plane (such as plane 630) and a spherical surface 602 in 3D space 600, as illustrated in Fig. 6.
  • Spherical surface 602 is defined with a preset radius and with respect to origin 605 of a coordinate system.
  • the preset radius is a first reference distance between each first reference point 601 and origin 605.
  • Origin 605 may be defined as the optical center of the imaging device when the imaging device is at an initial position.
  • the first reference points may or may not be uniformly distributed on the spherical surface. Assuming the horizontal viewing angle of the imaging device is ⁇ , the number of the first reference points N should be greater or equal to 360°/ ⁇ . Such that images taken corresponding to two adjacent first reference points include an overlapping area, which may prevent voids in the generated panorama.
  • the plurality of first reference points may include at least two subsets.
  • a first subset of the plurality of first reference points correspond to a first pitch angle ⁇ 1 650 for the imaging device.
  • a second subset of the plurality of first reference points correspond to a second pitch angle ⁇ 2 655.
  • the difference between the first pitch angle ⁇ 1 and the second pitch angle ⁇ 2 may be smaller than the vertical viewing angle of the imaging device, such that an image taken corresponding to a first reference point of the first subset and an image taken corresponding to a first reference point of the second subset have at least one object in common.
  • Fig. 6 illustrates the plurality of first reference points including two subsets in 3D space 600 in accordance with one embodiment.
  • 3D space 600 is defined by the coordinate system with origin 605, wherein the coordinate system has an x-axis 610, a y-axis 615, and a z-axis 620.
  • Spherical surface 602 is defined in 3D space 600 with respect to origin 605. The radius of spherical surface 602 is a first reference distance.
  • a plane 625 is defined, perpendicular to y-axis 615 and passing origin 605.
  • a first subset 640 of the first reference points is defined by an interception between a first plane 630 and spherical surface 602.
  • first reference points in first subset 640 are distributed on a circle 645 within plane 630.
  • the distribution of first subset 640 may or may not be uniform around circle 645.
  • Plane 630 may be parallel to plane 625.
  • plane 630 and plane 625 are horizontal planes.
  • the distance between each of the first reference points in first subset 640 and origin 605 is the first reference distance.
  • First pitch angle ⁇ 1 650 is defined as an angle between plane 625 and a line from a first reference point of first subset 640 to origin 605.
  • each first reference point 601 and origin 605 simplifies the setup of the first reference points and improves computational efficiency.
  • setting a fixed pitch angle for the first subset of the first reference points makes it convenient for the user to take images.
  • each first reference point of the first subset may be at a different distance from origin 605.
  • each first reference point of the first subset may correspond to a different pitch angle.
  • the number of the first reference points in first subset 640 may be greater or equal to 360°/ ⁇ , where ⁇ is the horizontal viewing angle of the imaging device. Such that images taken at two adjacent first reference points in first subset 640 may include an overlapping area, which prevents voids or holes from forming in the generated panorama.
  • a second subset 660 of the first reference points is defined with respect to an interception between a second plane 635 and spherical surface 602.
  • the first reference points in second subset 660 are distributed on a circle within plane 635.
  • the distribution of the second subset may or may not be uniform around the circle.
  • Plane 635 may be parallel to plane 625.
  • the distance between each first reference point in second subset 660 and origin 605 is the first reference distance (i.e., the radius of spherical surface 602) .
  • a second pitch angle ⁇ 2 655 is defined as an angle between plane 625 and a line from a first reference point of second subset 660 to origin 605.
  • the number of the first reference points in second subset 660 may be greater or equal to 360°/ ⁇ , where ⁇ is the horizontal viewing angle of the imaging device.
  • the difference between the first pitch angle ⁇ 1 and the second pitch angle ⁇ 2 may be smaller than the vertical viewing angle of the imaging device.
  • Pitch angles ⁇ 1 and ⁇ 2 may have positive or negative signs depending on the side of plane 625 which pitch angles ⁇ 1 and ⁇ 2 fall on.
  • pitch angles ⁇ 1 and ⁇ 2 falling on the same side of plane 625 may both have the same sign (i.e., both being positive or negative) .
  • pitch angles ⁇ 1 and ⁇ 2 falling on different sides of plane 625 may have opposite signs.
  • first reference points in first subset 640 and the number of the first reference points in second subset 660 may or may not be identical.
  • plane 630 may or may not be parallel to plane 625.
  • plane 635 may or may not be parallel to plane 630 or plane 625.
  • Additional subsets of the first reference points may be defined in 3D space 600.
  • a third subset of the first reference points may be defined with respect to an interception between a third plane and spherical surface 602, the third subset of the first reference points corresponding to a third pitch angle.
  • Each subset of images corresponding to a pitch angle may be used to generate a panorama corresponding to the pitch angle.
  • a panorama of a larger vertical viewing angle may be generated by combining the panoramas corresponding to different pitch angles.
  • a plurality of second reference points with respect to the coordinate system are defined.
  • Each of the second reference points corresponds to one of the first reference points.
  • a second reference point may be on a line connecting a corresponding first reference point and the origin.
  • the first reference point, the corresponding second reference point, and the origin are collinear.
  • a first reference distance is between each first reference point and the origin.
  • a second reference distance is between each second reference point and the origin. The second distance may be smaller than the first distance.
  • a spatial relationship of one of the first reference points, the corresponding second reference point, and the optical center of the imaging device are monitored by the imaging device.
  • the imaging device may determine whether the first reference point, the corresponding second reference point, and the optical center are collinear.
  • Fig. 7 illustrates a collinear situation 700 of the three points above in accordance with an embodiment.
  • a first reference point 710 and a corresponding second reference point 720 are presented on a display of the imaging device.
  • the optical center of the imaging device is presented as a third reference point 730 on the display of the imaging device.
  • First reference point 710 and corresponding second reference point 720 are fixed with respect to the 3D space.
  • Third reference point 730 is fixed with respect to the imaging device.
  • First reference point 710 and corresponding second reference point 720 determine a tolerance region 740 in the 3D space.
  • third reference point 730 falls in tolerance region 740, third reference point 730, first reference point 710, and corresponding second reference point 720 are considered to be collinear.
  • the user may move the imaging device until the optical center (i.e., third reference point 730) of the imaging device is collinear with first reference point 710 and corresponding second reference point 720.
  • an image of the 3D space is captured by the imaging device in response to the determination of the collinearity described above, as well as a triggering operation for image capture.
  • the triggering operation for image capture may be a manual triggering operation.
  • a user of the imaging device may perform a triggering operation to trigger image capture.
  • the triggering operation for image capture may be an automatic triggering operation.
  • the imaging device may perform a triggering operation to trigger image capture.
  • Figs. 8A and 8B illustrate an embodiment 800 of three reference points (e.g., a first reference point 810, a corresponding reference point 820, and a third reference point 830) presented on a display screen 860 of an imaging device 850.
  • Imaging device 850 may be a smartphone with a camera module. The field of view of imaging device 850 is displayed on screen 860 of imaging device 850.
  • First reference point 810, corresponding second reference point 820, and third reference point 830 may be displayed on screen 860.
  • First reference point 810 may represent one of the first reference points determined in step 510 as described above.
  • Second reference point 820 may represent a corresponding one of the second reference points determined in step 520 as described above.
  • Third reference point 830 may represent the optical center of imaging device 850 and may be fixed at a center of screen 860 of imaging device 850. Thus, third reference point 830 does not move on screen 860 when imaging device 850 is moved.
  • First reference point 810 and corresponding second reference point 820 are fixed with respect to the 3D space. Therefore, when the user moves imaging device 850, first reference point 810 and corresponding second reference point 820 move on screen 860 of imaging device 850. In addition, a virtual button 840 may be displayed on screen 860.
  • first reference point 810, corresponding second reference point 820 and third reference point 830 are not collinear. When the user moves imaging device 850, the spatial relationship of first reference point 810, corresponding second reference point 820, and third reference point 830 may change. As illustrated in Fig. 8B, first reference point 810, corresponding second reference point 820, and third reference point 830 may overlap as viewed through screen 860.
  • the three reference points are considered collinear in the 3D space.
  • the optical center is aligned with a preset shooting angle determined by first reference point 810 and corresponding second reference point 820.
  • the user may then tap virtual button 840 to take an image at the current position of imaging device 850.
  • positions, sizes, colors, and shapes of the reference points displayed on screen 860 of imaging device 850 are only for purposes of illustration. Positions of the reference points on the screen may vary. Different sizes, colors, and/or shapes of the reference points may be adopted. There may be more than one first reference point and more than one second reference point displaying on screen 860 of imaging device 850. As long as the third reference point 830 overlaps with a first reference point 810 and a corresponding second reference point 820, imaging device 850 may be triggered to take an image. Furthermore, button 840 may be a real button instead of a virtual button. Still further, imaging device 850 may be automatically triggered to take an image when the collinearity occurs.
  • steps 510 through 540 of process 500 may be repeated for taking a plurality of images in the 3D space.
  • the imaging device may capture a first image of the 3D space in response to the determination of the first one of the first reference points, the corresponding first one of the second reference points, and the optical center of the imaging device being collinear. Then, the imaging device may capture a second image of the 3D space in response to the determination of a second one of the first reference points, the corresponding second one of the second reference points, and the optical center of the imaging device being collinear.
  • the first image and the second image may include at least one object in common.
  • the steps are repeated until the imaging device captures images corresponding to all of the first reference points. A panorama can then be generated by concatenating the images captured.
  • the imaging device may move the imaging device to a first location having a first set of positions corresponding to a first set of the first reference points.
  • a first location having a first set of positions corresponding to a first set of the first reference points.
  • an image is taken in response to one of the first set of the first reference points, the corresponding second reference point, and the optical center of the imaging device being collinear.
  • the imaging device is rotated to a next position of the first set of positions. In this way, the imaging device takes a first set of images corresponding to the first set of positions at the first location.
  • the images captured corresponding to two adjacent ones of the first set of positions may include at least one object in common.
  • the user may then move the imaging device to a second location having a second set of positions corresponding to the second set of the first reference points to take a second set of images.
  • the steps are repeated until the user takes images corresponding to all positions corresponding to the first reference points.
  • a panorama of the 3D space may be generated based on the first set of images corresponding to the first set of positions and the second set of images corresponding to the second set of positions. Subsequently, 3D data may be generated based on the panorama.
  • Device 900 may comprise a coordinate unit 910, a condition unit 920, a determining unit 930, and a trigger unit 940, which may be implemented by computer-executable instructions stored in, for example, memory 230 and executed by processor 210 described above with respect to Fig. 2.
  • Coordinate unit 910 may be configured to establish a coordinate system in a 3D space of the scene.
  • the coordinate system may be generated based on an initial position of an imaging device, which may or may not be part of device 900.
  • the initial position of the imaging device may be calculated by data output from a motion sensor associated with the imaging device.
  • the motion sensor may be a three-axis accelerometer and/or a gyroscope.
  • the motion sensor may be integrated in the imaging device itself or attached to the imaging device to track the movement of the imaging device.
  • coordinate unit 910 may determine an origin of the coordinate system.
  • An arbitrary point in the coordinate system may be selected as the origin.
  • the optical center of the imaging device at the initial position may be preset as a default origin. The user may select another point in the 3D space to be the origin.
  • Condition unit 920 may determine, based on the coordinate system and the corresponding origin, a first condition with respect to the origin in the 3D space.
  • the first condition may define at least one position in the 3D space to place the imaging device for acquiring an image.
  • the first condition may be a region defined in the scene.
  • the user may be presented with a spherical region with a predefined radius in the field of view of the imaging device.
  • the optical center of the imaging device may move in or out of the spherical region in the 3D space.
  • the region outside of the spherical region may be rendered as a shadowed region such that the user is guided to maintain the optical center of the imaging device inside the spherical region.
  • the spherical region may be presented on the imaging device when the imaging device is out of the spherical region.
  • the spherical region may not be rendered when the imaging device is in the spherical region.
  • the size and position of the region corresponding to the first condition may be adjustable.
  • the predefined radius of the spherical region may be adjusted manually by the user or automatically based on geographical characteristics of the scene.
  • condition unit 920 may determine a second condition to further control the orientation of the imaging device.
  • the second condition controls at least one of a pitch angle, a yaw angle, or a roll angle of the imaging device.
  • the second condition may define a tolerant range for at least one of the angles for the imaging device.
  • the imaging device may be positioned to be perpendicular to a horizontal direction with a predefined tolerance while taking images of the scene. The user may be notified when the imaging device is not properly positioned.
  • the second condition may be a predefined shooting angle with respect to origin 420 in 3D space 400, as illustrated in Fig. 4 in accordance with an embodiment.
  • Condition unit 920 may generate a first reference point 450 with respect to origin 420 in 3D space 400.
  • the defined shooting angle is corresponding to a line 480, which connects first reference point 450 and origin 420.
  • the imaging device may be placed in a spherical region 430 and oriented towards first reference point 450 while acquiring an image. When the imaging device is placed at a position (such as point 440) that is not on line 480 defined by first reference point 450 and origin 420, the imaging device may have an actual shooting angle that is slightly different from the defined shooting angle.
  • the actual shooting angle may correspond to line 460, which connects point 440 and first reference point 450.
  • line 460 may be sufficiently close to line 480.
  • condition unit 920 may further define a second reference point 470 in 3D space 400, as illustrated in Fig. 4, to control the shooting angle of the imaging device.
  • Condition unit 920 may generate second reference point 470, which is collinear with first reference point 450 and origin 420.
  • the imaging device may be placed in spherical region 430 with the shooting angle corresponding to line 480 while acquiring an image. In other words, the imaging device acquires an image when first reference point 450, second reference point 470, and the optical center of the imaging device (such as point 490) are collinear. In this way, the actual shooting angle may be the same as the defined shooting angle. Thus, the position and the orientation of the imaging device may be better controlled.
  • condition unit 920 may define a plurality of shooting angles by generating a plurality of first reference points 450 and optionally a plurality of corresponding second reference points 470 with respect to origin 420 in 3D space 400.
  • the plurality of first reference points 450 may be placed in 3D space 400.
  • a first reference distance may be defined between each first reference point 450 and origin 420.
  • the position and the size of each first reference point 450 may be individually adjusted.
  • a second distance may be defined between each second reference point 470 and origin 420.
  • Each second reference point 470 is corresponding to a first reference point 450, such that each second reference point 470, corresponding first reference point 450, and origin 420 are collinear.
  • Line 480 connecting first reference point 450 and second reference point 470 defines a shooting angle.
  • the plurality of the shooting angles are defined in 3D space 400.
  • the imaging device may be aligned according to the shooting angles defined above, while the optical center of the imaging device is fixed at a desired position (e.g., inside spherical region 430) .
  • determining unit 930 may be configured to determine the position and the orientation of the imaging device.
  • the position and the orientation of the imaging device may be calculated by data output from a motion sensor.
  • the motion sensor may output a 4 ⁇ 4 pose matrix, which represents the position and the orientation of the imaging device.
  • the pose matrix may include a 3 ⁇ 3 rotation matrix and a translation vector as known in the art.
  • the position and the orientation of the imaging device may be derived from the pose matrix.
  • the determined position and the orientation of the imaging device may be compared with the conditions preset by condition unit 920. Subsequently, one or more operations may be determined by trigger unit 940 in response to the comparison results from determining unit 930.
  • Trigger unit 940 may generate a first prompt message in response to the imaging device satisfying the first condition or not. Furthermore, trigger unit 940 may generate a second prompt message in response to the orientation of the imaging device satisfying the second condition or not. Each of the first prompt message and the second prompt message may comprise at least one of a text, an image, or a sound. Still further, when the imaging device satisfies the condition (s) preset by condition unit 920, trigger unit 940 may trigger the imaging device to take at least one image of the scene. Alternatively, the trigger unit 940 may notify the user to take an image manually.
  • a "computer-readable medium” includes one or more of any suitable media for storing the executable instructions of a computer program such that the instruction execution machine, system, apparatus, or device may read (or fetch) the instructions from the computer-readable medium and execute the instructions for carrying out the described embodiments.
  • Suitable storage formats include one or more of an electronic, magnetic, optical, and electromagnetic format.
  • a non-exhaustive list of conventional exemplary computer-readable medium includes: a portable computer diskette; a random-access memory (RAM) ; a read-only memory (ROM) ; an erasable programmable read only memory (EPROM) ; a flash memory device; and optical storage devices, including a portable compact disc (CD) , a portable digital video disc (DVD) , and the like.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Studio Devices (AREA)

Abstract

L'invention concerne un procédé de capture d'une image d'un espace tridimensionnel à l'aide d'un dispositif d'imagerie. Le procédé consiste à déterminer, à partir d'un système de coordonnées ayant une origine dans l'espace 3D, une pluralité de premiers points de référence par rapport au système de coordonnées, à déterminer une pluralité de seconds points de référence par rapport au système de coordonnées, à déterminer que l'un des premiers points de référence, le second point de référence correspondant et un centre optique du dispositif d'imagerie sont colinéaires, et à capturer l'image de l'espace 3D en réponse à la détermination de la colinéarité et à une opération de déclenchement correspondant à la capture d'image.
PCT/CN2021/083432 2020-03-27 2021-03-26 Procédé et dispositif de collecte d'images d'une scène pour générer des données de réalité virtuelle WO2021190649A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202010232065.4 2020-03-27
CN202010232065.4A CN111432119B (zh) 2020-03-27 2020-03-27 图像拍摄方法、装置、计算机可读存储介质及电子设备

Publications (1)

Publication Number Publication Date
WO2021190649A1 true WO2021190649A1 (fr) 2021-09-30

Family

ID=71549102

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/083432 WO2021190649A1 (fr) 2020-03-27 2021-03-26 Procédé et dispositif de collecte d'images d'une scène pour générer des données de réalité virtuelle

Country Status (2)

Country Link
CN (1) CN111432119B (fr)
WO (1) WO2021190649A1 (fr)

Families Citing this family (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112489113B (zh) * 2020-11-25 2024-06-11 深圳地平线机器人科技有限公司 相机外参标定方法、装置及相机外参标定***
CN112843695B (zh) * 2020-12-31 2023-06-27 上海米哈游天命科技有限公司 拍摄图像的方法、装置、电子设备及存储介质
CN113382166B (zh) * 2021-06-08 2022-07-15 贝壳找房(北京)科技有限公司 摄像设备光心对齐方法和装置、存储介质、电子设备
CN114363526A (zh) * 2021-12-31 2022-04-15 北京有竹居网络技术有限公司 图像拍摄方法、装置和电子设备
CN115499594B (zh) * 2022-09-30 2023-06-30 如你所视(北京)科技有限公司 全景图像生成方法及计算机可读存储介质
CN115988322A (zh) * 2022-11-29 2023-04-18 北京百度网讯科技有限公司 生成全景图像的方法、装置、电子设备和存储介质

Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2005348447A (ja) * 2005-09-01 2005-12-15 Sony Corp 画像撮像装置及び方法
CN101377615A (zh) * 2007-08-29 2009-03-04 三星电子株式会社 拍摄全景画面的方法
JP4551990B2 (ja) * 2005-02-03 2010-09-29 名古屋市 パノラマ映像作成方法と作成装置
US20130147908A1 (en) * 2011-12-13 2013-06-13 Fih (Hong Kong) Limited Electronic device and method of capturing panoramic images using the electronic device
CN103167227A (zh) * 2011-12-14 2013-06-19 深圳富泰宏精密工业有限公司 全景拍照***及方法
CN103795924A (zh) * 2014-02-07 2014-05-14 宇龙计算机通信科技(深圳)有限公司 全景拍照装置及方法
CN103793936A (zh) * 2012-10-31 2014-05-14 波音公司 用于增强现实的自动化参考框架校准
CN106331527A (zh) * 2016-10-12 2017-01-11 腾讯科技（北京）有限公司 一种图像拼接方法及装置
US20170150047A1 (en) * 2015-11-23 2017-05-25 Samsung Electronics Co., Ltd. Electronic apparatus and method for controlling electronic apparatus thereof
CN109903227A (zh) * 2019-02-21 2019-06-18 武汉大学 基于相机几何位置关系的全景影像拼接方法

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103826103B (zh) * 2014-02-27 2017-03-22 浙江宇视科技有限公司 云台摄像机巡航控制方法
CN106815869B (zh) * 2016-10-28 2020-06-19 北京鑫洋泉电子科技有限公司 鱼眼相机的光心确定方法及装置
CN109743489A (zh) * 2019-02-14 2019-05-10 南京泓众电子科技有限公司 一种旋转式相机及大视角照片拍摄方法

Patent Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP4551990B2 (ja) * 2005-02-03 2010-09-29 名古屋市 パノラマ映像作成方法と作成装置
JP2005348447A (ja) * 2005-09-01 2005-12-15 Sony Corp 画像撮像装置及び方法
CN101377615A (zh) * 2007-08-29 2009-03-04 三星电子株式会社 拍摄全景画面的方法
US20130147908A1 (en) * 2011-12-13 2013-06-13 Fih (Hong Kong) Limited Electronic device and method of capturing panoramic images using the electronic device
CN103167227A (zh) * 2011-12-14 2013-06-19 深圳富泰宏精密工业有限公司 全景拍照***及方法
CN103793936A (zh) * 2012-10-31 2014-05-14 波音公司 用于增强现实的自动化参考框架校准
CN103795924A (zh) * 2014-02-07 2014-05-14 宇龙计算机通信科技(深圳)有限公司 全景拍照装置及方法
US20170150047A1 (en) * 2015-11-23 2017-05-25 Samsung Electronics Co., Ltd. Electronic apparatus and method for controlling electronic apparatus thereof
CN106331527A (zh) * 2016-10-12 2017-01-11 腾讯科技（北京）有限公司 一种图像拼接方法及装置
CN109903227A (zh) * 2019-02-21 2019-06-18 武汉大学 基于相机几何位置关系的全景影像拼接方法

Also Published As

Publication number Publication date
CN111432119A (zh) 2020-07-17
CN111432119B (zh) 2021-03-23

Similar Documents

Publication Publication Date Title
WO2021190649A1 (fr) Procédé et dispositif de collecte d'images d'une scène pour générer des données de réalité virtuelle
CN110769240B (zh) 基于拍照的3d建模***及方法、自动3d建模装置及方法
US11217019B2 (en) Presenting image transition sequences between viewing locations
TWI782332B (zh) 一種擴增實境資料呈現方法、設備及儲存媒體
US11854149B2 (en) Techniques for capturing and displaying partial motion in virtual or augmented reality scenes
JP5659304B2 (ja) 画像生成装置および画像生成方法
JP7223449B2 (ja) 撮影に基づく３ｄモデリングシステム
JP5865388B2 (ja) 画像生成装置および画像生成方法
JP6476657B2 (ja) 画像処理装置、画像処理方法、およびプログラム
US9467620B2 (en) Synthetic camera lenses
CN106357966A (zh) 全景图像拍摄装置及全景图像获取方法
JP2017212510A (ja) 画像管理装置、プログラム、画像管理システム及び情報端末
US8451346B2 (en) Optically projected mosaic rendering
US11645780B2 (en) Method and device for collecting images of a scene for generating virtual reality data
CN114283243A (zh) 数据处理方法、装置、计算机设备及存储介质
CN109698903A (zh) 图像获取方法及图像获取装置
JP2016139199A (ja) 画像処理装置、画像処理方法、およびプログラム
WO2022036512A1 (fr) Procédé et dispositif de traitement de données, terminal et support de stockage
TWI581631B (zh) 拍照輔助方法及電子裝置
CN113822936A (zh) 数据处理方法、装置、计算机设备及存储介质
KR20190032787A (ko) 360°비디오 생성 시스템 및 방법
US10885657B2 (en) Configuration for indicating image capture device position
Ban et al. Pixel of matter: new ways of seeing with an active volumetric filmmaking system
US20240087157A1 (en) Image processing method, recording medium, image processing apparatus, and image processing system
CA3102860C (fr) Systeme et methode de modelisation 3d utilisant la photographie et appareil et methode de modelisation 3d automatique

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21774897

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21774897

Country of ref document: EP

Kind code of ref document: A1