WO2012071445A2 - Guided navigation through geo-located panoramas - Google Patents
Guided navigation through geo-located panoramas Download PDFInfo
- Publication number
- WO2012071445A2 WO2012071445A2 PCT/US2011/061906 US2011061906W WO2012071445A2 WO 2012071445 A2 WO2012071445 A2 WO 2012071445A2 US 2011061906 W US2011061906 W US 2011061906W WO 2012071445 A2 WO2012071445 A2 WO 2012071445A2
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- virtual camera
- navigation
- panoramic image
- dimensional environment
- dimensional
- Prior art date
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T15/00—3D [Three Dimensional] image rendering
- G06T15/10—Geometric effects
- G06T15/20—Perspective computation
Definitions
- Embodiments relate generally to three-dimensional graphics.
- the three dimensional environment includes a virtual camera that defines what three dimensional data to display.
- the virtual camera has a perspective according to its position and orientation. By changing the perspective of the virtual camera, a user can navigate through the three dimensional environment.
- a geographic information system is one type of system that uses a virtual camera to navigate through a three-dimensional environment.
- a geographic information system is a system for storing, retrieving, manipulating, and displaying a substantially spherical three-dimensional model of the Earth.
- the three-dimensional model may include satellite imagery, maps, models of buildings and terrain, and other geographic features.
- the three-dimensional environment may be used to display geo-located photographs of real world scenes corresponding to various geographic locations.
- the geo- located photographs may correspond to a network of streets of a major metropolitan city. Such photographs may also include panoramic images that provide 360-degree street- level views.
- the virtual camera in the geographic information system may view the spherical three-dimensional model of the Earth from different perspectives.
- An aerial view of the model of the Earth may show satellite images, but the terrain and buildings may not be displayed.
- a ground-level view of the model may show the terrain and buildings in detail.
- some views of the model may not be accurate or visually acceptable representations of the image content displayed by the GIS.
- a photographic image displayed in the GIS may be accurate from only a single viewpoint or range of viewpoints based on, for example, the position of the camera used to capture the image.
- navigating between different views of an image in a three-dimensional environment may become difficult and visually unappealing to a user.
- a capability for guided navigation in an interactive virtual three-dimensional environment is provided. Such a capability may enhance user experience by providing the feeling of free-form navigation to a user. It may be necessary to constrain the user to certain areas of good visual quality, and subtly guide the user towards viewpoints with better rendering results without disrupting the metaphor of freeform navigation. Additionally, such a capability may enable users to "drive" down a street, follow curving roads, and turn around intersections within the interactive virtual three-dimensional environment. Further, this capability may be applicable to image-based rendering techniques in addition to any three-dimensional graphics system that incorporates navigation based on road networks and/or paths.
- a three-dimensional mesh of polygons for a first panoramic image and a second panoramic image is generated based on depth values associated with various points within a scene represented in the respective first and second panoramic images.
- the first panoramic image includes a link to the second panoramic image.
- the link represents an obstruction-free path (e.g., a street) that connects the panoramic images in three-dimensional space.
- a region of sufficient visual quality for each of the first and second panoramic images in the three-dimensional mesh is determined.
- the region of sufficient visual quality can be based on the depth values and may correspond to viewpoints from which the three-dimensional mesh can be rendered so as to be a visually accurate representation of the image objects in the first and second panoramic images.
- a navigation radius is then calculated for each of the first and second panoramic images based on the determined region of visual quality of each of the first and second panoramic images.
- a navigation channel relative to a path between the first and second panoramic images in the three-dimensional mesh is created based on the calculated navigation radius of each of image.
- the navigation channel is used to define a bounded volume of space within the three-dimensional environment in which a virtual camera can be moved.
- the navigation channel ensures that visual quality is maintained as the virtual camera moves between different viewpoints in the three-dimensional environment.
- FIG. 1 is a diagram illustrating a system for guided navigation along a street network, according to an embodiment.
- FIG. 2 is a diagram of an exemplary navigation channel within a depth mesh for guided navigation in a three-dimensional environment, according to an embodiment.
- FIG. 3A is a diagram illustrating exemplary paths for virtual cameras approaching a connection between two panoramas at different angles, according to an embodiment.
- FIG. 3B is a diagram illustrating exemplary paths for virtual cameras leaving a connection between two panoramas at different angles, according to an embodiment.
- FIG. 3C is a diagram illustrating an exemplary path for a virtual camera aligning with a street across an intersection, according to an embodiment.
- FIG. 3D is a diagram illustrating an exemplary path for a virtual camera aligning with different streets in an intersection, according to an embodiment.
- FIG. 3E is another diagram illustrating an exemplary path for a virtual camera aligning with different streets in an intersection, according to an embodiment.
- FIG. 4 is a flowchart of a method for guided navigation of a virtual camera along a street network toward a target, according to an embodiment. - ⁇
- FIG. 5 is a diagram of an exemplary computer system in which embodiments can be implemented.
- FIGS. 6A-B are diagrams illustrating the intersection of multiple navigation channels, according to an embodiment.
- Embodiments relate to guided navigation in an interactive virtual three- dimensional environment. While the present invention is described herein with reference to illustrative embodiments for particular applications, it should be understood that embodiments are not limited thereto. Other embodiments are possible, and modifications can be made to the embodiments within the spirit and scope of the teachings herein and additional fields in which the embodiments would be of significant utility. Further, when a particular feature, structure, or characteristic is described in connection with an embodiment, it is submitted that it is within the knowledge of one skilled in the relevant art to effect such feature, structure, or characteristic in connection with other embodiments whether or not explicitly described.
- references to "one embodiment,” “an embodiment,” “an example embodiment,” etc. indicate that the embodiment described may include a particular feature, structure, or characteristic, but every embodiment may not necessarily include the particular feature, structure, or characteristic. Moreover, such phrases are not necessarily referring to the same embodiment. Further, when a particular feature, structure, or characteristic is described in connection with an embodiment, it is submitted that it is within the knowledge of one skilled in the art to effect such feature, structure, or characteristic in connection with other embodiments whether or not explicitly described.
- panoramic image and “panorama” are used herein to refer broadly and inclusively to any panoramic images that provide a 360-degree view of a real world scene.
- Panoramas or panoramic images may be stored in the form of a plurality of panoramic images or panoramic image tiles.
- Such panoramas may further include three- dimensional images, including panoramic three-dimensional images that can be presented on a display.
- a display can be any type of electronic display for viewing images or can be any type of rendering device adapted to view three-dimensional images.
- each panonoramic image may be associated with depth information corresponding to various points within a scene represented by the image. As will be described in further detail below, this depth information can be used to generate three-dimensional geometry for the panoramas.
- FIG. 1 is a diagram illustrating a system 100 for guided navigation along a street network, according to an embodiment.
- System 100 includes a client 102 communicatively coupled to a server 140 across a network 130.
- Network 130 can be any network or combination of networks that can carry data communication.
- Such network can include, but is not limited to, a local area network, medium area network, and/or wide area network such as the Internet.
- Client 102 can be any computing device with a processor, local memory, a display, and one or more input devices (e.g., a mouse, QWERTY keyboard, touch-screen, microphone, joystick, or T9 keyboard).
- a computing device can include, but is not limited to, a mobile phone, a personal digital assistant (PDA), a computer, a cluster of computers, a set-top box, or other similar type of device capable of processing instructions.
- Server 140 similarly, can be implemented using any general -purpose computer capable of serving data to client 102. Although only server 140 is shown, any number of servers may be used as necessary.
- client 102 includes a user interaction module 110 and a
- Tenderer module 122 User interaction module 1 10 includes a target module 1 12, a motion module 114, a path planner module 1 16, and a path motion module 118.
- Embodiments of user interaction module 1 10, renderer module 122, motion model 114, target module 1 12, path planner module 116, and path motion module 1 18 may be implemented in hardware, software, firmware, or any combination thereof.
- GIS geographic information system
- client-server computer architecture For ease of explanation and illustrative purposes, embodiments of system 100 and its components will be described in the context of a geographic information system (GIS) operating in a client-server computer architecture, but embodiments are not intended to be limited thereto.
- GIS geographic information system
- Such a GIS can be used to render and navigate photographic panoramas in a virtual three-dimensional environment from the perspective of a virtual camera, as will be described in further detail below.
- An example of such a GIS includes, but is not limited to, Google Earth from Google Inc. of Mountain View, CA.
- client 102 may include a geospatial browser (not shown) configured to provide an interface to the GIS.
- a geospatial browser can be a standalone client application executable by client 102 or operable as a plug-in of an web or Internet browser.
- server 140 is a GIS server configured to transmit GIS data corresponding to a geographic region of interest on a substantially spherical representation of the Earth.
- a request may include a bounded geographic search area in the form of a bounding box having latitude/longitude data points associated with a geographic location.
- server 140 may communicate with a database or one or more different servers to which server 140 is communicatively coupled over a network (e.g., network 130).
- GIS data may include both image and non-image data.
- Image data may include, but is not limited to, satellite imagery, digital maps, aerial photographs, and street-level photographs.
- Non- image data may include, for example, metadata associated with the imagery and photographs.
- the street-level photographs may include panoramas that present 360-degree views of real-world scenes.
- each panorama may include or be associated with metadata used to specify information associated with the panorama.
- metadata may include, but is not limited to, a position of the virtual camera (e.g., yaw, pitch, and roll), GPS coordinates, and a panorama identifier (e.g., a string value that identifies the panorama in the GIS).
- target module 112 determines a target location in the three-dimensional environment based on a current view direction and/or view position of the virtual camera in the three-dimensional environment.
- target module 112 accepts an input from a user that specifies a target location, such as an address or coordinates.
- target module 112 may receive position data representing a position on a viewport selected by a user, extend a ray based on the position selected by the user, and determine the target location to be an intersection of the ray with an object in the three-dimensional model.
- the target position may be associated with a center of a photographic panorama (e.g., panorama center 204 of FIG. 2, described further below) in the three-dimensional environment.
- Path motion module 118 moves the virtual camera in the three-dimensional environment toward the target location along the path and orients the virtual camera to face the target location as the virtual camera moves along the path. Path motion module 118 may use the path to prefetch geographic information from server 140.
- Motion model 114 constructs a view specification.
- the view specification defines the virtual camera's viewable volume within a three dimensional space, and the position and orientation of the frustum with respect, for example, to a three dimensional map.
- the viewable volume is defined by a view frustum (or simply "frustum"), which may be, for example, in the shape of a truncated pyramid.
- the frustum may have minimum and maximum view distances that can change depending on the current field of view.
- the view specification also changes.
- View specification may be stored, for example, in a local memory of client 102, where it is used by Tenderer module 122 to draw GIS data.
- the stored view specification specifies navigation parameters of the virtual camera.
- the navigation parameters can be used to specify the position and orientation of the virtual camera in three-dimensional space.
- Such parameters can include, but are not limited to, directional coordinates, (e.g., latitude, longitude, north, south, in-between), altitude and pitch/tilt (e.g., level, down, up, in- between), yaw/roll (e.g., level, tilt clockwise, tilt counterclockwise, in-between), horizontal field of view, and vertical field of view.
- the photographic panorama includes depth information corresponding to various points within a scene represented by the panoramic image, according to an embodiment.
- the depth information can include, for example, depth values that describe the proximity of the various points in the scene to a first position.
- the first position can be, for example, the position of an image capturing device (e.g., a specialized digital camera) used to capture the image.
- a surface of an object represented in the scene may be represented as a collection of points.
- Each point in turn, may be represented as a vector, whereby each point is stored with respect to its distance to the image capturing device, and its angle with respect to the direction in which such an image capturing device is pointed.
- the depth information may be collected in a variety of ways, including, but not limited to, using a laser range finder and image matching.
- camera arrangements employing two or more cameras, spaced slightly apart yet looking at the same scene, may be used.
- image matching is used to analyze slight differences between the images captured by each camera in order to determine the distance at each point in the images.
- the distance information may be compiled by using a single video camera, mounted on a vehicle and traveling at a particular velocity, to capture images of scenes as the vehicle moves forward.
- a laser range finder may also be used in combination with the camera taking the image.
- image matching the subsequent frames of the captured images may be compared to extract the different distances between the objects and the camera. For example, image objects located at a further distance from the camera position will stay in the frame longer than image objects located closer to the camera position.
- the depth information is sent from a server as a depth map comprising a grid of discrete values, where each element of the grid corresponds with a pixel of a two-dimensional image.
- the value of the depth map at each pixel may represent the distance from a first position to an image object.
- the value of the depth map at each pixel may represent the distance from a camera position, of the camera used to capture the image, and the image object represented in the image. It would be apparent to a person skilled in the relevant art given this description that any one of various file formats may be used for such a depth map.
- the depth map may be stored as an extensible markup language (XML) file.
- XML extensible markup language
- the depth map and other information associated with the panoramic image can be stored independently of the image itself.
- a set of panoramic images can be retrieved from server 140 based on a current field of view associated with the virtual three-dimensional environment. Further, depth values associated with each panoramic image can be used to generate three-dimensional geometry for each panorama.
- the generated three- dimensional geometry may be, for example and without limitation, a watertight three- dimensional mesh of polygons (e.g., triangles) that can be generated and rendered by renderer module 122 with photographic textures with the three-dimensional environment in a display area of a geospatial browser implemented in client 102, as noted above.
- the metadata of each panoramic image includes connectivity information that specifies a spatial link, connection, or path to one or more neighboring panoramas.
- user interaction module 1 10 constructs a directed graph of spatially linked panoramas (or simply a "panorama graph” or a "connectivity graph") based on the connectivity information included in the metadata of each panoramic image.
- a panorama graph may be stored in, for example, a local memory (e.g., a cache memory) at client 102.
- the panorama graph can be used to guide a user of the GIS along the links between different panoramas rendered in the three-dimensional environment.
- such links correspond to a street network in a virtual three-dimensional environment of the GIS and the panoramic images represent a series of different street- level views along the streets of the network.
- the panorama graph can represent a network of streets or roads (or "road network”) between neighboring panoramas.
- client 102 can include a graphical user interface
- GUI which can be used to provide navigation controls to enable a user to navigate along such a road network or panorama graph within the three-dimensional environment
- the GIS can provide controls that enable a user to navigate between various locations and viewpoints associated with the panoramic images.
- navigation controls may be rendered as a three- dimensional overlay with a textured three-dimensional model of the panoramic image.
- Such navigation controls can enable the user to change the virtual camera's view position and view direction within the three-dimensional environment to navigate between locations associated with the different panoramic images.
- the connectivity information i.e., the metadata
- the depth information i.e., the photographic textures associated with each panorama
- the photographic textures may be streamed in the form of image tiles at different resolutions based on a viewpoint of the virtual camera in the three-dimensional environment.
- the panoramas and any associated information may be streamed based on the viewpoint of the virtual camera.
- a panorama of a street level scene may present an accurate representation of the scene from only a single point-of-view.
- the best visual quality, or the viewpoint(s) that present the most accurate visual depiction of the scene represented in the panorama may be associated with a location of the virtual camera within the three- dimensional environment corresponding to the exact position where the panorama was taken, i.e., corresponding to the original position of the image capturing device used to capture the panoramic image.
- this position may correspond to the center of the panorama.
- the visual quality of a rendered panorama in the three- dimensional environment may degrade as the virtual camera moves away from the center of the panoramas.
- embodiments may be configured to prevent the virtual camera from moving to viewpoints of the panorama having poor image quality when rendered in the three-dimensional environment. Moreover, as will be described in further detail below, embodiments enable the virtual camera (and the user) to follow areas of good image quality.
- an area or region of "known-good,” sufficient, or acceptable visual quality for a panorama is based on one or more criteria associated with the visual representation of the panorama in the three-dimensional environment.
- the region of sufficient visual quality corresponds to a region of the panorama for which a display resolution exceeds a predetermined threshold value.
- a region may be based on points within the panorama at which the display resolution is of sufficient quality so as to avoid distortion of objects in the scene represented by the panoramic image.
- such a region of the panorama may correspond to the nearest determined location of a facade plane rendered for the panorama in the three-dimensional environment.
- a connectivity or panorama graph can be generated by user interaction module 110 using connectivity information associated with a panorama corresponding to a location in the three-dimensional environment.
- path planner module 116 uses the connectivity information associated with one or more panoramas to construct a navigation channel between the panoramas. Such a navigation channel is used to constrain the virtual camera to a certain volume of space around the nearest panoramas in the generated panorama graph.
- path planner module 116 can calculate a navigation radius or a roaming distance relative to a position of the virtual camera corresponding to the most visually accurate viewpoint from which a representation of the panorama may be viewed in the three-dimensional environment.
- a position may correspond to the position of an image capturing device used to capture the panorama (e.g., the center of a panorama), as mentioned previously.
- the navigation radius or roaming distance may be a constant or a function, and may depend on several factors including, but not limited to, the distance between the panoramas and quality of the depth mesh.
- path planner module 1 16 calculates the navigation radius for a panorama by evaluating the distance to the nearest plane to be rendered from the panorama center in the three-dimensional environment. It would be apparent to a person skilled in the relevant art given this description that other considerations can be used to limit the navigation radius.
- renderer module 122 may stretch the textures to fill areas for which extensive depth information may not be available.
- a consideration that may limit the navigation radius may be a predetermined threshold value corresponding to an acceptable amount of texture stretching that would be seen from remote locations.
- Path motion module 1 18 can then create a navigation or collision channel using the navigation radius of the panorama associated with the initial location of the virtual camera and the navigation radius of the panorama associated with the target location, as described in further detail below with respect to FIG. 2.
- FIG. 2 is a diagram of an exemplary navigation channel 200 within a depth mesh
- a single connection links two panoramas along a path 210.
- Path 210 may represent, for example, a street in the three-dimensional environment. As shown in FIG. 2, path 210 extends from an initial position corresponding to a panorama center 204 (e.g., the center of the first panorama) to a target position corresponding to a panorama center 206 (e.g., the center of the second panorama).
- path motion module 1 18 may be used to create a shell
- path planner module 1 16 determines path 210 along a representation of a street in the three- dimensional environment.
- path planner module 116 of FIG. 1 may be used to determine a path 210 along a representation of a street between panorama center 204 and a panorama center 206 in the three-dimensional environment.
- Shell 220 of the navigation channel illustrated in FIG. 2 provides a constraint surface used for rendering portions of panoramic images in the three-dimensional environment.
- the constraint surface of the navigation channel defines a boundary beyond which navigation (e.g., movement of the virtual camera) in the three-dimensional environment is not allowed. Accordingly, this surface may be used as a hard-boundary for navigation in order to maintain the visual quality of the experience.
- the constraint surface can be updated by path motion module 118 to account for the immediate neighborhood of the virtual camera.
- depth mesh 202 may be a three-dimensional watertight mesh of polygons generated by renderer module 122 based on the depth maps associated with the two panoramas, as described above.
- path motion module 118 builds the shell by interpolating navigation spheres at each panorama along path 210 based on the calculated navigation radius of each panorama, as described above. For a visual example, if shell 220 were to be drawn in three-dimensional space and viewed from above, it would appear as a conical section, bounded by two spheres. Also as described above, each navigation sphere represents a region of "known-good" or acceptable viewpoints for each panorama within the three-dimensional environment.
- each panorama is geo-located.
- the locations of the panoramas in the virtual three-dimensional environment can correspond to geographic locations.
- movement of the virtual camera (e.g., based on user input) between different locations in the three-dimensional environment can correspond to movement between geographic locations in the GIS.
- FIG. 2 is presented for illustrative purposes only and that embodiments are not limited thereto.
- embodiments may be used for multiple panoramas linked together in a panorama graph.
- multiple panoramas may be associated with a representation of an intersection in the three-dimensional environment.
- the navigation channels of such multiple panoramas may intersect in unpredictable ways. Therefore, path motion module 1 18 may perform a series of collision tests using all possible navigation channels that are, for example, a predetermined number of links deep in the panorama or connectivity graph from the panorama being currently rendered (or "active panorama").
- path motion module 1 18 may construct the bounding geometry associated with a navigation channel for up to two panoramas from the active panorama.
- FIGS. 6A-B are diagrams 600 A and 600B, respectively, illustrating the intersection of multiple navigation channels 601, 602, and 603, according to an embodiment.
- a shell having multiple branches is generated based on navigation channels 601-603.
- each navigation channel represents each of three streets connected to the panorama corresponding to panorama center 620.
- navigation channels 601-603 can include navigation fillets 614, 615, and 616, which act as "collision-free zones" as the virtual camera moves from one navigation or collision channel to another.
- navigation fillets 614-616 have filleted edges (e.g., rounded comers). Such filleted edges can help to avoid the sharp corners associated with intersections, thereby providing for a smooth user experience during navigation. It is noted that embodiments may utilize various novel equations/algorithms for generating these rounded corners quickly.
- navigation fillets 614-616 are constructed by fitting a collision sphere 630 tangentially between two different navigation channels (e.g., navigation channels 61 1 and 612).
- navigation fillet 614 may be parameterized according to the distances of points of tangency relative to panorama center 620.
- the fillet width ("fw") of navigation fillet 614 may be computed, for example, by taking into account the navigation widths ("nw") corresponding to navigation channels 601 and 602.
- path motion module 118 can test for collision against the union of the constraint surfaces created by the navigation channels and fillets, as shown in diagram 600B of FIG. 6B, according to an embodiment.
- a union of constraint surfaces provided by the navigation channels in diagram 600B may represent the region of sufficient or acceptable visual quality, as described above, to which movement of the virtual camera may be constrained.
- path planner module 116 can determine the path to turn at an intersection in a representation of a street network in the three-dimensional environment. For example, at the turn, path planner module 1 16 may interpolate a spline at the turn in the path to smooth motion of the virtual camera. Guided Navigation Using Navigation Channel Constraints
- embodiments enable the virtual camera (and the user) to follow areas of good image quality as three-dimensional representations of panoramas are rendered in the three-dimensional environment.
- the panoramas provide ground level views of various scenes along one or more streets.
- the movement of the virtual camera follows representations of the streets in the three- dimensional environment.
- path motion module 118 is further configured to facilitate such navigation based on user input, e.g., keyboard and/or mouse events (e.g., up arrow, right arrow, double-click, etc.). Consequently, path motion module 1 18 guides the virtual camera on a path along a representation of a street while alleviating any undue burden for the user to try to stay on-track by using navigation controls and/or additional user input.
- user interaction module 1 10 is operable to change the meaning associated with certain user input (e.g., the arrow keys of a keyboard). This enables embodiments to subtly guide the user along roads when, for example, a "follow-roads" gesture is detected, as described in more detail below.
- motion module 114 and/or path motion module 118 can be configured to apply corrections to the motion of the virtual camera.
- a view direction correction and a view position correction are applied to the motion of the virtual camera as it is moved along the path.
- the view position correction moves the virtual camera towards the center of the link between the panoramas.
- the view direction correction aligns the view of the virtual camera slightly more towards the center of the path. Further, once the gesture has been activated for a predetermined period of time, the view of the virtual camera can be moved back to the center of the path. The application of such view corrections enables the virtual camera to follow curving road networks.
- follow roads may be activated by a user in a number of ways.
- the user may continue to hold down, for example, the up-arrow key for a period of time without touching the left or right arrow keys, while a view direction of the virtual camera roughly corresponds to a path along the road direction.
- a view direction of the virtual camera roughly corresponds to a path along the road direction.
- the view direction of the virtual camera is along a street represented in the three- dimensional environment within a certain angular tolerance (e.g., 45 degrees)
- the "follow roads" gesture may be activated, and path motion module 118 can determine the intention of the user may be to move along the street.
- the magnitude of the view direction and/or view position correction applied by path motion module 1 18 can be smoothly parameterized based on user input. For example, a predetermined threshold value based on time may be used for such parameterization. In this example, when the "follow-roads" gesture is detected, the magnitude of any corrections may be smoothly ramped up over a few second to some predetermined maximum correction amount. Further, conflicting user input or camera positioning may cause a smooth ramp-down of any correction amount.
- path motion module 118 Upon activation of the gesture, path motion module 118 gradually aligns the view of the virtual camera towards the road, and moves the user towards the center of the road, in accordance with an embodiment. From the user's perspective, this experience may feel similar to driving a car. In an embodiment, path motion module 1 18 will automatically follow the curves in the road network represented in the three-dimensional environment. A user may counteract any automatic turns of the virtual camera by, for example, hitting the left-right arrow key. In this case, path motion module 118 can discontinue the automatic navigation and wait for some predetermined period of time before starting to correct the view. It would be apparent to a person skilled in the relevant art given this description that embodiments may use any number of methods to infer or predict the intention of the user to follow roads and thus, whether such a "follow roads" gesture should be activated.
- path motion module 1 18 attempts to determine the intention of the user to turn a corner based on the view position and angle of the virtual camera.
- path planner module 116 determines a specific path which will execute the turn onto the new street, facing the new street direction, at the center of the road.
- path planner module 116 chooses the appropriate road and builds a new corresponding path dynamically, while also trying to detect whether the user wishes to discontinue to follow roads, as described above.
- the detected user input e.g., a long key-press
- path planner module 116 may also determine whether there is a panorama center (or region of increased visual quality, as described above) along the previous direction of motion of the virtual camera. Such a previous direction of motion may be, for example, within a certain tolerance based on the position of the panorama center (or region) and/or time to arrive at that position. If one is found, a short autopilot can be initiated such that the virtual camera remains moving close to the original direction for a short amount of time. This allows the view to arrive at a region of better visual quality. To the user, this may seem like a natural momentum decay or a "throw" gesture that may be found in some other well-known navigation systems, as would be apparent to a person skilled in the relevant art given this description. However, unlike these other systems, one of the goals for embodiments as described herein is to subtly guide the user towards the region of increased visual fidelity or quality, as described above.
- FIGS. 3A-E are diagrams 300A-E illustrating various possible use cases associated with subtly guiding a virtual camera (and user) towards and/or along street centers.
- Each of the diagrams shown in FIGS. 3A-E includes a road nework of panoramas associated with an intersection. Further, each diagram includes a navigation channel radius 302 for each of the panoramas. For ease of explanation, navigation channel widths are not shown.
- the view and path direction of a virtual camera is in the direction of an arrow associated with the virtual camera.
- a user may, for example, execute a "Move Forward" action (e.g., by selecting an arrow key, as described above).
- embodiments may employ any number of novel techniques to implement the desired movement of the virtal camera in the three-dimensional environment, as illustrated by diagrams 300A-E.
- such techniques may involve constructing various curves to get to each intersection point and comparing the relative tension associated with these curves.
- FIG. 3A is a diagram 300A illustrating exemplary paths of a virtual camera 310 and a virtual camera 320 approaching a street 312 (represented as a centerline of the intersection in the example shown in FIGS. 3A-E) between two panoramas at different angles, according to an embodiment.
- the path of virtual camera 310 is approaching street 312 connecting a panorama center 301 and a panorama center 303 at a grazing angle.
- this view direction may indicate that the user is interested in moving to a target position down street 312 rather than moving to a position across street 312. If no action is taken in regard to the movement of virtual camera 310, it would move past street 312, and navigate into an undesired location.
- a correction can therefore be taken to align the path of virtual camera 310 with street 312 in an "intercept course." If the user actually prefers to navigate past the street 312 to a specific target, the user can override the correction by using, for example, arrow keys on the user's keyboard or turning the view of virtual camera 310 to be more perpendicular to street 312, similar to the view direction of virtual camera 320, described below.
- virtual camera 320 In contrast to virtual camera 310, virtual camera 320 is approaching street 312 between two panoramas represented by panorama centers 304 and 305, but at a nearly perpendicular angle. In this case, the user may be interested in viewing a target 322 (e.g., a facade of a building 306) located across the street from the viewpoint. If the path of virtual camera 320 were to be corrected so as to place it on an intercept course, as with virtual camera 310, the result may be more jarring to the user than is desirable. Thus, the view angle for virtual camera 320 should not be corrected so as to be aligned along street 312. If the user actually does prefer to move down street 312, the user can align the view direction of virtual camera 320 similar to that of virtual camera 310 by manually overriding guided navigation through user input.
- a target 322 e.g., a facade of a building 306
- FIG. 3B is a diagram 300B illustrating exemplary paths of a virtual camera 330 and a virtual camera 340 moving away from street 312, at different angles, according to an embodiment.
- the path for virtual camera 340 is similar to that of virtual camera 320 of FIG. 3 A (i.e., toward a target location 342).
- the path for virtual camera 330 initially deviates from street 312 (the centerline) at a relatively small angle (e.g., as shown by initial path 331).
- the path of virtual camera 330 is corrected so that it aligns along street 312. if the user is interested in a building outside the navigation channel, they can override the correction by inputting, for example, the appropriate keystrokes or by turning the viewpoint to a more perpendicular angle with the road via, for example, navigation controls.
- FIG. 3C is a diagram 300C illustrating an exemplary path for a virtual camera 350 aligning with a street 352 across an intersection, according to an embodiment. It may be assumed that the virtual camera 350 's view-direction subtends an equal angle with respect to both streets 312 and 352. Although it may seem obvious from diagram 300C that the path of virtual camera 350 should be aligned with street 352, the angle of incidence of the path illustrated in the diagram may not be sufficient for alignment with street 352. Rather, given a fixed forward velocity, the street (352) along which virtual camera 350 should be aligned requires a minimum rate of turn, even if it requires a greater total angular deviation. Thus, rate-of-turn metrics may be utilized to select the appropriate street for path alignment based on the street that produces the minimum rate of turn relative to other streets.
- FIGS. 3D and 3E elucidate the interaction of navigation channel radius 302 with the guided navigation correction.
- FIG. 3D is a diagram 300D illustrating an exemplary path for a virtual camera 360 aligning with different streets in an intersection, according to an embodiment.
- FIG. 3E is a diagram 300E illustrating an exemplary path for a virtual camera 370 aligning with different streets in an intersection, according to an embodiment.
- the target paths for both correspond to street 312, which is located at a greater distance (e.g., relative to the distance of the initial position to street 352) from the initial position of each virtual camera as shown in diagrams 300D and 300E, respectively.
- the path for virtual camera 360 is corrected to the farther street (312) based on its current view direction.
- the path for virtual camera 370 is not corrected to target this farther street and is instead guided along street 352.
- the difference between the path of virtual camera 360 and the path of virtual camera 370 is the allowed distance deviation from the navigation channel radius 302.
- virtual camera 370 For example, if virtual camera 370 were to be corrected to street 312, then the viewport used to display the three-dimensional environment from the perspective of the virtual camera would deviate outside of the allowed navigation channel (e.g., as illustrated by navigation channel 200 of FIG. 2, described above). Navigation outside the navigation channel may result in poor or undesirable views that may be disallowed. Consequently, virtual camera 370 is corrected to street 352, which is closer in distance to this virtual camera's initial position.
- the allowed navigation channel e.g., as illustrated by navigation channel 200 of FIG. 2, described above.
- the virtual camera may have an initial position corresponding to an aerial perspective of the Earth.
- the user may select a target corresponding to a geographical location on the Earth by selecting an appropriate position within three-dimensional environment.
- the virtual camera may then incrementally zoom down to a lower altitude based on the selected target.
- a threshold altitude near ground- level may be reached that can be used to infer that the user intends to view ground-level objects up close.
- near ground-level where satellite imagery may become blurry or less useful
- factors may be taken into account to infer such user intent. Such factors can include, but are not limited to, proximity to the ground, the motion of the virtual camera, and user input (e.g., mouse movements).
- path motion module 118 maintains spatial awareness throughout the transition from the aerial perspective to the ground-level perspective and places the virtual camera in a useful position that maintains a maximum amount of visual context.
- path planner module 1 16 may automatically place the virtual camera (e.g., via an autopilot path) in an initial position corresponding to the center of a panorama in order to ensure that maximum visual quality is maintained.
- image objects e.g., a building
- a simple autopilot path to that position may not be sufficient.
- the connectivity or panorama graph can be used to reorient the view of the virtual camera to look along the road direction to a nearby panorama whose view is not obstructed by image objects. Looking along the road maximizes the visual continuity between the aerial "Earth" view and the street or ground level view rendering experiences while providing the user more visual context. Further, the reorientation of the virtual camera performed by path motion module 1 18 can occur graudually so as not to disorient the user.
- autopilot navigation can be immediately initiated to a target location in the three-dimensional environment determined by, for example, target module 112.
- motion model 114 can initiate a set of fetches or requests to server 140 over network 130 for panorama data associated with the target location, including panorama data for locations near the target location.
- target module 112 may determine the nearest panorama to the target location, and further choose the determined panorama as the autopilot destination. In an example, the selection of the panorama may be based on whether it is located within a certain distance threshold of the determined target location.
- the selection of the best panorama to be displayed for a target location may be determined by weighing several factors including, but not limited to, the proximity of the target position to the location associated with the panorama in the three- dimensional environment and the angular deviation of the virtual camera to face the target from the current position.
- Path planner module 116 can then calculate the street direction(s) from the panorama connectivity information associated with the panorama and received from server 140. Based on the current view direction, path planner module 116 can choose the closest street and the proper view direction down a street represented in the three- dimensional environment so as to minimize the angular rotation applied to the view. As soon as this information is known, path motion module 118 can begin to rotate the view of the virtual camera to align with the street.
- embodiments can also be confiured to handle any additional complications that may arise during this process. For example, subsequent outstanding fetches for panoramas may reveal a closer panorama than initially determined. In another example, another street may exist that was previously unknown to user interaction module 110. Therefore, embodiments of user interaction module 110, including its components, are operable to dynamically update the autopilot destination and orientation so as to account for such complications without disorienting the user.
- FIG. 4 is a flowchart illustrating a method 400 for guided navigation of a virtual camera along a street network toward a target, according to an embodiment.
- method 400 is described with respect to system 100 of FIG. 1, described above. However, method 400 is not intended to be limited thereto.
- Method 400 begins by determining a target at a step 402.
- the target may be any point or region in the three-dimensional environment.
- the three- dimensional environment may include a plurality of models and the target may be a model or a position on a model.
- a user may specify a target location.
- a user may enter in an address or coordinates to specify a location in the three dimensional environment.
- the location may be at ground- or street-level.
- a user may select a position in a viewport displaying the three- dimensional environment from the perspective of the virtual camera.
- the GIS may receive position data representing a position on a viewport selected by a user.
- the GIS may determine a point in the three-dimensional environment based on the position selected by the user and a focal length of the virtual camera.
- the distance between the determined point and the focal point may correspond to the focal length of the virtual camera.
- the GIS may extend a ray based on the position selected by the user.
- the GIS may extend the ray from a focal point or entrance pupil of the virtual camera.
- the ray may extend from the focal point or entrance pupil through the determined point.
- the target location may be determined based on the ray.
- the three-dimensional model may include a plurality of models, such as building models.
- the target location may be determined to be the intersection of the ray with a model in the three-dimensional environment. In this way, a target location may be determined based on a user selection in a viewport.
- a navigation channel relative to a path to the target location is generated at step 404.
- the path may be, for example, along a street network represented in the three-dimensional environment. Further, the path may be between locations corresponding to centers of panoramas rendered as three- dimensional models in the three-dimensional environment, as described above.
- the navigation channel defines a constraint surface and navigation boundary for the movement of a virtual camera within the three-dimensional environment, also as described above.
- a panoramic photograph may be located at each point in the series of points, and the panoramic photographs may be linked to each other in a panorama or connectivity graph based on connectivity information associated with each panorama. For example, when the virtual camera is located at a particular point, the GIS may display the photographic image data from the perspective of the virtual camera.
- the path may guide the virtual camera close to the target location.
- the path may guide the virtual camera to the position in the street network closest to the target location.
- the path may be determined such that it does not exceed a maximum distance (e.g., a maximum roaming distance relative to a center of a panorama).
- the virtual camera As the virtual camera is moved along the path within the navigation channel, it may be oriented to face the target location in step 406.
- the virtual camera may be automatically guided within the navigation channel toward the target location based on several factors including, but not limited to, a view direction and position of the virtual camera in the three-dimensional environment, as described above. Further, a user may override any automatic guidance through user input, also as described above.
- step 408 further includes selecting a panorama to be displayed based on the target location.
- the selection of the appropriate panorama may not simply be based solely on the proximity of a panorama to the target location. Another important factor may be, for example, the viewing angle at which the virtual camera must be moved in order to keep the target in view.
- the closest panorama may be further along the road than the target; however, to keep the target in view, the virtual camera would have to swing around and face nearly backwards.
- the selected panorama is the one closest to the virtual camera's initial position, or origin, but also located far enough from the target location in the three-dimensional environment such that the amount of camera rotation is minimized.
- Embodiments shown in FIGS. 1-4, or any part(s) or function(s) thereof, may be implemented using hardware, software modules, firmware, tangible computer readable media having instructions stored thereon, or a combination thereof and may be implemented in one or more computer systems or other processing systems.
- FIG. 5 illustrates an example computer system 500 in which embodiments, or portions thereof, may be implemented as computer-readable code.
- client 102 in FIG. 1 can be implemented in computer system 500 using hardware, software, firmware, tangible computer readable media having instructions stored thereon, or a combination thereof and may be implemented in one or more computer systems or other processing systems.
- Hardware, software, or any combination of such may embody any of the modules and components in FIGS. 1-4.
- programmable logic may execute on a commercially available processing platform or a special purpose device.
- programmable logic may execute on a commercially available processing platform or a special purpose device.
- One of ordinary skill in the art may appreciate that embodiments of the disclosed subject matter can be practiced with various computer system configurations, including multi-core multiprocessor systems, minicomputers, mainframe computers, computers linked or clustered with distributed functions, as well as pervasive or miniature computers that may be embedded into virtually any device.
- processor device may be a single processor, a plurality of processors, or combinations thereof.
- processor devices may have one or more processor "cores.”
- Processor device 504 may be a special purpose or a general purpose processor device.
- processor device 504 may also be a single processor in a multi-core/multiprocessor system, such system operating alone, or in a cluster of computing devices operating in a cluster or server farm.
- Processor device 504 is connected to a communication infrastructure 506, for example, a bus, message queue, network, or multi-core message-passing scheme.
- Computer system 500 also includes a main memory 508, for example, random access memory (RAM), and may also include a secondary memory 510.
- Secondary memory 510 may include, for example, a hard disk drive 512 and a removable storage drive 514.
- Removable storage drive 514 may comprise a floppy disk drive, a magnetic tape drive, an optical disk drive, a flash memory, or the like.
- the removable storage drive 514 reads from and/or writes to a removable storage unit 518 in a well known manner.
- Removable storage unit 518 may comprise a floppy disk, magnetic tape, optical disk, etc. which is read by and written to by removable storage drive 514.
- removable storage unit 518 includes a computer usable storage medium having stored therein computer software and/or data.
- secondary memory 510 may include other similar means for allowing computer programs or other instructions to be loaded into computer system 500.
- Such means may include, for example, a removable storage unit 522 and an interface 520.
- Examples of such means may include a program cartridge and cartridge interface (such as that found in video game devices), a removable memory chip (such as an EPROM, or PROM) and associated socket, and other removable storage units 522 and interfaces 520 which allow software and data to be transferred from the removable storage unit 522 to computer system 500.
- Computer system 500 may also include a communications interface 524.
- Communications interface 524 allows software and data to be transferred between computer system 500 and external devices.
- Communications interface 524 may include a modem, a network interface (such as an Ethernet card), a communications port, a PCMCIA slot and card, or the like.
- Software and data transferred via communications interface 524 may be in the form of signals, which may be electronic, electromagnetic, optical, or other signals capable of being received by communications interface 524. These signals may be provided to communications interface 524 via a communications path 526.
- Communications path 526 carries signals and may be implemented using wire or cable, fiber optics, a phone line, a cellular phone link, an RF link or other communications channels.
- Computer program medium and “computer usable medium” are used to generally refer to media such as removable storage unit 518, removable storage unit 522, and a hard disk installed in hard disk drive 512.
- Computer program medium and computer usable medium may also refer to memories, such as main memory 508 and secondary memory 510, which may be memory semiconductors (e.g. DRAMs, etc.).
- Computer programs are stored in main memory 508 and/or secondary memory 510. Computer programs may also be received via communications interface 524. Such computer programs, when executed, enable computer system 500 to implement embodiments as discussed herein. In particular, the computer programs, when executed, enable processor device 504 to implement the processes of embodiments of the present invention, such as the stages in the methods illustrated by flowchart 400 of FIG. 4, discussed above. Accordingly, such computer programs represent controllers of the computer system 500. Where embodiments are implemented using software, the software may be stored in a computer program product and loaded into computer system 500 using removable storage drive 514, interface 520, hard disk drive 512, or communications interface 524.
- Embodiments of the invention also may be directed to computer program products comprising software stored on any computer useable medium. Such software, when executed in one or more data processing device, causes a data processing device(s) to operate as described herein.
- Embodiments of the invention employ any computer useable or readable medium. Examples of computer useable mediums include, but are not limited to, primary storage devices (e.g., any type of random access memory), secondary storage devices (e.g., hard drives, floppy disks, CD ROMS, ZIP disks, tapes, magnetic storage devices, optical storage devices, MEMS, nano-technological storage device, etc.), and communication mediums (e.g., wired and wireless communications networks, local area networks, wide area networks, intranets, etc.).
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Computer Graphics (AREA)
- General Physics & Mathematics (AREA)
- Computer Hardware Design (AREA)
- General Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Computing Systems (AREA)
- Geometry (AREA)
- Processing Or Creating Images (AREA)
- Instructional Devices (AREA)
Abstract
Description
Claims
Priority Applications (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
AU2011332885A AU2011332885B2 (en) | 2010-11-24 | 2011-11-22 | Guided navigation through geo-located panoramas |
EP11799904.5A EP2643822B1 (en) | 2010-11-24 | 2011-11-22 | Guided navigation through geo-located panoramas |
JP2013541029A JP5899232B2 (en) | 2010-11-24 | 2011-11-22 | Navigation with guidance through geographically located panoramas |
CA2818695A CA2818695C (en) | 2010-11-24 | 2011-11-22 | Guided navigation through geo-located panoramas |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US41712010P | 2010-11-24 | 2010-11-24 | |
US61/417,120 | 2010-11-24 |
Publications (2)
Publication Number | Publication Date |
---|---|
WO2012071445A2 true WO2012071445A2 (en) | 2012-05-31 |
WO2012071445A3 WO2012071445A3 (en) | 2012-07-19 |
Family
ID=45401159
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/US2011/061906 WO2012071445A2 (en) | 2010-11-24 | 2011-11-22 | Guided navigation through geo-located panoramas |
Country Status (7)
Country | Link |
---|---|
US (1) | US8823707B2 (en) |
EP (1) | EP2643822B1 (en) |
JP (1) | JP5899232B2 (en) |
AU (1) | AU2011332885B2 (en) |
CA (1) | CA2818695C (en) |
DE (1) | DE202011110924U1 (en) |
WO (1) | WO2012071445A2 (en) |
Cited By (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2014210158A3 (en) * | 2013-06-28 | 2015-07-02 | Microsoft Corporation | Space carving based on human physical data |
CN105741314A (en) * | 2014-12-30 | 2016-07-06 | 诺基亚技术有限公司 | Methods And Apparatuses For Directional View In Panoramic Content |
US9631932B2 (en) | 2015-06-05 | 2017-04-25 | Nokia Technologies Oy | Crowd sourced interaction of browsing behavior in a 3D map |
Families Citing this family (69)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP5652097B2 (en) | 2010-10-01 | 2015-01-14 | ソニー株式会社 | Image processing apparatus, program, and image processing method |
US9848290B2 (en) * | 2011-03-10 | 2017-12-19 | Aruba Networks, Inc. | Location based computerized system and method thereof |
WO2013023063A1 (en) | 2011-08-09 | 2013-02-14 | Path 36 Llc | Digital media editing |
US9121724B2 (en) * | 2011-09-30 | 2015-09-01 | Apple Inc. | 3D position tracking for panoramic imagery navigation |
US20130141433A1 (en) * | 2011-12-02 | 2013-06-06 | Per Astrand | Methods, Systems and Computer Program Products for Creating Three Dimensional Meshes from Two Dimensional Images |
WO2013109976A1 (en) | 2012-01-20 | 2013-07-25 | Thermal Imaging Radar, LLC | Automated panoramic camera and sensor platform with computer and optional power supply |
US9396577B2 (en) * | 2012-02-16 | 2016-07-19 | Google Inc. | Using embedded camera parameters to determine a position for a three-dimensional model |
EP2677347A3 (en) * | 2012-06-02 | 2014-05-14 | Schlumberger Technology B.V. | Spatial data services |
US9218685B2 (en) * | 2012-06-05 | 2015-12-22 | Apple Inc. | System and method for highlighting a feature in a 3D map while preserving depth |
US10139985B2 (en) | 2012-06-22 | 2018-11-27 | Matterport, Inc. | Defining, displaying and interacting with tags in a three-dimensional model |
US10163261B2 (en) * | 2014-03-19 | 2018-12-25 | Matterport, Inc. | Selecting two-dimensional imagery data for display within a three-dimensional model |
US9786097B2 (en) * | 2012-06-22 | 2017-10-10 | Matterport, Inc. | Multi-modal method for interacting with 3D models |
US8928666B2 (en) * | 2012-10-11 | 2015-01-06 | Google Inc. | Navigating visual data associated with a point of interest |
US9305371B2 (en) | 2013-03-14 | 2016-04-05 | Uber Technologies, Inc. | Translated view navigation for visualizations |
US9712746B2 (en) | 2013-03-14 | 2017-07-18 | Microsoft Technology Licensing, Llc | Image capture and ordering |
US20140267587A1 (en) * | 2013-03-14 | 2014-09-18 | Microsoft Corporation | Panorama packet |
CN105074789B (en) | 2013-04-09 | 2019-03-12 | 热成像雷达有限责任公司 | Fire detection system |
WO2014169061A1 (en) | 2013-04-09 | 2014-10-16 | Thermal Imaging Radar, Llc. | Stepper motor control and fire detection system |
US9165397B2 (en) * | 2013-06-19 | 2015-10-20 | Google Inc. | Texture blending between view-dependent texture and base texture in a geographic information system |
CN108495051B (en) | 2013-08-09 | 2021-07-06 | 热成像雷达有限责任公司 | Method for analyzing thermal image data using multiple virtual devices and method for associating depth values with image pixels |
US9652667B2 (en) | 2014-03-04 | 2017-05-16 | Gopro, Inc. | Automatic generation of video from spherical content using audio/visual analysis |
US9392212B1 (en) | 2014-04-17 | 2016-07-12 | Visionary Vr, Inc. | System and method for presenting virtual reality content to a user |
US9189839B1 (en) | 2014-04-24 | 2015-11-17 | Google Inc. | Automatically generating panorama tours |
GB2526263B (en) * | 2014-05-08 | 2019-02-06 | Sony Interactive Entertainment Europe Ltd | Image capture method and apparatus |
US9002647B1 (en) | 2014-06-27 | 2015-04-07 | Google Inc. | Generating turn-by-turn direction previews |
US9418472B2 (en) * | 2014-07-17 | 2016-08-16 | Google Inc. | Blending between street view and earth view |
US9685194B2 (en) | 2014-07-23 | 2017-06-20 | Gopro, Inc. | Voice-based video tagging |
US10074013B2 (en) | 2014-07-23 | 2018-09-11 | Gopro, Inc. | Scene and activity identification in video summary generation |
US9734870B2 (en) | 2015-01-05 | 2017-08-15 | Gopro, Inc. | Media identifier generation for camera-captured media |
US9679605B2 (en) | 2015-01-29 | 2017-06-13 | Gopro, Inc. | Variable playback speed template for video editing application |
EP3065108A1 (en) * | 2015-03-04 | 2016-09-07 | Samsung Electronics Co., Ltd. | Displaying a virtual tour |
US10366509B2 (en) | 2015-03-31 | 2019-07-30 | Thermal Imaging Radar, LLC | Setting different background model sensitivities by user defined regions and background filters |
USD776181S1 (en) | 2015-04-06 | 2017-01-10 | Thermal Imaging Radar, LLC | Camera |
WO2016187235A1 (en) | 2015-05-20 | 2016-11-24 | Gopro, Inc. | Virtual lens simulation for video and photo cropping |
CN105761302A (en) * | 2015-05-29 | 2016-07-13 | 华北电力大学 | Three-dimensional digital navigation channel system and application thereof |
US9665170B1 (en) | 2015-06-10 | 2017-05-30 | Visionary Vr, Inc. | System and method for presenting virtual reality content to a user based on body posture |
KR101835434B1 (en) * | 2015-07-08 | 2018-03-09 | 고려대학교 산학협력단 | Method and Apparatus for generating a protection image, Method for mapping between image pixel and depth value |
US10204273B2 (en) | 2015-10-20 | 2019-02-12 | Gopro, Inc. | System and method of providing recommendations of moments of interest within video clips post capture |
US9721611B2 (en) | 2015-10-20 | 2017-08-01 | Gopro, Inc. | System and method of generating video from video clips based on moments of interest within the video clips |
US11627199B2 (en) * | 2016-01-05 | 2023-04-11 | Schneider Electric USA, Inc. | System and methods for creating a geospatial network model in a client environment |
US10109319B2 (en) | 2016-01-08 | 2018-10-23 | Gopro, Inc. | Digital media editing |
US9812175B2 (en) | 2016-02-04 | 2017-11-07 | Gopro, Inc. | Systems and methods for annotating a video |
US9838730B1 (en) | 2016-04-07 | 2017-12-05 | Gopro, Inc. | Systems and methods for audio track selection in video editing |
US9794632B1 (en) | 2016-04-07 | 2017-10-17 | Gopro, Inc. | Systems and methods for synchronization based on audio track changes in video editing |
US9838731B1 (en) | 2016-04-07 | 2017-12-05 | Gopro, Inc. | Systems and methods for audio track selection in video editing with audio mixing option |
US10185891B1 (en) | 2016-07-08 | 2019-01-22 | Gopro, Inc. | Systems and methods for compact convolutional neural networks |
JP7127539B2 (en) * | 2016-08-05 | 2022-08-30 | ソニーグループ株式会社 | Image processing device and image processing method |
US9836853B1 (en) | 2016-09-06 | 2017-12-05 | Gopro, Inc. | Three-dimensional convolutional neural networks for video highlight detection |
US10284809B1 (en) | 2016-11-07 | 2019-05-07 | Gopro, Inc. | Systems and methods for intelligently synchronizing events in visual content with musical features in audio content |
US10262639B1 (en) | 2016-11-08 | 2019-04-16 | Gopro, Inc. | Systems and methods for detecting musical features in audio content |
JP6849430B2 (en) * | 2016-12-27 | 2021-03-24 | キヤノン株式会社 | Image processing equipment, image processing methods, and programs |
US10762653B2 (en) * | 2016-12-27 | 2020-09-01 | Canon Kabushiki Kaisha | Generation apparatus of virtual viewpoint image, generation method, and storage medium |
US10534966B1 (en) | 2017-02-02 | 2020-01-14 | Gopro, Inc. | Systems and methods for identifying activities and/or events represented in a video |
US10127943B1 (en) | 2017-03-02 | 2018-11-13 | Gopro, Inc. | Systems and methods for modifying videos based on music |
US10185895B1 (en) | 2017-03-23 | 2019-01-22 | Gopro, Inc. | Systems and methods for classifying activities captured within images |
US10235798B2 (en) * | 2017-03-24 | 2019-03-19 | Mz Ip Holdings, Llc | System and method for rendering shadows for a virtual environment |
US10083718B1 (en) | 2017-03-24 | 2018-09-25 | Gopro, Inc. | Systems and methods for editing videos based on motion |
US10187690B1 (en) | 2017-04-24 | 2019-01-22 | Gopro, Inc. | Systems and methods to detect and correlate user responses to media content |
JP6796197B2 (en) * | 2017-04-28 | 2020-12-02 | 株式会社ソニー・インタラクティブエンタテインメント | Information processing equipment, information processing methods and programs |
US10540054B2 (en) * | 2017-09-20 | 2020-01-21 | Matterport, Inc. | Navigation point selection for navigating through virtual environments |
US10574886B2 (en) | 2017-11-02 | 2020-02-25 | Thermal Imaging Radar, LLC | Generating panoramic video for video management systems |
US10859389B2 (en) * | 2018-01-03 | 2020-12-08 | Wipro Limited | Method for generation of a safe navigation path for a vehicle and system thereof |
CN109002165B (en) * | 2018-07-16 | 2021-03-16 | 中国计量大学 | Virtual space real walking guide system with space positioning device |
US10818077B2 (en) | 2018-12-14 | 2020-10-27 | Canon Kabushiki Kaisha | Method, system and apparatus for controlling a virtual camera |
JP7468523B2 (en) * | 2019-06-05 | 2024-04-16 | ソニーグループ株式会社 | MOBILE BODY, POSITION ESTIMATION METHOD, AND PROGRAM |
WO2021015982A1 (en) * | 2019-07-22 | 2021-01-28 | Interdigital Vc Holdings, Inc. | A method and apparatus for delivering a volumetric video content |
CN110458871A (en) * | 2019-08-14 | 2019-11-15 | 上海霁目信息科技有限公司 | The method for registering of model and panorama sketch, system, equipment and medium and map |
US11601605B2 (en) | 2019-11-22 | 2023-03-07 | Thermal Imaging Radar, LLC | Thermal imaging camera device |
CN112802206B (en) * | 2021-02-07 | 2022-10-14 | 北京字节跳动网络技术有限公司 | Roaming view generation method, device, equipment and storage medium |
Citations (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20080291217A1 (en) | 2007-05-25 | 2008-11-27 | Google Inc. | Viewing and navigating within panoramic images, and applications thereof |
Family Cites Families (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6522325B1 (en) * | 1998-04-02 | 2003-02-18 | Kewazinga Corp. | Navigable telepresence method and system utilizing an array of cameras |
CA2688339C (en) * | 2007-05-25 | 2017-04-04 | Google Inc. | Rendering, viewing and annotating panoramic images, and applications thereof |
JP2010533282A (en) * | 2007-06-08 | 2010-10-21 | テレ アトラス ベスローテン フエンノートシャップ | Method and apparatus for generating a multi-view panorama |
US8525825B2 (en) * | 2008-02-27 | 2013-09-03 | Google Inc. | Using image content to facilitate navigation in panoramic image data |
US9454847B2 (en) * | 2009-02-24 | 2016-09-27 | Google Inc. | System and method of indicating transition between street level images |
-
2011
- 2011-11-22 DE DE202011110924.4U patent/DE202011110924U1/en not_active Expired - Lifetime
- 2011-11-22 EP EP11799904.5A patent/EP2643822B1/en active Active
- 2011-11-22 WO PCT/US2011/061906 patent/WO2012071445A2/en active Application Filing
- 2011-11-22 JP JP2013541029A patent/JP5899232B2/en active Active
- 2011-11-22 CA CA2818695A patent/CA2818695C/en active Active
- 2011-11-22 AU AU2011332885A patent/AU2011332885B2/en active Active
- 2011-11-22 US US13/302,470 patent/US8823707B2/en active Active
Patent Citations (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20080291217A1 (en) | 2007-05-25 | 2008-11-27 | Google Inc. | Viewing and navigating within panoramic images, and applications thereof |
Cited By (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2014210158A3 (en) * | 2013-06-28 | 2015-07-02 | Microsoft Corporation | Space carving based on human physical data |
US10330931B2 (en) | 2013-06-28 | 2019-06-25 | Microsoft Technology Licensing, Llc | Space carving based on human physical data |
CN105741314A (en) * | 2014-12-30 | 2016-07-06 | 诺基亚技术有限公司 | Methods And Apparatuses For Directional View In Panoramic Content |
US9631932B2 (en) | 2015-06-05 | 2017-04-25 | Nokia Technologies Oy | Crowd sourced interaction of browsing behavior in a 3D map |
Also Published As
Publication number | Publication date |
---|---|
JP5899232B2 (en) | 2016-04-06 |
CA2818695A1 (en) | 2012-05-31 |
CA2818695C (en) | 2018-06-26 |
AU2011332885B2 (en) | 2016-07-07 |
EP2643822A2 (en) | 2013-10-02 |
JP2014503882A (en) | 2014-02-13 |
DE202011110924U1 (en) | 2017-04-25 |
AU2011332885A1 (en) | 2013-06-13 |
US8823707B2 (en) | 2014-09-02 |
WO2012071445A3 (en) | 2012-07-19 |
EP2643822B1 (en) | 2017-03-22 |
US20120127169A1 (en) | 2012-05-24 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
AU2011332885B2 (en) | Guided navigation through geo-located panoramas | |
US9024947B2 (en) | Rendering and navigating photographic panoramas with depth information in a geographic information system | |
US8390617B1 (en) | Visualizing oblique images | |
US8686995B2 (en) | Path planning for street level navigation in a three-dimensional environment, and applications thereof | |
US8817067B1 (en) | Interface for applying a photogrammetry algorithm to panoramic photographic images | |
EP2297704B1 (en) | Panning using virtual surfaces | |
EP3432273B1 (en) | System and method of indicating transition between street level images | |
US20130035853A1 (en) | Prominence-Based Generation and Rendering of Map Features | |
US20180025542A1 (en) | Method and system for displaying and navigating an optimal multi-dimensional building model | |
US10878599B2 (en) | Soft-occlusion for computer graphics rendering | |
US20090289937A1 (en) | Multi-scale navigational visualtization | |
JP2014504384A (en) | Generation of 3D virtual tour from 2D images | |
US20150154798A1 (en) | Visual Transitions for Photo Tours Between Imagery in a 3D Space | |
US20110242271A1 (en) | Synthesizing Panoramic Three-Dimensional Images | |
US8884950B1 (en) | Pose data via user interaction | |
Huang et al. | Innovative Application and Improvement of Panoramic Digital Technology in Indoor Display Scenes |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 11799904 Country of ref document: EP Kind code of ref document: A2 |
|
ENP | Entry into the national phase |
Ref document number: 2818695 Country of ref document: CA |
|
ENP | Entry into the national phase |
Ref document number: 2013541029 Country of ref document: JP Kind code of ref document: A |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
REEP | Request for entry into the european phase |
Ref document number: 2011799904 Country of ref document: EP |
|
WWE | Wipo information: entry into national phase |
Ref document number: 2011799904 Country of ref document: EP |
|
ENP | Entry into the national phase |
Ref document number: 2011332885 Country of ref document: AU Date of ref document: 20111122 Kind code of ref document: A |