CN108450035B

CN108450035B - Navigating through a multi-dimensional image space

Info

Publication number: CN108450035B
Application number: CN201680053204.9A
Authority: CN
Inventors: 斯科特·爱德华·迪拉德; 温贝托·卡斯塔尼达; 梁淑君; 迈克尔·卡梅伦·琼斯; 克里斯多佛·格雷; 埃文·哈德斯蒂·帕克
Original assignee: Google LLC
Current assignee: Google LLC
Priority date: 2015-12-17
Filing date: 2016-12-13
Publication date: 2023-01-03
Anticipated expiration: 2036-12-13
Also published as: WO2017106170A3; GB2557142A; KR102230738B1; JP2020053058A; GB201803712D0; KR20180038503A; JP2018538588A; JP6818847B2; KR20200083680A; GB2557142B; WO2017106170A2; EP3338057A2; DE112016004195T5; US20170178404A1; KR102131822B1; JP6615988B2; US10217283B2; CN108450035A

Abstract

Aspects of the present disclosure generally relate to providing a user with an image navigation experience. For example, a first image (332) of a multi-dimensional space is provided with an overlay (502), the overlay (502) indicating a direction in which the space extends into the first image, such that a second image is connected to the first image along the direction of the overlay. User input is received indicating a sweep across a portion of the display. When a sweep occurs at least partially within an interaction zone (602) that defines an area surrounding the overlay line with which the user can interact with the space, the sweep indicates a request to display an image that is different from the first image. The second image is selected and provided for display based on the sweep and a connection map connecting the first image and the second image along the direction of the overlay line.

Description

Navigating through a multi-dimensional image space

Cross Reference to Related Applications

This application is a continuation of U.S. patent application No.14/972,843 filed on 12/17/2015, the disclosure of which is incorporated herein by reference.

Background

Various systems allow a user to view images sequentially, for example, in time or space. In some examples, these systems may provide a navigation experience in a remote or location of interest. Some systems allow users to feel as if they are rotating within the virtual world by clicking on an edge of a displayed portion of the panorama and making the panorama appear to "move" in the direction of the clicked edge.

Disclosure of Invention

One aspect of the present disclosure provides a computer-implemented method of navigating a multidimensional space. The method comprises the following steps: providing, by one or more processors, a first image of a multidimensional space for display on a display of a client computing device and an overlay line extending across a portion of the first image and indicating a direction in which the multidimensional space extends into the first image such that a second image is connected to the first image along the direction of the overlay line; receiving, by the one or more processors, user input indicating a sweep across a portion of the display, the sweep defined by a start pixel and an end pixel of the display; determining, by the one or more computing devices, based on the start pixel and the end pixel, that the sweep has occurred at least partially within an interaction region of the first image, the interaction region defining an area around the overlay graph that the user can interact with the multi-dimensional space; determining, by the one or more processors, that the sweep indicates a request to display an image different from the first image when the sweep occurs at least partially within the interaction zone; selecting, by the one or more computing devices, the second image based on a starting point of the sweep, an ending point of the sweep, and a connection map connecting the first image and the second image along the direction of the overlay line when the sweep indicates a request to display the image different from the first image; and providing, by the one or more computing devices, the second image for display on the display to provide a sensation of movement in the multi-dimensional space.

In one example, the method also includes: providing a transition image for display between the first image and the second image, the transition image being provided as a thumbnail image having less detail than the first image and the second image. In another example, the method also includes: instructions are provided to dissolve the overlay graph after a threshold period of time has elapsed without any user action on the overlay graph. In this example, after fading out the overlay graph, the method comprises: receiving a second user input on the display; and providing instructions to redisplay the overlay graph in response to the second user input. In another example, the method also includes: determining a direction and a magnitude of the sweep based on the start pixel of the sweep and the end pixel of the sweep, and selecting the second image is further based on the direction and magnitude.

In another example, the method also includes: providing the second image with a second overlay line extending across a portion of the second image and indicating a direction in which the multi-dimensional space extends into the second image, such that a third image is connected to the second image in the connection map along the direction of the second overlay line. In this example, the method includes: receiving a second user input indicating a second sweep; determining that the second sweep is within a threshold angle of the direction extending perpendicular to the multi-dimensional space into the second image; and translating across the multi-dimensional space of the second image when the second sweep is within a threshold angle that extends perpendicular to the multi-dimensional space to the direction in the second image. Alternatively, the method also includes: receiving a second user input indicating a second sweep; determining that the second sweep is within a threshold angle of the direction extending perpendicular to the multi-dimensional space into the second image; and changing an orientation within the second image when the second sweep is within a threshold angle that extends perpendicular to the direction in the second image into the multi-dimensional space. In another alternative, the method also includes: receiving a second user input indicating a second sweep; determining that the second sweep is within a threshold angle of the direction extending perpendicular to the multi-dimensional space into the second image; and when the second sweep is within a threshold angle perpendicular to the direction in which the multi-dimensional space extends into the second image, switching from the second image to a third image located on a second connection map adjacent to the connection map, the second and third images having no directional connection in the connection map. In this example, the method also includes: verifying a third overlay line for display with the second image, the third overlay line representing a second navigation path proximate to a current view of the second image, the third overlay line being provided such that the third overlay line and the second overlay line cross each other when displayed with the second image. Further, the method comprises: receiving a second user input along the third overlay line, the second user input indicating a request to transition from an image along the second overlay line to an image along the third overlay line; and providing, in response to the second user input, a third image for display, the third image being arranged in the connection map along the third overlay line. In addition, the method comprises: selecting a set of images for serial display as a transition between the second image and the third image based on connections between images in the connection map; and providing the set of images for display on the display. Further, prior to providing the set of images, the method also includes: filtering the set of images to remove at least one image based on a connection between two images of the set of images in a second connection map different from the first connection map such that a filtered set of images is provided for display as the transition between the second image and the third image.

Another aspect of the present disclosure provides a system. The system includes one or more computing devices, each having one or more processors. The one or more computing devices are configured to: providing a first image of a multi-dimensional space for display on a display of a client computing device and an overlay line extending across a portion of the first image and indicating a direction in which the multi-dimensional space extends into the first image such that a second image is connected to the first image along the direction of the overlay line; receiving user input indicating a sweep across a portion of the display, the sweep defined by a start pixel and an end pixel of the display; determining, based on the start pixel and the end pixel, that the sweep has occurred at least partially within an interaction region of the first image, the interaction region defining an area around the overlay line where the user can interact with the multi-dimensional space; determining that the sweep indicates a request to display an image different from the first image when the sweep occurs at least partially within the interaction zone; when the sweep indicates a request to display the image different from the first image, selecting the second image based on a start point of the sweep, an end point of the sweep, and a connection map connecting the first image and the second image along the direction of the overlay line; and providing the second image for display on the display so as to provide a sensation of movement in the multi-dimensional space.

In one example, the one or more computing devices are further configured to provide a transition image for display between the first image and the second image, the transition image being provided as a thumbnail image having less detail than the first image and the second image. In another example, the one or more computing devices are further configured to provide instructions to fade out the overlay graph after a threshold period of time has elapsed without any user action on the overlay graph. In this example, the one or more computing devices are further configured to, after fading out the overlay graph, receive a second user input on the display and provide instructions to redisplay the overlay graph in response to the second user input. In another example, the one or more computing devices are further configured to determine a direction and magnitude of the sweep based on the start pixel of the sweep and the end pixel of the sweep and select the second image further based on the direction and magnitude. In another example, the one or more computing devices are further configured to provide the second image with a second overlay line that extends across a portion of the second image and indicates a direction in which the multi-dimensional space extends into the second image, such that a third image is connected to the second image in the connection map along the direction of the second overlay line.

Another aspect of the disclosure provides a non-transitory computer readable storage device storing computer readable instructions of a program. The instructions, when executed by one or more processors, cause the one or more processors to perform a method. The method comprises the following steps: providing a first image of a multi-dimensional space for display on a display of a client computing device and an overlay line extending across a portion of the first image and indicating a direction in which the multi-dimensional space extends into the first image such that a second image is connected to the first image along the direction of the overlay line; receiving user input indicating a sweep across a portion of the display, the sweep defined by a start pixel and an end pixel of the display; determining, based on the start pixel and the end pixel, that the sweep has occurred at least partially within an interaction region of the first image, the interaction region defining an area around the overlay line where the user can interact with the multi-dimensional space; determining that the sweep indicates a request to display an image different from the first image when the sweep occurs at least partially within the interaction zone; selecting the second image based on a start point of the sweep, an end point of the sweep, and a connection map connecting the first image and the second image along the direction of the overlay line when the sweep indicates a request to display the image different from the first image; and providing the second image for display on the display so as to provide a sensation of movement in the multi-dimensional space.

Drawings

Fig. 1 is a functional diagram of an example system in accordance with aspects of the present disclosure.

Fig. 2 is a schematic diagram of the example system of fig. 1.

FIG. 3 is an example representation of images and data in accordance with aspects of the present disclosure.

Fig. 4 is a representation of an example graph diagram in accordance with aspects of the present disclosure.

Fig. 5 is an example client computing device and screen shot in accordance with aspects of the present disclosure.

FIG. 6 is a representation of an example client computing device, screen shot, and data in accordance with aspects of the present disclosure.

FIG. 7 is a representation of an example client computing device and data in accordance with aspects of the present disclosure.

Fig. 8A and 8B are examples of user input in accordance with aspects of the present disclosure.

FIG. 9 is a representation of example data in accordance with aspects of the present disclosure.

FIG. 10 is another example client computing device and screen shot in accordance with aspects of the present disclosure.

FIG. 11 is another example client computing device and screen shot in accordance with aspects of the present disclosure.

FIG. 12 is yet another example client computing device and screen shot in accordance with aspects of the present disclosure.

FIG. 13 is another representation of example data in accordance with aspects of the present disclosure.

FIG. 14 is another representation of example data in accordance with aspects of the present disclosure.

Fig. 15A is another example client computing device and screen shot in accordance with aspects of the present disclosure.

Fig. 15B is an example representation of an image map in accordance with aspects of the present disclosure.

Fig. 16 is a flow diagram in accordance with aspects of the present disclosure.

Detailed Description

SUMMARY

The present technology relates to an interface for enabling a user to navigate within a multi-dimensional environment in a first or third human view. In some examples, the environment may include a three-dimensional model rendered by mapping images to a model or series of geo-located (e.g., initially associated with orientation and location information) images using information identifying the two-dimensional or three-dimensional relationship of the images to each other.

To provide "realistic" motion in a multi-dimensional space, the interface may allow continuous motion, intuitive turns, looking around the scene, and moving forward and backward. For example, reference lines may be displayed to indicate to the user the direction in which the user may "traverse" in the multi-dimensional space using touch and/or motion controls. By sweeping in different directions relative to the line, the interface can easily identify the time and direction the user is trying to move, as compared to when the user is trying to simply change orientation and look around.

In order to provide an interface, multiple geo-located images must be available. In addition to being associated with geographic location information, images may also be connected to each other in one or more image graphs. The map may be generated using various techniques including manual and automatic linking based on the location and distance between images, the manner in which the images are captured (such as the images being captured by the camera as it is moved forward), and other methods of identifying the best image of a set of images for connecting to any given point or orientation in the image.

One or more server computing devices may access the one or more graphs to provide images for display to a user. For example, a client computing device of a user may send a request for an image of an identified location. The one or more server computing devices may access the one or more image maps to identify an image corresponding to the location. This image may then be provided to the user's client computing device.

In addition to providing images, the one or more server computing devices may also provide instructions to the client computing device for displaying the navigation overlay. This overlay may be represented as a line indicating to the user the direction in which the user may move in the multi-dimensional space represented by the image. The line itself may actually correspond to a connection between the image currently being viewed by the user and the other images in the one or more image maps. As an example, this line may correspond to a road along which the camera is moved in order to capture the image identified in the image map.

The user may then navigate through the multidimensional space using the line. For example, the lines may be used to suggest interaction regions to the user for moving from the image to a different image in one or more image maps. If the user sweeps within the interaction region of the line and generally parallel to the line or within some small angular difference from the line, the user's client computing device may "move" around in the multi-dimensional space by transitioning to a new image according to the characteristics of the sweep and the image map. In other examples, the sweep may be identified as a request to rotate the view within the current image or to translate in the current image.

Characteristics of the sweep including direction, magnitude and speed can be used to define how the view will change. The magnitude of the sweep or the pixel length can be used to determine how far to move the view forward or backward. Furthermore, the speed of sweeping (number of pixels per second) and even the acceleration of sweeping can be used to determine how fast the view appears to traverse the multi-dimensional space.

The direction may be determined by back projecting (converting from two to three dimensions) the current and previous screen coordinates onto the y = z plane in terms of Normalized Device Coordinates (NDC) and then projecting down onto the ground plane again. This allows vertical display movement to be mapped to forward movement in a multi-dimensional space, while horizontal display movement is mapped to lateral movement in a predictable manner independent of scene geometry or horizon.

As an example, a line may appear when a user taps a view represented on a display of a client computing device. The user may sweep along portions of the display. Using the initial position or pixels of the sweep and other positions along the sweep, the speed of the sweep can be determined. Once the sweep is complete or the user's finger is off the display, a transition animation (such as zooming and fading into a new image) is displayed to transition to the new image.

When the view will typically traverse multiple images in order to reach an image identified based on the characteristics of the sweep, the full resolution of these multiple images may not actually be displayed. Alternatively, a lower resolution version of the image (such as a thumbnail image) may be displayed as part of the transition between the current image and the identified image. When the actually recognized image is displayed, this image may be displayed at full resolution. This may save time and processing power.

Characteristics of the lines such as opacity, width, color, and location may be varied to allow a user to more easily understand the navigation of the multi-dimensional space. Additionally, to further reduce the interference of the line with the user's exploration of the multidimensional space, the line may not be shown when the user is not interacting with the interface. As an example, a line may be faded in or faded out as desired based on whether the user is interacting with the display.

In some examples, the connection map may branch, such as there being an intersection of two roads, or rather, there may be an intersection in one or more of the map maps. To keep the overlay simple and easy to understand, lines directly connected to the current line only within a short distance of the current viewpoint may appear to be superimposed on the image. In these branch regions, such as at traffic intersections where more than one road intersects each other, the user may want to change from one line to another. Of course, driving forward and making a 90 degree turn in the middle of the intersection may feel unnatural. In this regard, one or more image maps may be used to transcend the corners of the intersection between two lines by displaying images that are not on any one line as a transition between the two lines.

The features described herein allow a user to explore while simultaneously following a particular predetermined motion path inside a multidimensional space, while at the same time preventing the user from being "trapped" or moving in an inefficient manner. Further, the system can identify discrepancies when the user is trying to move versus when the user is trying to look around. Other systems require multiple types of input in order to distinguish between these types of movements. These systems may also require the user to point to a particular location in order to move to that location. This is less intuitive than allowing the user to sweep in order to move in a multi-dimensional space and does not allow continuous motion, as the user must tap or click an arrow each time he wants to move.

Further aspects, features and advantages of the present disclosure will be appreciated when considered with reference to the following description of the embodiments and the accompanying drawings. The same reference numbers in different drawings may identify the same or similar elements. Furthermore, the following description is not limiting; the scope of the present technology is defined by the appended claims and equivalents. Although certain processes in accordance with example embodiments are illustrated in the figures as occurring in a linear fashion, this is not a requirement unless explicitly stated herein. The different processes may be performed in a different order or simultaneously. Steps may also be added or omitted unless stated otherwise.

Example System

Fig. 1 and 2 include an example system 100 that can implement the features described above. It should not be taken as limiting the scope of the disclosure or the usefulness of the features described herein. In this example, system 100 may include

computing devices

110, 120, 130, and 140 and storage system 150. Each of the computing devices 110 may contain one or more processors 112, memory 114, and other components typically found in general purpose computing devices. The memory 114 of each of the

computing devices

110, 120, 130, and 140 may store information accessible by the one or more processors 112, including instructions 116 executable by the one or more processors 112.

The memory may also include data 118 that may be retrieved, manipulated or stored by the processor. The memory may be of any non-transitory type capable of storing information accessible by the processor, such as a hard drive, memory card, ROM, RAM, DVD, CD-ROM, writable and read-only memory.

The instructions 116 may be any set of instructions to be executed directly by one or more processors, such as machine code, or indirectly by one or more processors, such as scripts. In this regard, the terms "instructions," "applications," "steps," and "programs" may be used interchangeably herein. The instructions may be stored in an object code format directly processed by the processor, or in any other computing device language, including scripts or collections of independent source code modules that are interpreted on demand or compiled in advance. The function, method, and routine of the instructions are described in more detail below.

The data 118 may be retrieved, stored, or modified by the one or more processors 112 in accordance with the instructions 116. For example, although the subject matter described herein is not limited by any particular data structure, data may be stored in computer registers, in a relational database or XML document as a table with many different fields and records. The data may also be formatted in any computing device readable format such as, but not limited to, binary values, ASCII, or Unicode. Further, the data may include any information sufficient to identify the relevant information, such as numbers, descriptive text, proprietary codes, pointers, references to data stored in other memories, such as other network locations, or information used by the function to compute the relevant data.

The one or more processors 112 may be any conventional processor, such as a commercially available CPU. Alternatively, the processor may be a dedicated component, such as an application specific integrated circuit ("ASIC") or other hardware-based processor. Although not necessary, one or more of the computing devices 110 may include dedicated hardware components to perform specific computing processes, such as decoding video, matching video frames to images, distorting video, encoding distorted video, etc., faster or more efficiently.

Although fig. 1 functionally illustrates the processors, memories, and other elements of the computing device 110 as being within the same block, the processors, computers, computing devices, or memories may actually comprise multiple processors, computers, computing devices, or memories that may or may not be stored within the same physical housing. For example, the memory may be a hard drive or other storage medium located in a different housing than the housing of the computing device 110. Thus, references to a processor, computer, computing device, or memory are to be understood as including references to a processor, computer, computing device, or memory that may or may not operate in parallel. For example, the computing devices 110 may include server computing devices operating as a load balancing server farm, a distributed system, and so forth. Still further, while some of the functionality described below is indicated as occurring on a single computing device having a single processor, various aspects of the subject matter described herein may be implemented by multiple computing devices, e.g., communicating information over network 180.

Each of the computing devices 110 may be located at a different node of the network 180 and may be capable of communicating directly and indirectly with other nodes of the network 180. Although only a few computing devices are depicted in fig. 1 and 2, it should be appreciated that a typical system may include a large number of connected computing devices, with each different computing device located at a different node of the network 180. The network 180 and intermediate nodes described herein may be interconnected using various protocols and systems, such that the network may be part of the internet, world wide web, a particular intranet, a wide area network, or a local network. The network may utilize standard communication protocols (such as ethernet, wiFi, and HTTP), protocols proprietary to one or more companies, and various combinations of the above. Although certain advantages are obtained when information is transmitted or received as noted above, other aspects of the subject matter described herein are not limited to any particular manner of transmission of information.

By way of example, each of computing devices 110 may include a web server capable of communicating with storage system 150 and

computing devices

120, 130, and 140 via a network. For example, one or more of server computing devices 110 may use network 180 to communicate information to a user (such as

user

220, 230, or 240) and present the information to the user on a display (such as

display

122, 132, or 142 of

computing devices

120, 130, and 140). In this regard,

computing devices

120, 130, and 140 may be considered client computing devices and may perform all or some of the features described herein.

Each of the

client computing devices

120, 130, and 140 may be configured with one or more processors, memories, and instructions as described above, similar to the server computing device 110. Each

client computing device

120, 130, or 140 may be a personal computing device intended for use by a

user

220, 230, 240, and have all of the components typically used in connection with a personal computing device, such as a Central Processing Unit (CPU), memory (e.g., RAM and internal hard drives) that stores data and instructions, a display such as

display

122, 132, or 142 (e.g., a monitor having a screen, a touch screen, a projector, a television, or other device operable to display information), and a user input device 124 (e.g., a mouse, a keyboard, a touch screen, or a microphone). The client computing device may also include a camera for recording video streams, speakers, a network interface device, and all components for connecting these elements to each other.

Although the

client computing devices

120, 130, and 140 may each comprise a full-sized personal computing device, they may alternatively comprise mobile computing devices capable of wirelessly exchanging data with a server over a network such as the internet. By way of example only, the client computing device 120 may be a mobile phone or a device such as a wireless-enabled PDA, a tablet PC, or a netbook capable of obtaining information via the internet. In another example, the client computing device 130 may be a head-mounted computing system. As examples the user may input information using a keyboard, a keypad, a microphone, using a camera using visual signals or a touch screen.

As with the memory 114, the storage system 150 may be any type of computerized storage capable of storing information accessible by the server computing device 110, such as a hard drive, memory card, ROM, RAM, DVD, CD-ROM, writable and read-only memory. Further, storage system 150 may comprise a distributed storage system in which data is stored on multiple different storage devices that may be physically located at the same or different geographic locations. Storage system 150 may be connected to computing devices via network 180 as shown in fig. 1 and/or may be directly connected to any of

computing devices

110, 120, 130, and 140 (not shown).

The storage system 150 may store images and related information such as image identifiers, orientation and position of the images, orientation and position of the cameras capturing the images, and intrinsic camera settings (such as focal length, zoom, etc.). In addition to being associated with orientation and position information, images may be connected to each other in one or more images. In other words, from any given image, the image maps may indicate which other images are connected to the image and in which direction.

The map may be generated using various techniques including manual and automatic linking based on the location and distance between images, the manner in which the images are captured (such as the images being captured by the camera as it is moved forward), and other methods of identifying the best image of a set of images for connecting to any given point or orientation in the image. For example, as shown in example 300 of fig. 3, an image may be captured by maneuvering a vehicle along a road 302. In this example, the roadway includes a plurality of

lanes

304, 306, 308. At one time, a vehicle traveling in a lane 304 captures a series of images 310 of features in an environment including, among other things, the vehicle, such as a road 302 and

buildings

330, 332, 334, 336, 338. At another time, the same or a different vehicle traveling in lane 306 (and later switching to lane 308 between images 340 and 342) captures a series of images 320. As the one or more cameras that captured the series of images are moved forward, they may associate each image with information regarding the position of the camera at the time the image was captured, the orientation of the camera at the time the image was captured (e.g., relative to the orientation indicator 360), timestamp information indicating the date and time the image was captured, and so forth. The camera's change in position over time and along the series of images can be used to "link" the image pairs together via

lines

350, 352. Again, the position and orientation of the camera and the timestamp information of the image may be used to perform this linking manually or automatically based on the change in position of the camera over time. The links then create a connection of the image maps, or information describing the relative orientation between each adjacent pair of images. In this example, the result of the linking is two image maps corresponding to the series of

images

310 and 320 and the connections between these images.

The maps themselves may have different images and connections. Fig. 4 is an example of a series of

images

410, 420, 450. In this example, image maps 410 and 420 may be generated from images and connections that represent only images and connections along a street or map grid corresponding to a road, path, or the like, or by pairing more complex maps with images and connections that include only a grid. For example, image maps 410 and 420 may be generated from the

image series

310 and 320 as discussed above. The image map 420 actually includes only images 402-430 relating to the road 302 or mesh of roads in the map. The other connectivity graphs may then identify connections from the images in the first graph to images other than those in the first graph. For example, image graph 450 includes connections between

images

402, 404, 406, and 408 of image series 320 and

images

442, 444, 446, and 448 of image series 322. Further, image graph 450 includes connections to images 452 and 545, which may be images from within building 330, but are not included in any of image series 320 or 322. Again, these connections may be drawn manually or automatically as discussed above.

Example method

As previously discussed, the following operations need not be performed in the exact order described below. Rather, as mentioned above, various operations may be processed in a different order or simultaneously, and operations may be added or omitted.

As an example, a client computing device may provide a user with an image navigation experience. The client computing device may do so by communicating with one or more server computing devices to retrieve and display images. One or more server computing devices may access the one or more graphs to provide images for display to a user. For example, a client computing device of a user may send a request for an image of an identified location. The one or more server computing devices may access the one or more image maps to identify an image corresponding to the location. This image may then be provided to the user's client computing device.

In addition to providing images, the one or more server computing devices may provide instructions to the client computing device for displaying the navigation overlay. This overlay may be represented as a line indicating to the user the direction in which the user may move in the multi-dimensional space. For example, fig. 5 is an example view 500 of an image 444 of image map 310 displayed on display 122 of computing device 120. An overlay 502 is provided herein that indicates the direction the user can move into the image 444. The line itself may actually correspond to the connection between the image 444 currently being viewed by the user and the other images in the image map 310. As can be seen in this example, image 444 includes portions of

buildings

332, 334, 336, and 338.

The user may then navigate through the multidimensional space using the line. For example, the lines may be used to suggest interaction regions to the user for moving from the image to a different image in one or more image maps. As shown in example 600 of fig. 6, interaction zone 602 extends beyond the outer edge of line 502. Although the interactive area 602 is shown on the display 122 in fig. 6, this interactive area need not actually be displayed to the user and may be "hidden" in order to reduce clutter on the display. Because the interaction zone 602 extends beyond the outer edge of the line, this may allow some room for error when the user attempts to sweep his or her finger (or stylus). For example, referring to example 700 of fig. 7, region 702 corresponds to display 122. In this example, region 704 refers to an area of display 122 that is outside of interaction zone 602. Again, these areas may not necessarily be displayed to the user, and fig. 7 is used for purposes of illustrating the examples more clearly below.

If the user is within the interaction zone of the line and typically either parallel to the line or sweeping within some small angular difference from the line, the client computing device may "move" around in the multidimensional space. For example, as shown in examples 800A and 800B of fig. 8A and 8B, a user may use a finger (or stylus) to interact with the interaction zone 602 of lines by sweeping generally parallel to the lines 502 in the direction of

arrows

804 or 806. Because the sweep may not be exactly parallel to line 502, as shown in example 900 of fig. 9, small angular differences (θ 1), such as 10 or 15 degrees, from the direction of line 502 within interaction zone 602 may be ignored or considered to be "generally" parallel.

Returning to fig. 8A, as the user sweeps in the direction of arrow 804, a new image may be displayed that appears to move the user within the multi-dimensional space. For example, in response to a user sweeping within the interaction zone 602 generally parallel to the line 502, the display may transition to another image of the image map 310. As shown in example 1000 of fig. 10, image 470 is displayed transitioning to image map 310. In this example, the display appears to have moved the user further down lane 304 and generally along line 502 so that building 336 is no longer visible. This again allows the user to feel that he or she is moving in three-dimensional space even though he or she is viewing the two-dimensional image. The selection of the image to be displayed may depend on the characteristics of the sweep and one or more image maps as discussed below.

If the user sweeps outside this interaction region 602 or away from the line 502 so many pixels or within some specified angular distance from the line (e.g., greater than θ 1, but less than 90- θ 1), rather than transitioning to a new image, the user's client computing device may change the orientation of the view of the current image, or otherwise rotate the view within the current image. For example, if the user sweeps his or her finger across the display of example 600 from the left side of display 122 toward the right side of display 122 at an angle greater than θ 1 but less than 90- θ 1, rather than generally moving along line 502, the display may rotate within image 444. As shown in example 1100 of fig. 11, the display rotates from the position of example 600 within image 444 and more buildings 332 are displayed. This gives the user the sense that he or she has rotated or changed his or her orientation, as opposed to moving forward or backward in a multi-dimensional space. Thus, the interaction zone allows the client computing device to distinguish between requests to move around (i.e., move forward or backward) and requests to change the orientation of the user.

Further, if the user sweeps within or outside the interaction zone, generally perpendicular to the direction of line 502 or within a small angular distance θ 1 from the direction perpendicular to line 502 (or less than 90- θ 1 from the direction parallel to line 502), this may indicate that the user wishes to pan (sideways move) in the current image. For example, if the user were to sweep his or her finger across the display of example 600 from the left side of display 122 toward the right side of display 122 at an angle greater than 90- θ 1 in a direction parallel to line 502, rather than generally moving along line 502 or rotating within image 444, the display may pan within the current image. In this regard, if there is more than one line according to one or more image graphs, this movement may cause the graphs to actually "jump" to different lines. For example, from image 444, display may jump to image 404 of image map 420 as shown in example 1200 of FIG. 12. Here, the user appears to have moved from a point within lane 304 to a point within lane 306 by moving between

images

444 and 402. At this point, a new overlay 1202 corresponding to the image map 420 may be provided for display with the image 404.

Characteristics of the sweep including direction, magnitude, and speed can be used to define how the view will change. For example, while being displayed on-line, if the user sweeps along the interaction zone, the result may be a movement in a direction (within the image map) towards the start point of the sweep. In this regard, dragging down as shown in FIG. 8A advances the view, and dragging up as shown in FIG. 8B moves the view backward.

The magnitude of the sweep or the pixel length may be used to determine how far forward or backward the view is to be moved. The result may be no movement, for example, if the sweep does not cross the threshold minimum number of pixels. If the sweep satisfies a threshold minimum number of pixels, the movement in the multiple dimensions may correspond to the number of pixels the sweep spans. However, because the view is perspective, the relationship between the distance in the pixel and the distance in the multi-dimension can be exponential (as opposed to linear) as the plane in which the line appears will tilt towards the vanishing point in the image. In this regard, the distance in a pixel may be converted to a distance in a multi-dimensional space. An image along a line from the original image closest to a distance in the multi-dimensional space according to the one or more image maps may be identified as the image to which the view is to transition.

Furthermore, the speed of sweeping (number of pixels per second) and even the acceleration of sweeping can be used to determine how fast the view appears to traverse the multi-dimensional space. For example, the movement may initially correspond to the speed of the sweep, but this speed may slow down and stop at the image identified from the magnitude of the sweep. Where the distance determined based on the magnitude of the sweep is between two images, the speed (or acceleration) of the sweep may be used to determine which image to identify as the image to which the view will transition. For example, if the velocity (or acceleration) is relatively high or greater than some threshold velocity (or acceleration), then images further away from the original image may be selected. Meanwhile, if the velocity (or acceleration) is relatively low or below a threshold velocity (or acceleration), an image closer to the original image may be selected. In yet another example, where the velocity (or acceleration) meets some other threshold, the view may appear to continuously traverse the multi-dimensional space by transitioning between images along a line in one or more image maps in response according to the speed of the sweep until the user taps the display. This tap may slow down the movement to a stop or stop immediately on the current or next image. In yet another example, the speed of the sweep, which is generally perpendicular to the line, may be translated into a slower movement through multiple dimensions than with the same speed of the sweep, which is generally parallel to the line. Still further, the speed of the sweep may be determined based on where the sweep occurs. In this regard, the velocity may be determined by measuring meters per pixel at a point on the screen halfway between the bottom of the screen and the horizon, which is intuitively the "average" screen position of the sweep. So if the user drags his or her finger at this point, the pixels on the ground will move at the same speed as the user's finger.

The direction may be determined by back projecting (converting from two to three dimensions) the current and previous screen coordinates onto the y = z plane in terms of Normalized Device Coordinates (NDC) and then projecting down onto the ground plane again. For example, the example 1300 of fig. 13 is a diagram illustrating a relative direction (indicated by arrow 1302) of a plane of a screen 1304 of the computing device 120 with respect to a ground plane 1306. Between these two planes is an NDC plane 1308. Fig. 14 depicts the direction of the finger sweep shown by arrow 1404 on the plane of the screen 1304 relative to the movement shown by arrow 1406 along the ground plane 1306. Here, the user's finger initiates a sweep at point a on the screen 1304 and ends the sweep at point B on the screen. These points are converted to corresponding points B 'and A' on the NDC plane 1308. From the NDC plane, these points are projected to points C and D on the ground plane 1306. This allows vertical display movement to be mapped to forward movement in the multi-dimensional space and horizontal display movement to be mapped to lateral movement in the multi-dimensional space in a predictable manner independent of scene geometry or horizon.

As an example, a line may appear when a user taps a view represented on a display of a client computing device. The user may then sweep along portions of the display. Using the initial position or pixels of the sweep and other positions along the sweep, the speed of the sweep can be determined. Once the sweep has been completed or the user's finger has left the display, a transition animation (such as zooming and fading into a new image) is displayed to transition to the new image. As an example, if the speed is less than or less than the threshold, the next image along the line may be displayed as a new view. If the speed is greater than the threshold, the end time of the sweep and the location on the display are determined. The image closest to this position along the line is identified as the target image based on one or more image maps. In this example, the transition animation between images along the line continues until the target image is reached. In some examples, the images displayed during the transition animation and the target image may be retrieved in real-time from local memory or by verifying location information and requesting the images from one or more server computing devices while the transition animation is being played.

Since the view will typically traverse through multiple images in order to reach the image identified based on the characteristics of the sweep, the full resolution of these multiple images may not actually be displayed. Alternatively, a lower resolution version of the image (such as a thumbnail image) may be displayed as part of the transition between the current image and the identified image. When the actually recognized image is displayed, this image may be displayed at full resolution. This may save time and processing power.

The characteristics of the lines may be altered to allow the user to more easily understand the navigation through the multidimensional space. For example, the opacity of a line may be adjusted to allow a user to see more or less features below the line, thereby reducing the impact of the line on the user's ability to visually explore multiple dimensions. In this regard, the width of the line may correspond to the width of the road on which the line is covered. Similarly, the width of the lines and the interaction zone may be adjusted to prevent the lines from taking up too much of the display, while at the same time making the lines thick enough for the user to be able to interact with the lines using his or her fingers. The color of the line (e.g., blue) may be selected to complement the current view or to allow the line to protrude from the current view.

The position of the lines is not always the same as the connecting lines in one or more of the images. For example, where a line is displayed to correspond to the width of a road, the corresponding connections in one or more image maps may actually run down not in the middle of the road, such that the line does not correspond perfectly to the one or more image maps. In areas where the geometry of the connecting lines is tortuous, the connecting lines may actually fit into straighter lines as an overlay. In this regard, the line may not pass through the center of each image, but may have a smooth appearance when overlaid on the view.

In some examples, the connection map may branch, such as if there is an intersection of two roads or, less so, an intersection in one or more image maps. Example 1500A of fig. 15 depicts an example of a branch region where two

lines

1502 and 1504 cross each other. In this example, if the user sweep travels in this or the same way, the resulting motion follows the desired branch according to the image map or maps, and the view adjusts to continue the user facing forward along the line. By showing a number of branches, the available direction of movement is immediately apparent to the user. However, in order to keep the overlay simple and easy to understand, lines directly connected to the current line only within a short distance of the current viewpoint may appear overlaid on the imagery. Using a line pointing in a direction that the user can travel within a predetermined distance may prevent the user from being provided with conflicting or confusing information about where the user may go in the multidimensional space. This is particularly important in complex geometric regions or in complex navigation sections.

In these branch regions, such as at traffic intersections where more than one road intersects each other, the user may want to change from one line to another. Of course, driving forward and making a 90 degree turn in the middle of the intersection may feel unnatural. In this regard, by displaying images that are not on any one line as transitions, one or more image maps may be used to transcend the corners of the intersection between two lines. Fig. 15B depicts image map 1500B with

lines

1512 and 1514 corresponding to

lines

1504 and 1502, respectively. Thus, in the example of fig. 15A, the display 122 is displaying the image of the image E in the direction of the image G. If the user were to sweep within the interaction zone around 1502, this would correspond to the location of image H on image map 1500B. Rather than transitioning to image F, then to image G, then to image H in this example because the branch region transitions to image H, the display may appear to transition directly from image E to image H along path 1516. Alternatively, only a single image may be skipped and the display may appear to transition from image E to image F and then along path 1518 to image H.

To further reduce the interference of the line with the user's exploration of the multi-dimensions, the line may not be shown when the user is not interacting with the interface. This allows the user to see the entire 3D scene. For example, if the user taps and drags around (or taps and drags around, or moves and drags around) any other party in the scene, he or she will look around the scene when the line is not visible. If the user taps once on the image, a line may appear. Effects such as flashing of the line, highlighting then dimming, thickening then thinning of the line, or quickly making the line more or less opaque then returning to normal opacity may be used to indicate to the user the interactive nature of the line. If the user makes a dragging motion within the interaction zone, lines may appear and the image may appear to transition along the lines even when the lines are not visible. After some predetermined period of time (such as 2 seconds or more or less) on the display without input received by the client computing device, or after the second half of a single tap on the display for a second or more or less, the line may fade until it disappears, again to reduce the effect of the line on the image.

The interface may also provide other types of navigation other than between images along lines corresponding to the grid. For example, as noted above, a single tap may cause a line to appear. A single tap in this example may not cause the view to change, but instead, the view may appear to remain motionless. At the same time, the double tap may bring the user to an image in one or more image maps that is connected to the current image at or near the point of the double tap. Additionally, if the user is currently facing nearly perpendicular to the line, and the user is typically sweeping perpendicular to the line, the view may appear to remain perpendicular to the line, allowing the user to "sweep" along the road. Further, the pinch gesture may zoom in or out on a particular image without actually causing a transition to a new image.

Although the above examples refer to lines, various other overlays can be used to provide the user with an indication of navigable areas in multiple dimensions. For example, in addition to or in conjunction with a toggle switch for switching between looking around (changing orientation) and moving through multiple dimensions, a plurality of scroll bars may be placed on the map to provide visual feedback to the user as he or she moves through the multiple dimensions. In another example, wider lines may be used instead of finite lines that appear to blend laterally into the view. In yet another alternative, interactive zones may be suggested using a disk or short arrow-like indication that does not necessarily appear to extend far into the scene.

Fig. 16 is an example flow diagram 1600 of the various aspects described above that may be performed by one or more server computing devices, such as server computing device 110. In this example, a first image of a multidimensional space for display on a display of a client computing device is provided at block 1602. The first image is provided with an overlay line extending across a portion of the first image and indicating a direction in which the multi-dimensional space extends into the first image, such that the second image is connected to the first image along the direction of the overlay line. User input indicating a sweep across a portion of the display is received at block 1604. The sweep is defined by a start pixel and an end pixel of the display. Based on the start pixel and the end pixel, it is determined at block 1606 that a sweep has occurred at least partially within the interaction region of the first image. The interactive zone defines an area around the overlay line where a user can interact with the three-dimensional space. When a sweep has occurred at least partially within the interaction zone, a determination is made at block 1608 that the sweep indicates a request to display an image that is different from the first image. When the sweep indicates a request to display a different image than the first image, the second image is selected at block 1610 based on a starting point of the sweep, an ending point of the sweep, and a connection map connecting the first image and the second image along a direction that covers the map line. A second image for display on the display is provided at block 1612 to provide a sensation of movement in the multi-dimensional space.

Most of the above alternative examples are not mutually exclusive, but can be implemented in various combinations to achieve unique advantages. As these and other variations and combinations of the features discussed above can be utilized without departing from the subject matter defined by the claims, the foregoing description of the embodiments should be taken by way of illustration rather than by way of limitation of the subject matter defined by the claims. By way of example, the preceding operations need not be performed in the exact order described above. Rather, the various steps may be processed in a different order or simultaneously. Steps may also be omitted unless otherwise stated. Furthermore, the provision of examples described herein, as well as clauses phrased as "such as," "including," and the like, should not be interpreted as limiting the subject matter of the claims to the specific examples; rather, the example is intended to illustrate only one of many possible embodiments. In addition, the same reference numbers in different drawings may identify the same or similar elements.

INDUSTRIAL APPLICABILITY

The present technology enjoys wide industrial applicability including, but not limited to, interfaces for enabling a user to navigate within a multi-dimensional environment in a first or third human view.

Claims

1. A method of navigating a multi-dimensional space, the method comprising:

providing, by one or more processors, a first image of a multidimensional space for display on a display of a client computing device and an overlay line extending across a portion of the first image and indicating a direction in which the multidimensional space extends into the first image such that a second image is connected to the first image along the direction of the overlay line;

receiving, by the one or more processors, user input indicative of a sweep of the display across a portion, the sweep defined by a start pixel and an end pixel of the display;

determining, by the one or more computing devices, based on the start pixel and the end pixel, that the sweep occurred at least partially within an interaction region of the first image, the interaction region defining an area around the overlay line that the user can interact with the multi-dimensional space;

determining, by the one or more processors, that the sweep indicates a request to display an image different from the first image when the sweep occurs at least partially within the interaction zone;

selecting, by the one or more computing devices, the second image based on a starting point of the sweep, an ending point of the sweep, and a connection map connecting the first image and the second image along the direction of the overlay line when the sweep indicates a request to display the image different from the first image;

providing the second image with a second overlay line extending across a portion of the second image and indicating a direction in which the multi-dimensional space extends into the second image, such that a third image is connected to the second image in the connection map along the direction of the second overlay line;

receiving a second user input indicating a second sweep;

determining that the second sweep is within a threshold angle perpendicular to the direction that the multi-dimensional space extends into the second image; and

switching from the second image to a fourth image located on a second connection map adjacent to the connection map in a direction perpendicular to the direction in which the multi-dimensional space extends into the second image when the second sweep is within a threshold angle perpendicular to the direction in which the multi-dimensional space extends into the second image, the second and fourth images being neither connected to each other in the connection map nor connected to each other in the second connection map.

2. The method of claim 1, further comprising providing a transition image for display between the first image and the second image, the transition image being provided as a thumbnail image having less detail than the first image and the second image.

3. The method of claim 1, further comprising providing instructions to fade out the overlay after a threshold period of time has elapsed without any user action on the overlay.

4. The method of claim 3, further comprising, after fading out the overlay graph:

receiving a second user input on the display; and

providing instructions to redisplay the overlay graph in response to the second user input.

5. The method of claim 1, further comprising:

determining a direction and a magnitude of the sweep based on the start pixel of the sweep and the end pixel of the sweep, an

Wherein selecting the second image is further based on the direction and magnitude.

6. The method of claim 1, further comprising:

receiving a third user input indicative of a third sweep;

determining that the third sweep is within a threshold angle perpendicular to the direction that the multi-dimensional space extends into the fourth image; and

translating across the multi-dimensional space of the fourth image when the third sweep is within a threshold angle perpendicular to the direction that the multi-dimensional space extends into the fourth image.

7. The method of claim 1, further comprising:

receiving a third user input indicative of a third sweep;

changing an orientation within the fourth image when the third sweep is within a threshold angle perpendicular to the direction in which the multi-dimensional space extends into the fourth image.

8. The method of claim 1, further comprising: providing a third overlay line for display with the second image, the third overlay line representing a second navigation path proximate to a current view of the second image, the third overlay line being provided such that the third overlay line and the second overlay line cross each other when displayed with the second image.

9. The method of claim 8, further comprising:

receiving a second user input along the third overlay line, the second user input indicating a request to transition from an image along the second overlay line to an image along the third overlay line; and

providing, for display, a fifth image in response to the second user input, the fifth image arranged in the connection map along the third overlay line.

10. The method of claim 9, further comprising:

selecting a set of images for serial display as a transition between the second image and the fifth image based on connections between images in the connection map; and

providing the set of images for display on the display.

11. The method of claim 10, further comprising, prior to providing the set of images, filtering the set of images based on connections between two images of the set of images in a third connection map different from the connection map to remove at least one image such that a filtered set of images is provided for display as the transition between the second image and the fifth image.

12. A system comprising one or more computing devices, each computing device having one or more processors, the one or more computing devices configured to:

providing a first image of a multi-dimensional space for display on a display of a client computing device and an overlay line extending across a portion of the first image and indicating a direction in which the multi-dimensional space extends into the first image such that a second image is connected to the first image along the direction of the overlay line;

receiving user input indicative of a sweep of the display across a portion, the sweep defined by a start pixel and an end pixel of the display;

determining, based on the start pixel and the end pixel, that the sweep has occurred at least partially within an interaction region of the first image, the interaction region defining an area around the overlay line where the user can interact with the multi-dimensional space;

determining that the sweep indicates a request to display an image different from the first image when the sweep occurs at least partially within the interaction zone;

selecting the second image based on a start point of the sweep, an end point of the sweep, and a connection map connecting the first image and the second image along the direction of the overlay line when the sweep indicates a request to display the image different from the first image;

receiving a second user input indicating a second sweep;

13. The system of claim 12, wherein the one or more computing devices are further configured to provide a transition image for display between the first image and the second image, the transition image being provided as a thumbnail image having less detail than the first image and the second image.

14. The system of claim 12, wherein the one or more computing devices are further configured to provide instructions to fade out the overlay after a threshold period of time has elapsed without any user action on the overlay.

15. The system of claim 14, wherein the one or more computing devices are further configured to, after fading out the overlay graph:

receiving a second user input on the display; and is

16. The system of claim 12, wherein the one or more computing devices are further configured to:

determining a direction and a magnitude of the sweep based on the start pixel of the sweep and the end pixel of the sweep, and

selecting the second image is further based on the direction and magnitude.

17. A non-transitory computer-readable storage device storing computer-readable instructions of a program, which when executed by one or more processors, cause the one or more processors to perform a method, the method comprising:

when the sweep has occurred at least partially within the interaction zone, determining that the sweep indicates a request to display an image different from the first image;

receiving a second user input indicating a second sweep;