US20170024904A1 - Augmented reality vision system for tracking and geolocating objects of interest - Google Patents
Augmented reality vision system for tracking and geolocating objects of interest Download PDFInfo
- Publication number
- US20170024904A1 US20170024904A1 US15/286,161 US201615286161A US2017024904A1 US 20170024904 A1 US20170024904 A1 US 20170024904A1 US 201615286161 A US201615286161 A US 201615286161A US 2017024904 A1 US2017024904 A1 US 2017024904A1
- Authority
- US
- United States
- Prior art keywords
- view
- objects
- interest
- user
- location
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
- G06T19/006—Mixed reality
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T11/00—2D [Two Dimensional] image generation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T11/00—2D [Two Dimensional] image generation
- G06T11/60—Editing figures and text; Combining figures or text
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
- G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/10—Image acquisition
- G06V10/12—Details of acquisition arrangements; Constructional details thereof
- G06V10/14—Optical characteristics of the device performing the acquisition or on the illumination arrangements
- G06V10/147—Details of sensors, e.g. sensor lenses
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/22—Image preprocessing by selection of a specific region containing or referencing a pattern; Locating or processing of specific regions to guide the detection or recognition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/20—Scenes; Scene-specific elements in augmented reality scenes
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/45—Cameras or camera modules comprising electronic image sensors; Control thereof for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
-
- H04N5/2258—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
- G06T2207/10021—Stereoscopic video; Stereoscopic image sequence
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30181—Earth observation
- G06T2207/30184—Infrastructure
Definitions
- Embodiments of the present invention generally relate to augmented reality and, more particularly, to a method and apparatus for tracking and geolocating objects of interest.
- binoculars are entirely optical devices allowing users to zoom in on a particular real world area from a long distance. If a user is attempting to view precise movements of an object at a distance, such as a car, truck or person, the user is able to use a binocular lens switching mechanism to change to a different magnification. In other binoculars, a “zooming” function is provided which can vary magnification ranges using a switch or lever. However, once the user increases magnification level, the user may experience difficulty in finding the object of interest within the “zoomed” scene.
- An apparatus and/or method for tracking and geolocating objects of interest comprising one or more optical sensors for capturing one or more images of a scene, wherein the one or more optical sensors capture a wide field of view and corresponding narrow field of view for the one or more mages of a scene, a localization module, coupled to the one or more optical sensors for determining the location of the apparatus, and determining the location of one more objects in the one or more images based on the location of the apparatus, and an augmented reality module, coupled to the localization module, for enhancing a view of the scene on a display based on the determined location of the one or more objects.
- FIG. 1 depicts an apparatus for tracking objects in accordance with exemplary embodiments of the present invention
- FIG. 2 depicts an embodiment of the apparatus as a binocular unit in accordance with exemplary embodiments of the present invention
- FIG. 3 depicts an image captured by narrow field of view optics in accordance with exemplary embodiments of the present invention
- FIG. 4 depicts an image captured by wide field of view optics in accordance with exemplary embodiments of the present invention
- FIG. 5 depicts a wide field of view image, where a helicopter is placed at a particular location in accordance with exemplary embodiments of the present invention
- FIG. 6 depicts a wide field of view image, where a tank is placed at a particular location in accordance with exemplary embodiments of the present invention
- FIG. 7A depicts a plurality of binocular units viewing portions of a scene in accordance with exemplary embodiments of the present invention
- FIG. 7B depicts an exemplary use of the apparatus 100 in accordance with exemplary embodiments of the present invention.
- FIG. 8 depicts a method for object detection in accordance with exemplary embodiments of the present invention.
- Embodiments of the present invention generally relate to a method and apparatus for tracking and geolocating objects of interest.
- an apparatus for tracking objects of interest comprises an optical sensor used in conjunction with positional sensors such as inertial measurement units (IMU) and global navigation satellite systems (such as geographical positioning satellite (GPS) systems, Glonass and Galileo) to locate the apparatus and a laser rangefinder, to geographically localize objects in reference to the location of the optical sensor.
- positional sensors such as inertial measurement units (IMU) and global navigation satellite systems (such as geographical positioning satellite (GPS) systems, Glonass and Galileo) to locate the apparatus and a laser rangefinder, to geographically localize objects in reference to the location of the optical sensor.
- Objects of interest are tracked as a user zooms the optical sensors, or the user switches from a wide field of view to a narrow field of view.
- Embodiments of the present invention preserve a location of an object of interest within the user's view throughout a change in zoom or field of view.
- an augmented reality system may provide the user with indicia of the location of the object within the user's view, as well as other pertinent information.
- An observed scene is augmented with information, labels of real world objects and guidance information on moving a camera unit or binocular unit to view objects of interest in a user's field of view. The augmentation appears stable with respect to the observed scene.
- FIG. 1 depicts an apparatus for tracking objects 100 in accordance with exemplary embodiments of the present invention.
- the apparatus 100 comprises an object recognition module 102 , sensors 103 1 to 103 n and 105 1 to 105 n , a tracking module 104 , an augmented reality module 106 , a localization module 108 , a reasoning module 110 , a database 112 , and output devices 116 .
- the database 112 further comprises a knowledge base 133 and scene and language data 135 .
- the sensors 103 1 to 103 n are optical sensors and the sensors 105 1 to 105 n are positional sensors.
- sensors 103 1 to 103 n may comprise infrared (IR) sensors, visible sensors, night-vision sensors, radiation signature sensors, radio-wave sensors or other types of optical sensors.
- Sensors 103 1 to 103 n simultaneously capture one or more images of a scene 153 , while the sensors 105 1 to 105 n capture data about the geographic location and orientation of the apparatus 100 .
- all of the sensors are physically coupled to the apparatus 100 .
- the one or more sensors 103 1 to 103 n may be housed in a telescope, a binocular unit, a headset, bifocals, or the like. In some instances, the sensors may be remotely located from the apparatus.
- One or more of the sensors 103 1 to 103 n comprise a sensor for capturing wide field of view images, e.g. 50 degrees to 80 degrees horizontal field of view, and one or more of the sensors 103 1 to 103 n comprise a sensor for capturing a narrow field of view images, e.g. 1 degree to 10 degrees.
- a wide angle image of the scene 153 provides context for narrower field of view images.
- the images captured by sensors 103 1 to 103 n are coupled to the object recognition module 102 .
- the object recognition module 102 performs object recognition on the wide field of view images and the narrow field of view images to recognize objects in all of the images.
- invariant image features are used to recognize objects in the images as described in commonly assigned and issued U.S. Pat.
- the object recognition module 102 is coupled to database 112 to receive invariant feature records and the like to assist in object recognition.
- the object recognition module 102 may also receive scene and language data 135 from the database 112 to localize wide angle images, as described in commonly assigned, co-pending U.S. patent application Ser. No. 13/493,654 filed Jun. 11th, 2012, herein incorporated by reference in its entirety.
- a GNSS received provides an initial bearing of apparatus 100 and an IMU can provide relative movements of the apparatus 100 such as rotation and acceleration. Together, six degrees of freedom (6DOF) can be obtained using the various sensors.
- 6DOF degrees of freedom
- a GPS signal may only be sporadically available.
- the apparatus 100 can still provide tracking of objects of interest during periods of sporadic GNSS reception.
- the signals previously received from the GNSS may be stored in a memory of the apparatus 100 .
- the localization module 108 calculates a projected geographical location of the apparatus 100 based on the previously stored GNSS location and trajectory of the apparatus 100 in conjunction with the trajectory of the objects of interest.
- the localization module 108 couples the localized object and apparatus information to the tracking module 104 .
- the tracking module 104 correlates the objects in the wide field of view mages and the narrow field of view images. For example, in wide field of view images, a dog and cat are recognized at a distance. In narrow field of view images, the dog and cat are recognized and correlated with the dog and cat in the wide field of view images because the location of the dog and cat are known from the localization module 108 .
- the tracking module 104 tracks the location of the dog and cat when a user switches from viewing a wide field of view of the scene to a narrow field of view of the scene 153 . This process is referred to as visual odometry, as described in commonly assigned, co-pending U.S. patent application Ser.
- a high resolution camera e.g., a camera that can capture greater than 50 MegaPixels (MP) may be used instead of two views of a scene with differing field of view.
- the camera enables a user to capture the wide field of view with a very high resolution, for example, 50 MP or more, and as the user zooms into a particular area of the wide field of view of the scene, there is enough pixel data to represent the narrow field of view of the scene also.
- the object recognition module 102 uses the wide field of view image to detect objects of interest, and the localization module 108 may use a laser rangefinder to calculate precise geographic coordinates of the objects of interest.
- the tracking module 104 enables a user of the apparatus 100 to track the objects of interest as the zoom on the camera is changed.
- the sensors 105 1 to 105 n may comprise navigation sensors such as a GNSS receiver, an IMU unit, a magnetometer, pressure sensors, a laser range-finder and the like.
- the localization module 108 localizes the apparatus 100 using the sensors 105 1 to 105 n and the narrow/wide field of view images captured by sensors 103 1 to 103 n . A three-dimensional location and orientation is established for the apparatus 100 with more accuracy than with the positional sensors 105 1 to 105 n alone.
- the object recognition module 102 may recognize objects within the wide field of view and narrow field of view of the scene 153 .
- a user of the apparatus 100 may select objects of interest by manually inputting selections in the field of view using a laser rangefinder or the like.
- the object recognition module 102 scans both fields of views and detects objects automatically, based on training data stored in database 112 . While a user pans the optical sensors 103 1 . . . n the object recognition module 102 detects objects of interest, for example, facial recognition, movement recognition, bird recognition, or the like.
- the localization module 108 further receives object recognition information from the object recognition module 102 to aid in localization.
- the localization module 108 uses the location of the park bench to determine the accuracy of the estimated distance d 1 of the oak tree from the apparatus 100 .
- the localization module 108 also locates objects of interest in the user's fields of view and generates guides to locate objects of interest.
- the apparatus 100 may automatically detect different types of birds in a scene 153 and automatically guide a user towards the birds, as well as augmented a user display with information regarding the birds.
- the user may mark a target bird, stationary or moving, while in a “zoomed out” field of view, and be guided by the registration of images on how to move the camera while zooming in without losing the target object, ensuring the target is still in view.
- the reasoning module 110 uses the localization knowledge and the object recognition knowledge from the database 112 to generate relevant content to be presented to the user.
- a word model consisting of geo-spatial information such as digital terrain, geographical tags and 30 models can be used in the reasoning module 110 .
- dynamic geographic data can be transmitted to the reasoning module 110 from an external source to augment the reasoning generated by the reasoning module 110 .
- overlay content can be customized in the reasoning module 110 .
- the overlay is also used in aiding in navigation of a user of the apparatus 100 .
- the apparatus 100 is a video see-through system, i.e., a system with an optical sensor as well as an optical viewfinder where content is rendered on the optical viewfinder as the user of apparatus 100 views a scene through the optical viewfinder.
- the content is fused with the captured images from the sensor 103 1 to 103 n for display on the viewfinder, allowing the user to view the augmented content overlaid on the scene 153 .
- Overlaid content may include dynamic information such as weather information, social networking information, flight information, traffic information, star-map information, and other external geographically located information fed into the system.
- dynamic information such as weather information, social networking information, flight information, traffic information, star-map information, and other external geographically located information fed into the system.
- Another example of geographically located information may include overlaying real-estate house and commercial sales information, tourist attractions, and the like on a view of the apparatus 100 .
- a visually impaired person wearing a single head or body mounted camera may be aided using the apparatus 100 .
- an implicit image mosaic or landmark images are generated by the AR tracking module 101 and stored in database 112 .
- An object of interest can be designated in a certain view by image processing by the object recognition module 102 or manually by the user. The visually impaired user is then able to move around, but be guided back to the designated view-point or location based on the geolocated apparatus 100 as well as through alignment of images from the camera with landmark and mosaic images stored in the database 112 .
- the database 112 may store images or videos of people of interest, such as celebrities, athletes, news-worthy figures, or the like.
- the object recognition module 102 may then perform object recognition while a user is scanning a scene and match against the people of interest stored in database 112 to identify people of interest in the scene.
- FIG. 2 depicts an embodiment of the apparatus 100 as a binocular unit 200 in accordance with exemplary embodiments of the present invention.
- the binocular unit 200 comprises a wide field of view optic 202 and a narrow field of view optic 204 , i.e., a wide field of view lens and a narrow field of view lens, an eyepiece 201 , a GNSS 210 , a magnetometer 214 , an IMU 212 and a laser rangefinder 213 .
- the laser rangefinder 213 may be replaced with an ultrasonic rangefinder or by estimating range from ray tracing to a 30 model of the scene given the user's location and view-point.
- the computer system 250 may be housed internally in the binocular unit 200 .
- the computer system 250 is remote from the binocular unit 200 and the binocular unit 200 transmits and receives data from the computer system 250 through wired transmission lines, or wirelessly.
- the wide FOV optic 202 is coupled to the camera 206 .
- the narrow FOV optic is coupled to the camera 208 .
- the camera 206 and camera 208 are coupled to a computer system 250 .
- the wide FOV optic 202 and the narrow FOV optic 204 are both coupled to the eyepiece 201 .
- the eyepiece 201 comprises, according to one embodiment, a first lens 201 A and a second lens 201 B. In other embodiments, the eyepiece 201 may comprise only a single viewing lens, or more than two viewing lenses.
- a user of the binocular unit 200 is able to switch between viewing the wide FOV optic view and he narrow FOV optic view in the eyepiece 201 .
- the laser rangefinder 213 outputs a laser towards a direction aligned with the orientation of the binocular unit 200 .
- the laser rangefinder 213 establishes distances to objects within the field of view of the binocular unit 200 .
- the laser rangefinder may be used to map the terrain into three-dimensional space, and store the 30 terrain map in the database 272 .
- the computer system 250 includes a processor 252 , various support circuits 256 , and memory 254 .
- the processor 252 may include one or more microprocessors known in the art.
- the support circuits 256 for the processor 252 include conventional cache, power supplies, clock circuits, data registers, I/O interface 257 , and the like.
- the I/O interface 257 may be directly coupled to the memory 254 or coupled through the supporting circuits 256 .
- the I/O interface 257 may also be configured for communication with input devices and/or output devices such as network devices, various storage devices, mouse, keyboard, display, video and audio sensors and the like.
- the memory 254 stores non-transient processor-executable instructions and/or data that may be executed by and/or used by the processor 252 .
- These processor-executable instructions may comprise firmware, software, and the like, or some combination thereof.
- Modules having processor-executable instructions that are stored in the memory 254 comprise an augmented reality (AR) tracking module 258 .
- the AR tracking module 258 further comprises an object recognition module 260 , a localization module 262 , a tracking module 264 , an augmented reality module 266 , a reasoning module 268 and a communication module 270 .
- the computer 250 may be programmed with one or more operating systems (generally referred to as operating system (OS)), which may include OS/2, Java Virtual Machine, Linux, SOLARIS, UNIX, HPUX, AIX, WINDOWS, WINDOWS95, WINDOWS98, WINDOWS NT, AND WINDOWS2000, WINDOWS ME, WINDOWS XP, WINDOWS SERVER, WINDOWS 8, Mac OS X, IOS, ANDROID among other known platforms.
- OS operating system
- OS operating system
- the memory 254 may include one or more of the following random access memory, read only memory, magneto-resistive read/write memory, optical read/write memory, cache memory, magnetic read/write memory, and the like, as well as signal-bearing media as described below.
- the camera 206 and the camera 208 are coupled to the support circuits 256 and the I/O interface 257 of the computer system 250 .
- the I/O interface 257 is further coupled to an overlay 214 of the eyepiece 201 .
- the object recognition module 260 recognizes objects within the wide FOV optic 202 and the narrow FOV optic 204 as the images are received by the cameras 206 and 208 , i.e., in real-time as the user of the binocular unit 200 scans a viewing area.
- the user can scan a larger area than that visible in the wide FOV optic 202 and have the object recognition module 260 scan for objects in the larger area.
- the user can identify, using the laser rangefinder 213 , or other visual means, an object of interest to track or recognize, and the object recognition module 260 will perform object recognition on that particular object, or the user designated area.
- the objects recognized by the object recognition module 260 are localized using the localization module 262 and tracked by the tracking module 264 .
- the tracking module 264 may also notify the augmented reality module 266 that a visual indication should be rendered to indicate the location of a particular recognized object, even if the object moves out of view of the binocular unit 200 .
- the augmented reality module 266 then creates an overlay for the narrow field of view based on the recognized object, requested content by the user, markers indicating the location of recognized objects, amongst additional augmented reality content known to those of ordinary skill in the art.
- the content is then coupled from the augmented reality module 266 to the binocular unit 200 and, in particular, to the eyepiece 201 as overlay 214 .
- the overlay 214 overlays the real-time view through the lenses 201 A and 201 B, i.e., as the user surveys a scene through the lenses 201 A and 201 B, the overlay 214 is also visible through the lenses 201 A and 201 B.
- the augmented reality module 266 generates the overlay 214 in geometrical and geographical correspondence with the terrain visible to the user through the binocular unit 200 .
- a binocular unit 200 may observe, with the wide FOV optic 202 , a scene depicted by the image 300 shown in FIG. 3 and with the narrow FOV optic 204 observe a scene depicted by the image 400 shown in FIG. 4 .
- image 300 a user has marked a landmark as a landmark of interest after the object recognition module 260 has recognized the objects in the image 300 and the localization module 262 localized the landmark of interest.
- the augmented reality module 266 places the marker 302 appropriately to indicate the landmark of interest according to the modified view, because the localization module 262 has precisely located the landmark in geographic space.
- the object may be tracked so that when a user of the unit 200 moves, the tracked objects of interest remain centered in the near field of view.
- the augmented reality module 266 can overlay the view from a binocular unit 200 to create a simulation.
- FIG. 5 depicts a wide FOV image 500 , where a user of a binocular unit 200 indicates that a helicopter 502 should be placed at a particular location.
- the augmented reality module 266 places the helicopter in the preferred location, and overlay the helicopter in real time on overlay 214 , where the position of the helicopter in the user's view changes as the user adjusts the binocular unit 200 .
- the AR module 266 scales, rotates and translates the helicopter simulation based on the geographic localization data of the binocular unit 200 and the rotational movement of the IMU 212 .
- Image 600 depicts another example of inserting simulated objects into a scene for aiding in simulated operations.
- Image 600 depicts a narrow field of view scene with a tank object 602 inserted into the scene.
- the tank 602 may be animated by the augmented reality module 266 , with a size, location and orientation corresponding to the other objects recognized by the object recognition module 260 and the orientation of the device according to the localization module 262 to create a mixed reality and simulated environment to aid in user training, such as procedural training, strategic training or the like.
- the reasoning module 268 may occlude the tank as it is animated to conform to the terrain depicted in image 600 .
- a user of the binocular unit 200 may scan a scene by spatially moving the binocular unit 200 to geographically locate the entire scanned contents of the scene.
- the reasoning module 268 can then generate a three-dimensional terrain based on the geographically located objects, terrain, and the like.
- the wide field of view and narrow field of view images can be used to construct a 30 map of an observed area by imaging the area based on multiple images, and then estimating the 30 by a motion stereo process accurately using the geographic location provided by the various sensors 210 - 214 , as described in U.S.
- FIG. 7A depicts a plurality of binocular units viewing portions of a scene 700 in accordance with exemplary embodiments of the present invention.
- FIG. 7 depicts a guidance system based on image processing and alignment of wide and narrow imagery.
- the binocular unit (BU) 702 , BU 704 and BU 706 comprises an augmented tracking module as depicted in FIG. 2 , communicating wired or wirelessly, between each other through the network 701 .
- each binocular unit is coupled, wired or wirelessly, to a remote computer system such as computer system 250 through the network 701 .
- the scene 700 comprises one or more objects, the one or more objects comprising a first person 708 , a vehicle 710 and a second person 712 .
- the scene 700 is being observed by three users: a first user operating BU 702 , a second user operating BU 704 and a third user operating BU 706 , each oriented in a different direction.
- each of the BU 702 , 704 and 706 may be oriented in three-dimensional space, as opposed to the 20 plane depicted.
- BU 704 may be positioned at a height above BU 706
- BU 704 may be looking at an area in scene 700 lower in height than the area n scene 700 being viewed by BU 706 .
- the BU 706 has person 708 and vehicle 710 in view.
- the operator of BU 706 identifies person 708 and vehicle 710 as objects of interest using a laser rangefinder, or in some other manner well known in the art such as delimiting a particular area wherein the person 708 and the vehicle 710 are located and directing the unit 706 to perform object recognition.
- the BUs is all coupled to the network 701 .
- the BUs 702 , 704 and 706 create an ad-hoc network and communicate directly with each other.
- the scene 700 is an example view of a field containing several objects: a person 708 , a vehicle 710 , a second person 712 and several trees and bushes 732 - 736 .
- the user of binocular unit 706 may mark the location using a laser rangefinder, or other means, of the person 708 and the vehicle 710 .
- a marker 709 identifies the location of the first person 708 and the marker 711 identifies the location of the vehicle 711 .
- the marker and location are shared across the network 701 with the other Bus 702 and 704 .
- the BU 705 marks the location of the second person 712 with a marker 713 .
- the first person 708 and the vehicle 710 are in the view 707 of BU 707 ; however they are out of the view of BU 704 . If the operator of BU 706 directs the attention of the user of the BU 704 towards the vehicle 710 and the first person 708 , the user will have indications in their view 705 showing where to turn their BU 704 to view the person 708 and the vehicle 710 . For example, indicator 714 is shown as pointing in the direction that the BU unit 704 must turn in order to see person 708 . Indicator 716 indicates the direction the BU 704 must turn to view vehicle 710 .
- the localization module of BU unit 706 since the localization module of BU unit 706 has localized vehicle 710 and first person 708 , the location is available to all units on the network 701 .
- the AR module of each of the BU units generates the indicators 716 and 714 .
- the AR module of the BU unit 706 generates an indicator 718 pointing in the direction of the person 712 and the AR module of the BU unit 702 generates a marker 720 pointing in the direction of the location of person 712 .
- the BU 702 also has first person 708 in its view 703 .
- the user of BU 702 may also mark the location of first person 708 using the laser rangefinder. Since both BU unit 702 and BU unit 706 have marked the location of person 708 , the localization module may localize person 708 more accurately given two perspectives and two distances from the BUs.
- the localization module may localize person 708 more accurately given two perspectives and two distances from the BUs.
- a user of any of the Bus pan their own view they do not lose sight of the objects in the scene 700 because of the indicators 714 , 716 , 718 and 720 , as well as the markers 709 , 711 and 712 .
- the AR modules of each BU may augment the BU view of the scene based on the determined location of the objects with many other types of data such as distance, motion estimation, threat level, or the like.
- facial and object recognition may be applied in an urban setting for recognizing faces and well-known structures to aid in guidance towards a particular landmark.
- a user of the BU 702 or even a mobile phone with a camera, may designate and identify real world objects in wide and narrow field of views, while panning the phone or keeping the device still. The object is then located according to the discussion above geographically in world space, and in local view.
- the AR module 106 then provides audio and/or visual guidance to another user on how to move their device to locate the target object.
- users performing the object tagging may attach messages to their tagged objects, or other tags such as images, video, audio or the like.
- Other users of mobile devices may then have objects of interest highlighted on their displays, or have directions towards the highlighted objects on their displays.
- one of the viewers of a scene may be viewing the scene aerially, while other viewers of a scene are on the ground and either party may tag/highlight objects of interest for the users to locate or observe.
- Binocular units 702 , 704 and 706 may be replaced with, for example, a head-mounted display/vision unit, a mobile camera or the like.
- a head-mounted display or vision unit may be wearable by a user, who can view the surrounding scene through the visor or glasses in the head-mounted unit, and also view the AR enhancements in the visor or glasses overlaid on top of the user's view of the scene.
- head-mounted units 702 , 704 and 706 may be engaged in a session of gaming (i.e., playing a video game) where objects are rendered onto each user's field of view with regard to each unit's frame of reference, view and angle.
- gaming i.e., playing a video game
- the scene can become a virtual gaming arena where users can view and use real-world objects with simulated objects to achieve particular goals, such as capturing a flag or scoring a point in a sport.
- virtual advertisements may be displayed on each user's view in a geographically sensible position as in-game advertisements.
- FIG. 7B depicts an exemplary use of the apparatus 100 in accordance with exemplary embodiments of the present invention.
- a user 740 uses the apparatus 100 , here shown as binocular unit 742 , to observe a scene 750 of a cityscape containing several landmarks and objects of interest.
- the view of unit 742 may have an augmented display showing the building 752 tagged with a tag 756 indicating that building 752 is the “Trans America Building”.
- the building 754 may also be tagged as the “Ferry Building Market Place”.
- the user 740 may attach a geographical tag (geo-tag) to the building 754 stating “Meet you at the Ferry Building Place” for members of their party as a coordination point.
- geo-tag geographical tag
- embodiments of the present invention may apply to tourism. Users may augment the view from their viewing units with landmarks and points of interest. Geo-tags may be left for friends. In addition, markups can be left on GOOGLE Earth or other mapping services.
- different detection algorithms can be plugged in to the object detection of the binocular units, or may be available in the cloud.
- the images captured by the binocular units may be processed by a local plug-in module or may be sent to a remote cloud for analysis.
- tracking by another system such as a unmanned aerial vehicle (UAV) may be sent to the binocular units for display.
- UAV unmanned aerial vehicle
- the user 740 may mark a moving object, while a second user using another viewing unit may perform an object detection search to detect the moving object and recognize the moving object.
- Geo-tags when users create Geo-tags, they may be stored in a local ad-hoc network created by similar viewing units throughout a particular area, or units manually synced together. Geo-tags and other messages may then be shared directly from one binocular unit another binocular unit.
- the geo-tags and other landmark/objects of interest tags and markings are stored on external servers, “cloud servers” for example, and each unit may access the cloud to access geo-tags associated with them (i.e., tags within the user's group), or global geo-tags depending on their preference.
- a second user's location is determined to be in the same vicinity as one of the geo-tags and the cloud server transmits the tags within the user's area.
- Geo-tags, objects of interest and guided directions towards objects of interest may then be downloaded from the external cloud servers and used to overlay an AR display on an AR viewing unit.
- users can share geo-tags, objects of interest, landmarks, and the like, with other users by giving an input command, such as a speech command or at the press of a menu item.
- the apparatus 100 may have a application programming interface (API) which allows developers to plug-in particular applications of object detection such as in recreational sports, bird-watching, tourism, hunting, hiking, law enforcement and the like.
- API application programming interface
- FIG. 8 depicts a method 800 for object detection in accordance with exemplary embodiments of the present invention.
- the method 800 depicts the functionality of the inferencing module augmented reality tracking module 258 of FIG. 2 , as executed by the processor 252 .
- the method begins at step 802 and proceeds to step 804 .
- one or more optical sensors capture one or more images of a scene containing wide field of view and narrow field of view images.
- the mages are captured and processed in real-time, i.e., as a user views a scene in the narrow field of view, a arger field of view mage containing the scene is also being captured by the optical sensors.
- the localization module 262 determines the location of the optical sensors.
- the user is directly holding, or otherwise coupled with, the location of the optical sensors, so that the location of the optical sensors is essentially equivalent to the location of the user of BU 200 , for example.
- the object recognition module recognizes objects in the images and, using the laser rangefinder 213 of the BU 200 , the distance between the optical sensor and the objects can be determined. Since the location of the BU 200 has been determined, and the distance to the objects has been determined, the location of the objects can also be accurately determined. In addition, if several binocular units observe the same objects, another degree of accuracy is added to the determined location of the objects.
- the AR module 266 enhances the view through the BU 200 based upon the determined locations of the objects.
- the AR module 266 overlays content on the user's view.
- the content may include markers of objects of interest, indicators directing the user of BU 200 to objects of interest outside of the current field of view, weather information, flight map information, and other information.
- the tracking module 264 tracks the one or more objects from the wide field of view to the narrow field of view as a user switches between the two fields of view.
- the user of BU 200 may view the wide FOV scene and sight a bird of interest. However, in order to see the bird in more detail, the user of BU 200 switches to a narrow field of view.
- the tracking module 264 tracks the objects from the wide FOV to the narrow FOV, so the user will be presented with a narrow FOV already containing the bird.
- the AR module 266 overlays an indication of the direction in which to rotate the BU 200 to locate the bird of interest.
- the BU 200 may use the communication module 270 to broadcast the determined locations of the BU 200 in addition to the objects of interest found, to other people in the area with mobile devices, binocular units, mobile computers or the like.
- the other people in the area may send the location of objects of interest outside of the user's field of view to the communication module 270 .
- the AR module 266 updates the overlay on the enhanced view indicating the direction in which to look to find those objects of interest outside the user's field of view.
- the method 800 moves to step 816 .
- the enhanced view is updated in real-time based on the determined location of the one or more detected objects as the user relocates the BU 200 .
- the view is further enhanced to incorporate external sources of geographically located information sources. For example, traffic may be overlaid on the BU 200 view, weather information, social media information, or the like.
- the AR module 266 may overlay virtual advertisements relevant to the user location. The method terminates at step 820 .
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Multimedia (AREA)
- Computer Graphics (AREA)
- Computer Hardware Design (AREA)
- General Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Computer Vision & Pattern Recognition (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Vascular Medicine (AREA)
- Signal Processing (AREA)
- Human Computer Interaction (AREA)
- Processing Or Creating Images (AREA)
- User Interface Of Digital Computer (AREA)
Abstract
Methods and apparatuses for tracking objects comprise one or more optical sensors for capturing one or more images of a scene, wherein the one or more optical sensors capture a wide field of view and corresponding narrow field of view for the one or more mages of a scene, a localization module, coupled to the one or more optical sensors for determining the location of the apparatus, and determining the location of one more objects in the one or more images based on the location of the apparatus and an augmented reality module, coupled to the localization module, for enhancing a view of the scene on a display based on the determined location of the one or more objects.
Description
- This application is a continuation of and claims priority to and the benefit of U.S. patent application Ser. No. 13/916,702, filed Jun. 13, 2013, which is and claims priority to and the benefit of U.S. provisional patent application No. 61/675,734 filed Jul. 25th, 2012 and U.S. provisional patent application No. 61/790,715 filed on Mar. 15th 2013, the disclosures of which are herein incorporated by reference in their entirety.
- This invention was made with U.S. government support under contract number N00014-11-C-0433. The U.S. government has certain rights in this invention.
- Field of the Invention
- Embodiments of the present invention generally relate to augmented reality and, more particularly, to a method and apparatus for tracking and geolocating objects of interest.
- Description of the Related Art
- Currently, binoculars are entirely optical devices allowing users to zoom in on a particular real world area from a long distance. If a user is attempting to view precise movements of an object at a distance, such as a car, truck or person, the user is able to use a binocular lens switching mechanism to change to a different magnification. In other binoculars, a “zooming” function is provided which can vary magnification ranges using a switch or lever. However, once the user increases magnification level, the user may experience difficulty in finding the object of interest within the “zoomed” scene.
- Further, in conventional binocular system, if there are several binocular users in communication and one user has identified one or more objects of interest such as interesting wildlife, people, or the like, difficulty arises in signaling the location of the object of interest to other binocular users. The user who sighted the object of interest may use landmarks, but this method is imprecise and landmarks may not be in view of other user's binocular systems. In addition there may be several similar landmarks, making it more difficult to identify the precise location of objects of interest.
- Therefore, there is a need in the art for a method and apparatus for precisely determining the geolocation of distant objects, in addition to tracking and sharing of the location for those objects.
- An apparatus and/or method for tracking and geolocating objects of interest comprising one or more optical sensors for capturing one or more images of a scene, wherein the one or more optical sensors capture a wide field of view and corresponding narrow field of view for the one or more mages of a scene, a localization module, coupled to the one or more optical sensors for determining the location of the apparatus, and determining the location of one more objects in the one or more images based on the location of the apparatus, and an augmented reality module, coupled to the localization module, for enhancing a view of the scene on a display based on the determined location of the one or more objects.
- Various advantages, aspects and features of the present disclosure, as well as details of an illustrated embodiment thereof, are more fully understood from the following description and drawings.
- So that the manner in which the above recited features of the present invention can be understood in detail, a more particular description of the invention, briefly summarized above, may be had by reference to embodiments, some of which are illustrated in the appended drawings. It is to be noted, however, that the appended drawings illustrate only typical embodiments of this invention and are therefore not to be considered limiting of its scope, for the invention may admit to other equally effective embodiments.
-
FIG. 1 depicts an apparatus for tracking objects in accordance with exemplary embodiments of the present invention; -
FIG. 2 depicts an embodiment of the apparatus as a binocular unit in accordance with exemplary embodiments of the present invention; -
FIG. 3 depicts an image captured by narrow field of view optics in accordance with exemplary embodiments of the present invention; -
FIG. 4 depicts an image captured by wide field of view optics in accordance with exemplary embodiments of the present invention; -
FIG. 5 depicts a wide field of view image, where a helicopter is placed at a particular location in accordance with exemplary embodiments of the present invention; -
FIG. 6 depicts a wide field of view image, where a tank is placed at a particular location in accordance with exemplary embodiments of the present invention; -
FIG. 7A depicts a plurality of binocular units viewing portions of a scene in accordance with exemplary embodiments of the present invention; -
FIG. 7B depicts an exemplary use of the apparatus 100 in accordance with exemplary embodiments of the present invention; and -
FIG. 8 depicts a method for object detection in accordance with exemplary embodiments of the present invention. - Embodiments of the present invention generally relate to a method and apparatus for tracking and geolocating objects of interest. According to one embodiment, an apparatus for tracking objects of interest comprises an optical sensor used in conjunction with positional sensors such as inertial measurement units (IMU) and global navigation satellite systems (such as geographical positioning satellite (GPS) systems, Glonass and Galileo) to locate the apparatus and a laser rangefinder, to geographically localize objects in reference to the location of the optical sensor. Objects of interest are tracked as a user zooms the optical sensors, or the user switches from a wide field of view to a narrow field of view. Embodiments of the present invention preserve a location of an object of interest within the user's view throughout a change in zoom or field of view. Additionally, if the object of interest escapes the user's view during a zooming function, an augmented reality system may provide the user with indicia of the location of the object within the user's view, as well as other pertinent information. An observed scene is augmented with information, labels of real world objects and guidance information on moving a camera unit or binocular unit to view objects of interest in a user's field of view. The augmentation appears stable with respect to the observed scene.
-
FIG. 1 depicts an apparatus for tracking objects 100 in accordance with exemplary embodiments of the present invention. The apparatus 100 comprises anobject recognition module 102,sensors 103 1 to 103 n and 105 1 to 105 n, atracking module 104, an augmentedreality module 106, alocalization module 108, areasoning module 110, adatabase 112, andoutput devices 116. According to exemplary embodiments, thedatabase 112 further comprises aknowledge base 133 and scene andlanguage data 135. - According to one embodiment, the
sensors 103 1 to 103 n are optical sensors and thesensors 105 1 to 105 n are positional sensors. In some embodiments,sensors 103 1 to 103 n may comprise infrared (IR) sensors, visible sensors, night-vision sensors, radiation signature sensors, radio-wave sensors or other types of optical sensors.Sensors 103 1 to 103 n simultaneously capture one or more images of ascene 153, while thesensors 105 1 to 105 n capture data about the geographic location and orientation of the apparatus 100. According to an exemplary embodiment, all of the sensors are physically coupled to the apparatus 100. In some embodiments, the one ormore sensors 103 1 to 103 n may be housed in a telescope, a binocular unit, a headset, bifocals, or the like. In some instances, the sensors may be remotely located from the apparatus. - One or more of the
sensors 103 1 to 103 n comprise a sensor for capturing wide field of view images, e.g. 50 degrees to 80 degrees horizontal field of view, and one or more of thesensors 103 1 to 103 n comprise a sensor for capturing a narrow field of view images, e.g. 1 degree to 10 degrees. A wide angle image of thescene 153 provides context for narrower field of view images. The images captured bysensors 103 1 to 103 n are coupled to theobject recognition module 102. Theobject recognition module 102 performs object recognition on the wide field of view images and the narrow field of view images to recognize objects in all of the images. According to exemplary embodiments, invariant image features are used to recognize objects in the images as described in commonly assigned and issued U.S. Pat. No. 8,330,819 filed on Apr. 12th, 2012, commonly assigned U.S. Pat. No. 8,345,988 B2 filed on Jun. 22nd, 2005 and U.S. Pat. No. 8,243,991 B2, filed on Aug. 14, 2012, herein incorporated by reference in their entirety. - Simultaneously capturing narrow field of view images with wide field of view images of the
scene 153 allow the vision algorithms to have additional context in recognizing objects of interest in a scene. The narrow and wide FOV images in conjunction with various positional sensors aid in high-fidelity localization, i.e., highly accurate geolocalization of objects of interest can be achieved. Further, theobject recognition module 102 is coupled todatabase 112 to receive invariant feature records and the like to assist in object recognition. Theobject recognition module 102 may also receive scene andlanguage data 135 from thedatabase 112 to localize wide angle images, as described in commonly assigned, co-pending U.S. patent application Ser. No. 13/493,654 filed Jun. 11th, 2012, herein incorporated by reference in its entirety. - For example, a GNSS received provides an initial bearing of apparatus 100 and an IMU can provide relative movements of the apparatus 100 such as rotation and acceleration. Together, six degrees of freedom (6DOF) can be obtained using the various sensors. In some instances, a GPS signal may only be sporadically available. According to some embodiment, the apparatus 100 can still provide tracking of objects of interest during periods of sporadic GNSS reception. The signals previously received from the GNSS may be stored in a memory of the apparatus 100. The
localization module 108 calculates a projected geographical location of the apparatus 100 based on the previously stored GNSS location and trajectory of the apparatus 100 in conjunction with the trajectory of the objects of interest. - The
localization module 108 couples the localized object and apparatus information to thetracking module 104. Thetracking module 104 correlates the objects in the wide field of view mages and the narrow field of view images. For example, in wide field of view images, a dog and cat are recognized at a distance. In narrow field of view images, the dog and cat are recognized and correlated with the dog and cat in the wide field of view images because the location of the dog and cat are known from thelocalization module 108. Thetracking module 104 tracks the location of the dog and cat when a user switches from viewing a wide field of view of the scene to a narrow field of view of thescene 153. This process is referred to as visual odometry, as described in commonly assigned, co-pending U.S. patent application Ser. No. 13/217,014 filed on Aug. 24th, 2011, herein incorporated by reference in its entirety. Accordingly, the visual odometry is performed on both the wide field of view and the narrow field of view simultaneously or in parallel. The wide field of view provides robustness, while the narrow field of view provides accuracy. Users may be provided with preset field of view angles; for example, a lens with ten steps of fields of view may be provided to the user. According to some embodiments, to support multiple fields of view, prior geometric calibration may be performed between images taken with different fields of view. - According to other embodiments, a high resolution camera, e.g., a camera that can capture greater than 50 MegaPixels (MP) may be used instead of two views of a scene with differing field of view. According to this embodiment, the camera enables a user to capture the wide field of view with a very high resolution, for example, 50 MP or more, and as the user zooms into a particular area of the wide field of view of the scene, there is enough pixel data to represent the narrow field of view of the scene also. The
object recognition module 102 uses the wide field of view image to detect objects of interest, and thelocalization module 108 may use a laser rangefinder to calculate precise geographic coordinates of the objects of interest. Thetracking module 104 enables a user of the apparatus 100 to track the objects of interest as the zoom on the camera is changed. - According to exemplary embodiments, the
sensors 105 1 to 105 n may comprise navigation sensors such as a GNSS receiver, an IMU unit, a magnetometer, pressure sensors, a laser range-finder and the like. Thelocalization module 108 localizes the apparatus 100 using thesensors 105 1 to 105 n and the narrow/wide field of view images captured bysensors 103 1 to 103 n. A three-dimensional location and orientation is established for the apparatus 100 with more accuracy than with thepositional sensors 105 1 to 105 n alone. Refer to commonly assigned, co-pending U.S. patent application Ser. No. 13/217,014 filed on Aug. 24th 2011, and commonly assigned U.S. Pat. No. 8,174,568 filed on Dec. 3rd, 2007, herein incorporated by reference in their entireties, for more detail regarding calculating 30 coordinates based on an image and navigational sensors. - Once the location of the apparatus 100 is calculated, the
object recognition module 102 may recognize objects within the wide field of view and narrow field of view of thescene 153. A user of the apparatus 100 may select objects of interest by manually inputting selections in the field of view using a laser rangefinder or the like. In other instances, theobject recognition module 102 scans both fields of views and detects objects automatically, based on training data stored indatabase 112. While a user pans theoptical sensors 103 1 . . . n theobject recognition module 102 detects objects of interest, for example, facial recognition, movement recognition, bird recognition, or the like. According to exemplary embodiments, thelocalization module 108 further receives object recognition information from theobject recognition module 102 to aid in localization. For example, if theobject recognition module 102 recognizes an oak tree at a distance d1 from a park bench at a known location inscene 153 in a wide angle image, and a distance d2 from the apparatus 100, thelocalization module 108 uses the location of the park bench to determine the accuracy of the estimated distance d1 of the oak tree from the apparatus 100. - The
localization module 108 also locates objects of interest in the user's fields of view and generates guides to locate objects of interest. For example, the apparatus 100 may automatically detect different types of birds in ascene 153 and automatically guide a user towards the birds, as well as augmented a user display with information regarding the birds. The user may mark a target bird, stationary or moving, while in a “zoomed out” field of view, and be guided by the registration of images on how to move the camera while zooming in without losing the target object, ensuring the target is still in view. - The
reasoning module 110 uses the localization knowledge and the object recognition knowledge from thedatabase 112 to generate relevant content to be presented to the user. A word model consisting of geo-spatial information such as digital terrain, geographical tags and 30 models can be used in thereasoning module 110. Similarly dynamic geographic data can be transmitted to thereasoning module 110 from an external source to augment the reasoning generated by thereasoning module 110. Based on geo-location and context, overlay content can be customized in thereasoning module 110. The overlay is also used in aiding in navigation of a user of the apparatus 100. - The overlay content is then projected using the augmented
reality module 106 in the accurate coordinate system of the narrow field of view optical sensors as derived by thelocalization module 108. Additionally the world model may be used by thereasoning module 110 to determine occlusion information during the rendering process. According to one embodiment, the apparatus 100 is a video see-through system, i.e., a system with an optical sensor as well as an optical viewfinder where content is rendered on the optical viewfinder as the user of apparatus 100 views a scene through the optical viewfinder. In this embodiment, the content is fused with the captured images from thesensor 103 1 to 103 n for display on the viewfinder, allowing the user to view the augmented content overlaid on thescene 153. - Overlaid content may include dynamic information such as weather information, social networking information, flight information, traffic information, star-map information, and other external geographically located information fed into the system. Another example of geographically located information may include overlaying real-estate house and commercial sales information, tourist attractions, and the like on a view of the apparatus 100.
- According to one embodiment, a visually impaired person wearing a single head or body mounted camera may be aided using the apparatus 100. As a visually impaired individual pans his or her head, an implicit image mosaic or landmark images are generated by the
AR tracking module 101 and stored indatabase 112. An object of interest can be designated in a certain view by image processing by theobject recognition module 102 or manually by the user. The visually impaired user is then able to move around, but be guided back to the designated view-point or location based on the geolocated apparatus 100 as well as through alignment of images from the camera with landmark and mosaic images stored in thedatabase 112. - According to other embodiments, the
database 112 may store images or videos of people of interest, such as celebrities, athletes, news-worthy figures, or the like. Theobject recognition module 102 may then perform object recognition while a user is scanning a scene and match against the people of interest stored indatabase 112 to identify people of interest in the scene. -
FIG. 2 depicts an embodiment of the apparatus 100 as abinocular unit 200 in accordance with exemplary embodiments of the present invention. Thebinocular unit 200 comprises a wide field ofview optic 202 and a narrow field ofview optic 204, i.e., a wide field of view lens and a narrow field of view lens, aneyepiece 201, aGNSS 210, amagnetometer 214, anIMU 212 and alaser rangefinder 213. According to some embodiments, thelaser rangefinder 213 may be replaced with an ultrasonic rangefinder or by estimating range from ray tracing to a 30 model of the scene given the user's location and view-point. - According to some embodiments, the
computer system 250 may be housed internally in thebinocular unit 200. In other embodiments, thecomputer system 250 is remote from thebinocular unit 200 and thebinocular unit 200 transmits and receives data from thecomputer system 250 through wired transmission lines, or wirelessly. - The
wide FOV optic 202 is coupled to thecamera 206. The narrow FOV optic is coupled to thecamera 208. Thecamera 206 andcamera 208 are coupled to acomputer system 250. Thewide FOV optic 202 and the narrow FOV optic 204 are both coupled to theeyepiece 201. Theeyepiece 201 comprises, according to one embodiment, afirst lens 201A and asecond lens 201B. In other embodiments, theeyepiece 201 may comprise only a single viewing lens, or more than two viewing lenses. - In some embodiments, a user of the
binocular unit 200 is able to switch between viewing the wide FOV optic view and he narrow FOV optic view in theeyepiece 201. Thelaser rangefinder 213 outputs a laser towards a direction aligned with the orientation of thebinocular unit 200. Thelaser rangefinder 213 establishes distances to objects within the field of view of thebinocular unit 200. In addition, the laser rangefinder may be used to map the terrain into three-dimensional space, and store the 30 terrain map in thedatabase 272. - The
computer system 250 includes aprocessor 252,various support circuits 256, andmemory 254. Theprocessor 252 may include one or more microprocessors known in the art. Thesupport circuits 256 for theprocessor 252 include conventional cache, power supplies, clock circuits, data registers, I/O interface 257, and the like. The I/O interface 257 may be directly coupled to thememory 254 or coupled through the supportingcircuits 256. The I/O interface 257 may also be configured for communication with input devices and/or output devices such as network devices, various storage devices, mouse, keyboard, display, video and audio sensors and the like. - The
memory 254, or computer readable medium, stores non-transient processor-executable instructions and/or data that may be executed by and/or used by theprocessor 252. These processor-executable instructions may comprise firmware, software, and the like, or some combination thereof. Modules having processor-executable instructions that are stored in thememory 254 comprise an augmented reality (AR)tracking module 258. TheAR tracking module 258 further comprises anobject recognition module 260, alocalization module 262, atracking module 264, anaugmented reality module 266, areasoning module 268 and acommunication module 270. - The
computer 250 may be programmed with one or more operating systems (generally referred to as operating system (OS)), which may include OS/2, Java Virtual Machine, Linux, SOLARIS, UNIX, HPUX, AIX, WINDOWS, WINDOWS95, WINDOWS98, WINDOWS NT, AND WINDOWS2000, WINDOWS ME, WINDOWS XP, WINDOWS SERVER, WINDOWS 8, Mac OS X, IOS, ANDROID among other known platforms. At least a portion of the operating system may be disposed in the memory 404. - The
memory 254 may include one or more of the following random access memory, read only memory, magneto-resistive read/write memory, optical read/write memory, cache memory, magnetic read/write memory, and the like, as well as signal-bearing media as described below. - The
camera 206 and thecamera 208 are coupled to thesupport circuits 256 and the I/O interface 257 of thecomputer system 250. The I/O interface 257 is further coupled to anoverlay 214 of theeyepiece 201. Theobject recognition module 260 recognizes objects within thewide FOV optic 202 and the narrow FOV optic 204 as the images are received by thecameras binocular unit 200 scans a viewing area. According to one embodiment, the user can scan a larger area than that visible in thewide FOV optic 202 and have theobject recognition module 260 scan for objects in the larger area. Additionally, the user can identify, using thelaser rangefinder 213, or other visual means, an object of interest to track or recognize, and theobject recognition module 260 will perform object recognition on that particular object, or the user designated area. - The objects recognized by the
object recognition module 260 are localized using thelocalization module 262 and tracked by thetracking module 264. Thetracking module 264 may also notify theaugmented reality module 266 that a visual indication should be rendered to indicate the location of a particular recognized object, even if the object moves out of view of thebinocular unit 200. Theaugmented reality module 266 then creates an overlay for the narrow field of view based on the recognized object, requested content by the user, markers indicating the location of recognized objects, amongst additional augmented reality content known to those of ordinary skill in the art. The content is then coupled from theaugmented reality module 266 to thebinocular unit 200 and, in particular, to theeyepiece 201 asoverlay 214. According to this embodiment, theoverlay 214 overlays the real-time view through thelenses lenses overlay 214 is also visible through thelenses augmented reality module 266 generates theoverlay 214 in geometrical and geographical correspondence with the terrain visible to the user through thebinocular unit 200. - For example, a
binocular unit 200 may observe, with thewide FOV optic 202, a scene depicted by theimage 300 shown inFIG. 3 and with the narrow FOV optic 204 observe a scene depicted by theimage 400 shown inFIG. 4 . Inimage 300, a user has marked a landmark as a landmark of interest after theobject recognition module 260 has recognized the objects in theimage 300 and thelocalization module 262 localized the landmark of interest. When the user zooms or alternates views to the narrow field of view from theeyepiece 201, theaugmented reality module 266 places themarker 302 appropriately to indicate the landmark of interest according to the modified view, because thelocalization module 262 has precisely located the landmark in geographic space. According to other embodiments, the object may be tracked so that when a user of theunit 200 moves, the tracked objects of interest remain centered in the near field of view. - According to another embodiment, the
augmented reality module 266 can overlay the view from abinocular unit 200 to create a simulation. For example,FIG. 5 depicts awide FOV image 500, where a user of abinocular unit 200 indicates that ahelicopter 502 should be placed at a particular location. Theaugmented reality module 266 places the helicopter in the preferred location, and overlay the helicopter in real time onoverlay 214, where the position of the helicopter in the user's view changes as the user adjusts thebinocular unit 200. TheAR module 266 scales, rotates and translates the helicopter simulation based on the geographic localization data of thebinocular unit 200 and the rotational movement of theIMU 212.FIG. 6 depicts another example of inserting simulated objects into a scene for aiding in simulated operations.Image 600 depicts a narrow field of view scene with atank object 602 inserted into the scene. According to exemplary embodiments, thetank 602 may be animated by theaugmented reality module 266, with a size, location and orientation corresponding to the other objects recognized by theobject recognition module 260 and the orientation of the device according to thelocalization module 262 to create a mixed reality and simulated environment to aid in user training, such as procedural training, strategic training or the like. - Additionally, the
reasoning module 268 may occlude the tank as it is animated to conform to the terrain depicted inimage 600. According to another embodiment, a user of thebinocular unit 200 may scan a scene by spatially moving thebinocular unit 200 to geographically locate the entire scanned contents of the scene. Thereasoning module 268 can then generate a three-dimensional terrain based on the geographically located objects, terrain, and the like. The wide field of view and narrow field of view images can be used to construct a 30 map of an observed area by imaging the area based on multiple images, and then estimating the 30 by a motion stereo process accurately using the geographic location provided by the various sensors 210-214, as described in U.S. patent application Ser. No. 13/493,654 filed on Jun. 11th, 2012, herein incorporated by reference in its entirety. -
FIG. 7A depicts a plurality of binocular units viewing portions of ascene 700 in accordance with exemplary embodiments of the present invention. In accordance with exemplary embodiments,FIG. 7 depicts a guidance system based on image processing and alignment of wide and narrow imagery. In this embodiment, the binocular unit (BU) 702,BU 704 andBU 706 comprises an augmented tracking module as depicted inFIG. 2 , communicating wired or wirelessly, between each other through thenetwork 701. In other embodiments, each binocular unit is coupled, wired or wirelessly, to a remote computer system such ascomputer system 250 through thenetwork 701. - The
scene 700 comprises one or more objects, the one or more objects comprising afirst person 708, avehicle 710 and asecond person 712. Thescene 700 is being observed by three users: a firstuser operating BU 702, a seconduser operating BU 704 and a thirduser operating BU 706, each oriented in a different direction. Though not shown inFIG. 7 , each of theBU BU 704 may be positioned at a height aboveBU 706, andBU 704 may be looking at an area inscene 700 lower in height than thearea n scene 700 being viewed byBU 706. - The
BU 706 hasperson 708 andvehicle 710 in view. The operator ofBU 706 identifiesperson 708 andvehicle 710 as objects of interest using a laser rangefinder, or in some other manner well known in the art such as delimiting a particular area wherein theperson 708 and thevehicle 710 are located and directing theunit 706 to perform object recognition. - The BUs is all coupled to the
network 701. According to some embodiments, theBUs - The
scene 700 is an example view of a field containing several objects: aperson 708, avehicle 710, asecond person 712 and several trees and bushes 732-736. According to one embodiment of the present invention, the user ofbinocular unit 706 may mark the location using a laser rangefinder, or other means, of theperson 708 and thevehicle 710. Amarker 709 identifies the location of thefirst person 708 and themarker 711 identifies the location of thevehicle 711. The marker and location are shared across thenetwork 701 with theother Bus BU 705 marks the location of thesecond person 712 with amarker 713. - The
first person 708 and thevehicle 710 are in theview 707 ofBU 707; however they are out of the view ofBU 704. If the operator ofBU 706 directs the attention of the user of theBU 704 towards thevehicle 710 and thefirst person 708, the user will have indications in theirview 705 showing where to turn theirBU 704 to view theperson 708 and thevehicle 710. For example,indicator 714 is shown as pointing in the direction that theBU unit 704 must turn in order to seeperson 708.Indicator 716 indicates the direction theBU 704 must turn to viewvehicle 710. According to this embodiment, since the localization module ofBU unit 706 has localizedvehicle 710 andfirst person 708, the location is available to all units on thenetwork 701. The AR module of each of the BU units generates theindicators - Similarly, the AR module of the
BU unit 706 generates anindicator 718 pointing in the direction of theperson 712 and the AR module of theBU unit 702 generates amarker 720 pointing in the direction of the location ofperson 712. - The
BU 702 also hasfirst person 708 in itsview 703. The user ofBU 702 may also mark the location offirst person 708 using the laser rangefinder. Since bothBU unit 702 andBU unit 706 have marked the location ofperson 708, the localization module may localizeperson 708 more accurately given two perspectives and two distances from the BUs. In addition, when a user of any of the Bus pan their own view, they do not lose sight of the objects in thescene 700 because of theindicators markers - According to other embodiments, the AR modules of each BU may augment the BU view of the scene based on the determined location of the objects with many other types of data such as distance, motion estimation, threat level, or the like. In other instances, facial and object recognition may be applied in an urban setting for recognizing faces and well-known structures to aid in guidance towards a particular landmark. A user of the
BU 702, or even a mobile phone with a camera, may designate and identify real world objects in wide and narrow field of views, while panning the phone or keeping the device still. The object is then located according to the discussion above geographically in world space, and in local view. TheAR module 106 then provides audio and/or visual guidance to another user on how to move their device to locate the target object. In addition, users performing the object tagging may attach messages to their tagged objects, or other tags such as images, video, audio or the like. - Other users of mobile devices may then have objects of interest highlighted on their displays, or have directions towards the highlighted objects on their displays. In some embodiments, one of the viewers of a scene may be viewing the scene aerially, while other viewers of a scene are on the ground and either party may tag/highlight objects of interest for the users to locate or observe.
- Those of ordinary skill in the art would recognize that the
Binocular units - For example, several users of head-mounted
units -
FIG. 7B depicts an exemplary use of the apparatus 100 in accordance with exemplary embodiments of the present invention. Auser 740 uses the apparatus 100, here shown asbinocular unit 742, to observe ascene 750 of a cityscape containing several landmarks and objects of interest. For example, the view ofunit 742 may have an augmented display showing thebuilding 752 tagged with atag 756 indicating that building 752 is the “Trans America Building”. Thebuilding 754 may also be tagged as the “Ferry Building Market Place”. In addition, theuser 740 may attach a geographical tag (geo-tag) to thebuilding 754 stating “Meet you at the Ferry Building Place” for members of their party as a coordination point. Accordingly, embodiments of the present invention may apply to tourism. Users may augment the view from their viewing units with landmarks and points of interest. Geo-tags may be left for friends. In addition, markups can be left on GOOGLE Earth or other mapping services. In exemplary embodiments, different detection algorithms can be plugged in to the object detection of the binocular units, or may be available in the cloud. The images captured by the binocular units may be processed by a local plug-in module or may be sent to a remote cloud for analysis. Alternatively, tracking by another system such as a unmanned aerial vehicle (UAV) may be sent to the binocular units for display. For moving objects, theuser 740 may mark a moving object, while a second user using another viewing unit may perform an object detection search to detect the moving object and recognize the moving object. - According to some instances, when users create Geo-tags, they may be stored in a local ad-hoc network created by similar viewing units throughout a particular area, or units manually synced together. Geo-tags and other messages may then be shared directly from one binocular unit another binocular unit. In other embodiments, the geo-tags and other landmark/objects of interest tags and markings are stored on external servers, “cloud servers” for example, and each unit may access the cloud to access geo-tags associated with them (i.e., tags within the user's group), or global geo-tags depending on their preference. In this embodiment, a second user's location is determined to be in the same vicinity as one of the geo-tags and the cloud server transmits the tags within the user's area. Geo-tags, objects of interest and guided directions towards objects of interest may then be downloaded from the external cloud servers and used to overlay an AR display on an AR viewing unit. In other embodiments, users can share geo-tags, objects of interest, landmarks, and the like, with other users by giving an input command, such as a speech command or at the press of a menu item.
- According to exemplary embodiments, the apparatus 100 may have a application programming interface (API) which allows developers to plug-in particular applications of object detection such as in recreational sports, bird-watching, tourism, hunting, hiking, law enforcement and the like.
-
FIG. 8 depicts amethod 800 for object detection in accordance with exemplary embodiments of the present invention. Themethod 800 depicts the functionality of the inferencing module augmentedreality tracking module 258 ofFIG. 2 , as executed by theprocessor 252. The method begins atstep 802 and proceeds to step 804. - At
step 804, one or more optical sensors capture one or more images of a scene containing wide field of view and narrow field of view images. In some embodiments, the mages are captured and processed in real-time, i.e., as a user views a scene in the narrow field of view, a arger field of view mage containing the scene is also being captured by the optical sensors. - At
step 806, thelocalization module 262, determines the location of the optical sensors. In this embodiment, the user is directly holding, or otherwise coupled with, the location of the optical sensors, so that the location of the optical sensors is essentially equivalent to the location of the user ofBU 200, for example. The object recognition module recognizes objects in the images and, using thelaser rangefinder 213 of theBU 200, the distance between the optical sensor and the objects can be determined. Since the location of theBU 200 has been determined, and the distance to the objects has been determined, the location of the objects can also be accurately determined. In addition, if several binocular units observe the same objects, another degree of accuracy is added to the determined location of the objects. - At
step 808, theAR module 266 enhances the view through theBU 200 based upon the determined locations of the objects. As described above, theAR module 266 overlays content on the user's view. The content may include markers of objects of interest, indicators directing the user ofBU 200 to objects of interest outside of the current field of view, weather information, flight map information, and other information. - At
step 810, thetracking module 264 tracks the one or more objects from the wide field of view to the narrow field of view as a user switches between the two fields of view. For example, the user ofBU 200 may view the wide FOV scene and sight a bird of interest. However, in order to see the bird in more detail, the user ofBU 200 switches to a narrow field of view. - Under ordinary circumstances, the user would have difficulty in finding the bird again in the narrow field of view. However, the
tracking module 264 tracks the objects from the wide FOV to the narrow FOV, so the user will be presented with a narrow FOV already containing the bird. According to another embodiment, when a user switches from a wide FOV to a narrow FOV, theAR module 266 overlays an indication of the direction in which to rotate theBU 200 to locate the bird of interest. - At
step 812, theBU 200 may use thecommunication module 270 to broadcast the determined locations of theBU 200 in addition to the objects of interest found, to other people in the area with mobile devices, binocular units, mobile computers or the like. In addition, the other people in the area may send the location of objects of interest outside of the user's field of view to thecommunication module 270. Atstep 814, accordingly, theAR module 266 updates the overlay on the enhanced view indicating the direction in which to look to find those objects of interest outside the user's field of view. - Subsequently, the
method 800 moves to step 816. Atstep 816, the enhanced view is updated in real-time based on the determined location of the one or more detected objects as the user relocates theBU 200. Atstep 818, the view is further enhanced to incorporate external sources of geographically located information sources. For example, traffic may be overlaid on theBU 200 view, weather information, social media information, or the like. According to some embodiments, based on the determined location of theBU 200 and selected objects of interest, theAR module 266 may overlay virtual advertisements relevant to the user location. The method terminates atstep 820. - Various elements, devices, modules and circuits are described above in association with their respective functions. These elements, devices, modules and circuits are considered means for performing their respective functions as described herein. While the foregoing is directed to embodiments of the present invention, other and further embodiments of the invention may be devised without departing from the basic scope thereof, and the scope thereof is determined by the claims that follow.
Claims (31)
1.-22. (canceled)
23. A user-carried or user-worn apparatus for tracking objects, the apparatus comprising:
one or more optical sensors for capturing images of a scene, wherein the images include a wide field of view image of the scene and a narrow field of view image of the scene;
a display for displaying a displayed view of the scene; and
a processor coupled to a storage medium, the storage medium storing processor-executable instructions, which when executed by the processor, performs a method comprising:
geolocating the one or more optical sensors and a location of one or more objects depicted in the captured images based on a location of the one or more optical sensors;
recognizing an object of interest in the wide field of view image;
recognizing the object of interest in the narrow field of view image;
correlating a location of the recognized object of interest in the wide field of view image and the narrow field of view image; and
enhancing the displayed view of the scene on the display based on the geolocating of the one or more objects, wherein the enhanced, displayed view comprises overlay content for the recognized object of interest.
24. The apparatus of claim 23 , wherein the method performed by the processor based on the processor-executable instructions further comprises:
broadcasting the tracked location of the one or more objects and receiving tracked location information of objects outside a field of view of the one or more optical sensors, the received, tracked location information being received from other apparatuses.
25. The apparatus of claim 23 , wherein the user-carried or user-worn apparatus is configured to exchange tracked location information of the one or more objects with at least one other user-carried or user-worn apparatus and enhance the displayed view of the scene on the display based on the geolocating of the one or more objects by the apparatus and also based on tracked location information received from the at least one other user-carried or user-worn apparatus.
26. The apparatus of claim 23 , wherein the apparatus is in communication with one or more similar apparatuses such that users of the apparatuses utilize the apparatuses in a gaming activity in a physical area.
27. The apparatus of claim 23 , wherein when there are multiple instances of the apparatus in remote locations, geolocation is performed with respect to all of the sensors included in the multiple instances of the apparatus.
28. The apparatus of claim 23 , wherein the enhancing of the displayed view further comprises augmenting the displayed view to indicate location of the one or more objects outside the field of view of the one or more optical sensors.
29. The apparatus of claim 28 , wherein the augmentation of the displayed view includes insertion of one or more markers of objects of interest.
30. The apparatus of claim 28 , wherein the augmentation of the displayed view includes insertion of one or more indicators associated with one or more objects of interest.
31. The apparatus of claim 30 , wherein the one or more indicators provide direction to the one or more objects of interest.
32. The apparatus of claim 31 , wherein the one or more indicators provide direction to the one or more objects of interest outside of the current field of view outside a field of view of the one or more optical sensors.
33. The apparatus of claim 23 , wherein the correlating is performed by tracking the location of the recognized object of interest during transition from the wide field of view image to the narrow field of view image.
34. The apparatus of claim 33 , wherein the overlay content for the recognized object of interest is scaled in accordance with the tracking of the location of the recognized object of interest from the wide field of view to the narrow field of view.
35. The apparatus of claim 23 , wherein the method performed by the processor based on the processor-executable instructions further comprises:
maintaining the enhanced, displayed view consistent in real-time with determined location of the recognized object of interest as a user of the apparatus relocates the apparatus.
36. The apparatus of claim 23 , wherein the method performed by the processor based on the processor-executable instructions further comprises:
inserting objects into the enhanced, displayed view based on geographic data that indicates that the inserted object is to be occluded by another object in the enhanced, displayed view.
37. The apparatus of claim 23 , wherein enhancing the displayed view comprises overlaying geographically located information from external sources onto the displayed view.
38. A method for tracking objects by a user-carried or user-worn apparatus, the method comprising:
capturing, using one or more optical sensors, images of a scene, wherein the images include a wide field of view image of the scene and a narrow field of view image of the scene;
geolocating the one or more optical sensors, and a location of one or more objects depicted in the captured images based on a location of the one or more optical sensors;
recognizing an object of interest in the wide field of view image;
recognizing the object of interest in the narrow field of view image;
correlating a location of the recognized object of interest in the wide field of view image and the narrow field of view image; and
enhancing a displayed view of the scene on a display of the user-carried or user-worn apparatus based on the geolocating of the one or more objects, wherein the enhanced, displayed view comprises overlay content for the recognized object of interest.
39. The method of claim 38 , further comprising broadcasting the tracked location of the one or more objects and receiving tracked location information of objects outside a field of view of the one or more optical sensors, the received, tracked location information being received from other apparatuses.
40. The method of claim 38 , wherein the user-carried or user-worn apparatus is configured to exchange tracked location information of the one or more objects with at least one other user-carried or user-worn apparatus and enhance the displayed view of the scene on the display based on the geolocating of the one or more objects by the apparatus and also based on tracked location information received from the at least one other user-carried or user-worn apparatus.
41. The method of claim 38 , wherein when there are multiple instances of the apparatus in remote locations, geolocation is performed with respect to all of the sensors included in the multiple instances of the apparatus.
42. The method of claim 38 , wherein the enhancing of the displayed view further comprises augmenting the displayed view to indicate location of the one or more objects outside the field of view of the one or more optical sensors.
43. The method of claim 42 , wherein the augmentation of the displayed view includes insertion of one or more markers of objects of interest.
44. The method of claim 42 , wherein the augmentation of the displayed view includes insertion of one or more indicators associated with one or more objects of interest.
45. The method of claim 44 , wherein the one or more indicators provide direction to the one or more objects of interest.
46. The method of claim 44 , wherein the one or more indicators provide direction to the one or more objects of interest outside of the current field of view outside a field of view of the one or more optical sensors.
47. The method of claim 38 , wherein the correlating is performed by tracking the location of the recognized object of interest during transition from the wide field of view image to the narrow field of view image.
47. The apparatus of claim 47 , wherein the overlay content for the recognized object of interest is scaled in accordance with the tracking of the location of the recognized object of interest from the wide field of view to the narrow field of view.
48. The method of claim 38 , further comprising maintaining the enhanced, displayed view consistent in real-time with determined location of the recognized object of interest as a user of the apparatus relocates the apparatus.
49. The method of claim 38 , further comprising inserting objects into the enhanced, displayed view based on geographic data that indicates that the inserted object is to be occluded by another object in the enhanced, displayed view.
50. The method of claim 38 , wherein enhancing the displayed view comprises overlaying geographically located information from external sources onto the displayed view.
51. The method of claim 38 , further comprising:
capturing, using a first lens of the one or more optical sensors, the wide field of view image; and
capturing, using a second lens the one or more optical sensors, the narrow field of view image.
Priority Applications (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US15/286,161 US20170024904A1 (en) | 2012-07-25 | 2016-10-05 | Augmented reality vision system for tracking and geolocating objects of interest |
US17/157,065 US11423586B2 (en) | 2012-07-25 | 2021-01-25 | Augmented reality vision system for tracking and geolocating objects of interest |
Applications Claiming Priority (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201261675734P | 2012-07-25 | 2012-07-25 | |
US201361790715P | 2013-03-15 | 2013-03-15 | |
US13/916,702 US9495783B1 (en) | 2012-07-25 | 2013-06-13 | Augmented reality vision system for tracking and geolocating objects of interest |
US15/286,161 US20170024904A1 (en) | 2012-07-25 | 2016-10-05 | Augmented reality vision system for tracking and geolocating objects of interest |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US13/916,702 Continuation US9495783B1 (en) | 2012-07-25 | 2013-06-13 | Augmented reality vision system for tracking and geolocating objects of interest |
Related Child Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US17/157,065 Continuation US11423586B2 (en) | 2012-07-25 | 2021-01-25 | Augmented reality vision system for tracking and geolocating objects of interest |
Publications (1)
Publication Number | Publication Date |
---|---|
US20170024904A1 true US20170024904A1 (en) | 2017-01-26 |
Family
ID=57235007
Family Applications (3)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US13/916,702 Active 2034-03-20 US9495783B1 (en) | 2012-07-25 | 2013-06-13 | Augmented reality vision system for tracking and geolocating objects of interest |
US15/286,161 Abandoned US20170024904A1 (en) | 2012-07-25 | 2016-10-05 | Augmented reality vision system for tracking and geolocating objects of interest |
US17/157,065 Active US11423586B2 (en) | 2012-07-25 | 2021-01-25 | Augmented reality vision system for tracking and geolocating objects of interest |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US13/916,702 Active 2034-03-20 US9495783B1 (en) | 2012-07-25 | 2013-06-13 | Augmented reality vision system for tracking and geolocating objects of interest |
Family Applications After (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US17/157,065 Active US11423586B2 (en) | 2012-07-25 | 2021-01-25 | Augmented reality vision system for tracking and geolocating objects of interest |
Country Status (1)
Country | Link |
---|---|
US (3) | US9495783B1 (en) |
Cited By (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9875403B2 (en) * | 2012-12-12 | 2018-01-23 | Thales | Method for accurately geolocating an image sensor installed on board an aircraft |
WO2018151382A1 (en) * | 2017-02-16 | 2018-08-23 | (주)원우이엔지 | Telescope including camera function, and image display method using same |
CN110220493A (en) * | 2018-03-02 | 2019-09-10 | 北京图森未来科技有限公司 | A kind of binocular distance measuring method and its device |
Families Citing this family (63)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2015139005A1 (en) * | 2014-03-14 | 2015-09-17 | Sony Computer Entertainment Inc. | Methods and systems tracking head mounted display (hmd) and calibrations for hmd headband adjustments |
US9713871B2 (en) * | 2015-04-27 | 2017-07-25 | Microsoft Technology Licensing, Llc | Enhanced configuration and control of robots |
US10007413B2 (en) | 2015-04-27 | 2018-06-26 | Microsoft Technology Licensing, Llc | Mixed environment display of attached control elements |
US10339629B2 (en) * | 2015-10-09 | 2019-07-02 | Samsung Electronics Co., Ltd. | Method for providing indication in multi-dimensional media in electronic device |
US9652896B1 (en) | 2015-10-30 | 2017-05-16 | Snap Inc. | Image based tracking in augmented reality systems |
US9984499B1 (en) | 2015-11-30 | 2018-05-29 | Snap Inc. | Image and point cloud based tracking and in augmented reality systems |
US20170180652A1 (en) * | 2015-12-21 | 2017-06-22 | Jim S. Baca | Enhanced imaging |
US20170286383A1 (en) * | 2016-03-30 | 2017-10-05 | Microsoft Technology Licensing, Llc | Augmented imaging assistance for visual impairment |
US11032471B2 (en) * | 2016-06-30 | 2021-06-08 | Nokia Technologies Oy | Method and apparatus for providing a visual indication of a point of interest outside of a user's view |
US10339708B2 (en) * | 2016-11-01 | 2019-07-02 | Google Inc. | Map summarization and localization |
US10482327B1 (en) | 2017-01-18 | 2019-11-19 | Snap Inc. | Media overlay selection system |
US10319149B1 (en) | 2017-02-17 | 2019-06-11 | Snap Inc. | Augmented reality anamorphosis system |
US10074381B1 (en) * | 2017-02-20 | 2018-09-11 | Snap Inc. | Augmented reality speech balloon system |
KR102330829B1 (en) | 2017-03-27 | 2021-11-24 | 삼성전자주식회사 | Method and apparatus for providing augmented reality function in electornic device |
US10387730B1 (en) | 2017-04-20 | 2019-08-20 | Snap Inc. | Augmented reality typography personalization system |
US11410330B2 (en) * | 2017-05-30 | 2022-08-09 | Edx Technologies, Inc. | Methods, devices, and systems for determining field of view and producing augmented reality |
US10530997B2 (en) | 2017-07-13 | 2020-01-07 | Zillow Group, Inc. | Connecting and using building interior data acquired from mobile devices |
US10375306B2 (en) | 2017-07-13 | 2019-08-06 | Zillow Group, Inc. | Capture and use of building interior data from mobile devices |
US10740974B1 (en) | 2017-09-15 | 2020-08-11 | Snap Inc. | Augmented reality system |
US10231085B1 (en) | 2017-09-30 | 2019-03-12 | Oracle International Corporation | Scaling out moving objects for geo-fence proximity determination |
US11417246B2 (en) | 2017-10-19 | 2022-08-16 | The Quantum Group, Inc. | Personal augmented reality |
US10921881B2 (en) * | 2018-01-18 | 2021-02-16 | Valve Corporation | Position tracking system for head-mounted displays that includes sensor integrated circuits |
US11250050B2 (en) * | 2018-03-01 | 2022-02-15 | The Software Mackiev Company | System for multi-tagging images |
WO2019175789A1 (en) * | 2018-03-15 | 2019-09-19 | ГИОРГАДЗЕ, Анико Тенгизовна | Method for selecting a virtual advertising object to subsequently display to a user |
JP6748139B2 (en) * | 2018-04-02 | 2020-08-26 | ファナック株式会社 | Visual guidance device, visual guidance system, and visual guidance method |
US10643386B2 (en) | 2018-04-11 | 2020-05-05 | Zillow Group, Inc. | Presenting image transition sequences between viewing locations |
US10929494B2 (en) * | 2018-04-16 | 2021-02-23 | Stops.com Ltd. | Systems and methods for tagging objects for augmented reality |
US11527044B2 (en) * | 2018-06-27 | 2022-12-13 | Samsung Electronics Co., Ltd. | System and method for augmented reality |
CN110858895B (en) * | 2018-08-22 | 2023-01-24 | 虹软科技股份有限公司 | Image processing method and device |
US10997760B2 (en) | 2018-08-31 | 2021-05-04 | Snap Inc. | Augmented reality anthropomorphization system |
CA3058602C (en) | 2018-10-11 | 2023-01-24 | Zillow Group, Inc. | Automated mapping information generation from inter-connected images |
US10809066B2 (en) | 2018-10-11 | 2020-10-20 | Zillow Group, Inc. | Automated mapping information generation from inter-connected images |
US10708507B1 (en) | 2018-10-11 | 2020-07-07 | Zillow Group, Inc. | Automated control of image acquisition via use of acquisition device sensors |
US10972806B1 (en) * | 2018-11-30 | 2021-04-06 | Snap Inc. | Generating customized graphics based on location information |
US11122099B2 (en) * | 2018-11-30 | 2021-09-14 | Motorola Solutions, Inc. | Device, system and method for providing audio summarization data from video |
US10902685B2 (en) | 2018-12-13 | 2021-01-26 | John T. Daly | Augmented reality remote authoring and social media platform and system |
US11972529B2 (en) | 2019-02-01 | 2024-04-30 | Snap Inc. | Augmented reality system |
US11693126B2 (en) * | 2019-04-10 | 2023-07-04 | Trimble Inc. | Augmented reality distance measurement |
US11232623B2 (en) * | 2019-06-28 | 2022-01-25 | Unity IPR ApS | Method and system for creating a neural net based lossy renderer |
US12014120B2 (en) | 2019-08-28 | 2024-06-18 | MFTB Holdco, Inc. | Automated tools for generating mapping information for buildings |
US11243656B2 (en) | 2019-08-28 | 2022-02-08 | Zillow, Inc. | Automated tools for generating mapping information for buildings |
US11164368B2 (en) | 2019-10-07 | 2021-11-02 | Zillow, Inc. | Providing simulated lighting information for three-dimensional building models |
US11164361B2 (en) | 2019-10-28 | 2021-11-02 | Zillow, Inc. | Generating floor maps for buildings from automated analysis of visual data of the buildings' interiors |
US10825247B1 (en) | 2019-11-12 | 2020-11-03 | Zillow Group, Inc. | Presenting integrated building information using three-dimensional building models |
US11676344B2 (en) | 2019-11-12 | 2023-06-13 | MFTB Holdco, Inc. | Presenting building information using building models |
FR3104290B1 (en) * | 2019-12-05 | 2022-01-07 | Airbus Defence & Space Sas | SIMULATION BINOCULARS, AND SIMULATION SYSTEM AND METHODS |
US11816757B1 (en) * | 2019-12-11 | 2023-11-14 | Meta Platforms Technologies, Llc | Device-side capture of data representative of an artificial reality environment |
US11405549B2 (en) | 2020-06-05 | 2022-08-02 | Zillow, Inc. | Automated generation on mobile devices of panorama images for building locations and subsequent use |
US11756081B2 (en) * | 2020-06-12 | 2023-09-12 | International Business Machines Corporation | Rendering privacy aware advertisements in mixed reality space |
US11514674B2 (en) | 2020-09-04 | 2022-11-29 | Zillow, Inc. | Automated analysis of image contents to determine the acquisition location of the image |
US11592969B2 (en) | 2020-10-13 | 2023-02-28 | MFTB Holdco, Inc. | Automated tools for generating building mapping information |
US11481925B1 (en) | 2020-11-23 | 2022-10-25 | Zillow, Inc. | Automated determination of image acquisition locations in building interiors using determined room shapes |
CA3142154A1 (en) | 2021-01-08 | 2022-07-08 | Zillow, Inc. | Automated determination of image acquisition locations in building interiors using multiple data capture devices |
US11252329B1 (en) | 2021-01-08 | 2022-02-15 | Zillow, Inc. | Automated determination of image acquisition locations in building interiors using multiple data capture devices |
US11836973B2 (en) | 2021-02-25 | 2023-12-05 | MFTB Holdco, Inc. | Automated direction of capturing in-room information for use in usability assessment of buildings |
US11790648B2 (en) | 2021-02-25 | 2023-10-17 | MFTB Holdco, Inc. | Automated usability assessment of buildings using visual data of captured in-room images |
US11501492B1 (en) | 2021-07-27 | 2022-11-15 | Zillow, Inc. | Automated room shape determination using visual data of multiple captured in-room images |
US11842464B2 (en) | 2021-09-22 | 2023-12-12 | MFTB Holdco, Inc. | Automated exchange and use of attribute information between building images of multiple types |
CN114587650B (en) * | 2022-02-06 | 2024-06-11 | 上海诠视传感技术有限公司 | Tooth root canal orifice treatment auxiliary navigation method and system based on mixed reality technology |
CN114587657B (en) * | 2022-02-06 | 2024-05-31 | 上海诠视传感技术有限公司 | Auxiliary navigation method and system for oral implantation based on mixed reality technology |
US20230400329A1 (en) * | 2022-06-13 | 2023-12-14 | Inmapz Inc. | Marking location on a digital interactive map |
US11830135B1 (en) | 2022-07-13 | 2023-11-28 | MFTB Holdco, Inc. | Automated building identification using floor plans and acquired building images |
KR102669969B1 (en) * | 2022-08-22 | 2024-05-29 | 주식회사 브이알크루 | Apparatus and method for supporting tactical training using visual localization |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5495576A (en) * | 1993-01-11 | 1996-02-27 | Ritchey; Kurtis J. | Panoramic image based virtual reality/telepresence audio-visual system and method |
US20060077255A1 (en) * | 2004-08-10 | 2006-04-13 | Hui Cheng | Method and system for performing adaptive image acquisition |
US20090097710A1 (en) * | 2006-05-22 | 2009-04-16 | Rafael Advanced Defense Systems Ltd. | Methods and system for communication and displaying points-of-interest |
US20090321636A1 (en) * | 2008-06-26 | 2009-12-31 | Lynntech, Inc. | Method of searching for a thermal target |
Family Cites Families (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
FR2683330B1 (en) * | 1991-10-31 | 1994-11-25 | Thomson Csf | COMPUTER BINOCULAR. |
US8174568B2 (en) * | 2006-12-01 | 2012-05-08 | Sri International | Unified framework for precise vision-aided navigation |
US8660355B2 (en) * | 2010-03-19 | 2014-02-25 | Digimarc Corporation | Methods and systems for determining image processing operations relevant to particular imagery |
US8660369B2 (en) * | 2010-10-25 | 2014-02-25 | Disney Enterprises, Inc. | Systems and methods using mobile devices for augmented reality |
US20120201472A1 (en) * | 2011-02-08 | 2012-08-09 | Autonomy Corporation Ltd | System for the tagging and augmentation of geographically-specific locations using a visual data stream |
WO2013023706A1 (en) * | 2011-08-18 | 2013-02-21 | Layar B.V. | Computer-vision based augmented reality system |
EP2817785B1 (en) * | 2012-02-23 | 2019-05-15 | Charles D. Huston | System and method for creating an environment and for sharing a location based experience in an environment |
-
2013
- 2013-06-13 US US13/916,702 patent/US9495783B1/en active Active
-
2016
- 2016-10-05 US US15/286,161 patent/US20170024904A1/en not_active Abandoned
-
2021
- 2021-01-25 US US17/157,065 patent/US11423586B2/en active Active
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5495576A (en) * | 1993-01-11 | 1996-02-27 | Ritchey; Kurtis J. | Panoramic image based virtual reality/telepresence audio-visual system and method |
US20060077255A1 (en) * | 2004-08-10 | 2006-04-13 | Hui Cheng | Method and system for performing adaptive image acquisition |
US20090097710A1 (en) * | 2006-05-22 | 2009-04-16 | Rafael Advanced Defense Systems Ltd. | Methods and system for communication and displaying points-of-interest |
US20090321636A1 (en) * | 2008-06-26 | 2009-12-31 | Lynntech, Inc. | Method of searching for a thermal target |
Cited By (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9875403B2 (en) * | 2012-12-12 | 2018-01-23 | Thales | Method for accurately geolocating an image sensor installed on board an aircraft |
WO2018151382A1 (en) * | 2017-02-16 | 2018-08-23 | (주)원우이엔지 | Telescope including camera function, and image display method using same |
CN110220493A (en) * | 2018-03-02 | 2019-09-10 | 北京图森未来科技有限公司 | A kind of binocular distance measuring method and its device |
US11274922B2 (en) | 2018-03-02 | 2022-03-15 | Beijing Tusen Zhitu Technology Co., Ltd. | Method and apparatus for binocular ranging |
US12031842B2 (en) | 2018-03-02 | 2024-07-09 | Beijing Tusen Zhitu Technology Co., Ltd. | Method and apparatus for binocular ranging |
Also Published As
Publication number | Publication date |
---|---|
US11423586B2 (en) | 2022-08-23 |
US20210142530A1 (en) | 2021-05-13 |
US9495783B1 (en) | 2016-11-15 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11423586B2 (en) | Augmented reality vision system for tracking and geolocating objects of interest | |
US11226175B2 (en) | Devices with network-connected scopes for allowing a target to be simultaneously tracked by multiple devices | |
US11789523B2 (en) | Electronic device displays an image of an obstructed target | |
US10366511B2 (en) | Method and system for image georegistration | |
US10354407B2 (en) | Camera for locating hidden objects | |
US10896327B1 (en) | Device with a camera for locating hidden object | |
US10533826B2 (en) | Vehicle-mounted device with network-connected scopes for allowing a target to be simultaneously tracked by multiple other devices | |
CN112525185B (en) | AR navigation method based on positioning and AR head-mounted display device | |
WO2017160381A1 (en) | System for georeferenced, geo-oriented real time video streams |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STPP | Information on status: patent application and granting procedure in general |
Free format text: TC RETURN OF APPEAL |
|
STCV | Information on status: appeal procedure |
Free format text: EXAMINER'S ANSWER TO APPEAL BRIEF MAILED |
|
STCV | Information on status: appeal procedure |
Free format text: ON APPEAL -- AWAITING DECISION BY THE BOARD OF APPEALS |
|
STCV | Information on status: appeal procedure |
Free format text: BOARD OF APPEALS DECISION RENDERED |
|
STCB | Information on status: application discontinuation |
Free format text: ABANDONED -- AFTER EXAMINER'S ANSWER OR BOARD OF APPEALS DECISION |