US11977676B2 - Adjusting content of a head mounted display - Google Patents
Adjusting content of a head mounted display Download PDFInfo
- Publication number
- US11977676B2 US11977676B2 US17/369,314 US202117369314A US11977676B2 US 11977676 B2 US11977676 B2 US 11977676B2 US 202117369314 A US202117369314 A US 202117369314A US 11977676 B2 US11977676 B2 US 11977676B2
- Authority
- US
- United States
- Prior art keywords
- user
- view
- field
- virtual content
- content
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 claims abstract description 44
- 238000003860 storage Methods 0.000 claims description 25
- 230000004424 eye movement Effects 0.000 claims description 21
- 210000003128 head Anatomy 0.000 claims description 20
- 230000004044 response Effects 0.000 claims description 11
- 230000008859 change Effects 0.000 claims description 4
- 238000010586 diagram Methods 0.000 description 30
- 230000033001 locomotion Effects 0.000 description 15
- 230000006870 function Effects 0.000 description 11
- 230000003190 augmentative effect Effects 0.000 description 7
- 238000004891 communication Methods 0.000 description 6
- 239000004065 semiconductor Substances 0.000 description 6
- 230000003287 optical effect Effects 0.000 description 5
- 230000008569 process Effects 0.000 description 5
- 230000009471 action Effects 0.000 description 3
- 238000004422 calculation algorithm Methods 0.000 description 2
- 230000001413 cellular effect Effects 0.000 description 2
- 210000005069 ears Anatomy 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 239000004973 liquid crystal related substance Substances 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 229910052710 silicon Inorganic materials 0.000 description 2
- 239000010703 silicon Substances 0.000 description 2
- OKTJSMMVPCPJKN-UHFFFAOYSA-N Carbon Chemical compound [C] OKTJSMMVPCPJKN-UHFFFAOYSA-N 0.000 description 1
- VYPSYNLAJGMNEJ-UHFFFAOYSA-N Silicium dioxide Chemical compound O=[Si]=O VYPSYNLAJGMNEJ-UHFFFAOYSA-N 0.000 description 1
- XUIMIQQOPSSXEZ-UHFFFAOYSA-N Silicon Chemical compound [Si] XUIMIQQOPSSXEZ-UHFFFAOYSA-N 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 239000011521 glass Substances 0.000 description 1
- 229910021389 graphene Inorganic materials 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 230000007774 longterm Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000001465 metallisation Methods 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 239000002159 nanocrystal Substances 0.000 description 1
- 230000006855 networking Effects 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 229910052814 silicon oxide Inorganic materials 0.000 description 1
- 238000004088 simulation Methods 0.000 description 1
- 238000010200 validation analysis Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/013—Eye tracking input arrangements
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/0093—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/017—Head mounted
- G02B27/0172—Head mounted characterised by optical features
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/0179—Display position adjusting means not related to the information to be displayed
- G02B2027/0187—Display position adjusting means not related to the information to be displayed slaved to motion of at least a part of the body of the user, e.g. head, eye
Definitions
- the subject matter disclosed herein relates to head mounted display units and more particularly relates to adjusting content presented on a display of the head mounted display unit.
- Head mounted displays or headsets may include a display that can be used for viewing augmented or virtual reality environments.
- these systems operate based on registering the wearer's eyes in a known location relative to the display. An incorrect or erroneous registration, however, may impact which content is visible to the user.
- An apparatus in one embodiment, includes a processor and a memory that stores code executable by the processor to determine a field of view for a user relative to a display of a head-mounted display unit, detect that the user is attempting to look at content that is presented on the display of the head-mounted display unit but is out of the user's field of view, and adjust the content that is out of the user's field of view to make the content visible to the user.
- a method for adjusting content of a head mounted display includes determining, by a processor, a field of view for a user relative to a display of a head-mounted display unit, detecting that the user is attempting to look at content that is presented on the display of the head-mounted display unit but is out of the user's field of view, and adjusting the content that is out of the user's field of view to make the content visible to the user.
- a program product for adjusting content of a head mounted display includes computer readable storage medium that stores code executable by a processor.
- the code is executable by the processor to determine a field of view for a user relative to a display of a head-mounted display unit, detect that the user is attempting to look at content that is presented on the display of the head-mounted display unit but is out of the user's field of view, and adjust the content that is out of the user's field of view to make the content visible to the user.
- FIG. 1 is a schematic block diagram illustrating one embodiment of a system for adjusting content of a head mounted display
- FIG. 2 is a schematic block diagram illustrating one embodiment of an apparatus for adjusting content of a head mounted display
- FIG. 3 is a schematic block diagram illustrating one embodiment of another apparatus for adjusting content of a head mounted display
- FIG. 4 is an illustration of an example embodiment of system for adjusting content of a head mounted display
- FIG. 5 is a schematic flow chart diagram illustrating one embodiment of a method for adjusting content of a head mounted display
- FIG. 6 is a schematic flow chart diagram illustrating one embodiment of another method for adjusting content of a head mounted display.
- embodiments may be embodied as a system, method, or program product. Accordingly, embodiments may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a “circuit,” “module” or “system.” Furthermore, embodiments may take the form of a program product embodied in one or more computer readable storage devices storing machine readable code, computer readable code, and/or program code, referred hereafter as code. The storage devices may be tangible, non-transitory, and/or non-transmission. The storage devices may not embody signals. In a certain embodiment, the storage devices only employ signals for accessing code.
- modules may be implemented as a hardware circuit comprising custom very large scale integrated (“VLSI”) circuits or gate arrays, off-the-shelf semiconductors such as logic chips, transistors, or other discrete components.
- VLSI very large scale integrated
- a module may also be implemented in programmable hardware devices such as a field programmable gate array (“FPGA”), programmable array logic, programmable logic devices or the like.
- FPGA field programmable gate array
- Modules may also be implemented in code and/or software for execution by various types of processors.
- An identified module of code may, for instance, comprise one or more physical or logical blocks of executable code which may, for instance, be organized as an object, procedure, or function. Nevertheless, the executables of an identified module need not be physically located together but may comprise disparate instructions stored in different locations which, when joined logically together, comprise the module and achieve the stated purpose for the module.
- a module of code may be a single instruction, or many instructions, and may even be distributed over several different code segments, among different programs, and across several memory devices.
- operational data may be identified and illustrated herein within modules and may be embodied in any suitable form and organized within any suitable type of data structure. The operational data may be collected as a single data set or may be distributed over different locations including over different computer readable storage devices.
- the software portions are stored on one or more computer readable storage devices.
- the computer readable medium may be a computer readable storage medium.
- the computer readable storage medium may be a storage device storing the code.
- the storage device may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, holographic, micromechanical, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing.
- a storage device More specific examples (a non-exhaustive list) of the storage device would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
- a computer readable storage medium may be any tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device.
- Code for carrying out operations for embodiments may be written in any combination of one or more programming languages including an object oriented programming language such as Python, Ruby, R, Java, Java Script, Smalltalk, C++, C sharp, Lisp, Clojure, PHP, or the like, and conventional procedural programming languages, such as the “C” programming language, or the like, and/or machine languages such as assembly languages.
- the code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server.
- the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider).
- LAN local area network
- WAN wide area network
- Internet Service Provider an Internet Service Provider
- the embodiments may transmit data between electronic devices.
- the embodiments may further convert the data from a first format to a second format, including converting the data from a non-standard format to a standard format and/or converting the data from the standard format to a non-standard format.
- the embodiments may modify, update, and/or process the data.
- the embodiments may store the received, converted, modified, updated, and/or processed data.
- the embodiments may provide remote access to the data including the updated data.
- the embodiments may make the data and/or updated data available in real time.
- the embodiments may generate and transmit a message based on the data and/or updated data in real time.
- the embodiments may securely communicate encrypted data.
- the embodiments may organize data for efficient validation. In addition, the embodiments may validate the data in response to an action and/or a lack of an action.
- the code may also be stored in a storage device that can direct a computer, other programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions stored in the storage device produce an article of manufacture including instructions which implement the function/act specified in the schematic flowchart diagrams and/or schematic block diagrams block or blocks.
- the code may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other devices to produce a computer implemented process such that the code which execute on the computer or other programmable apparatus provide processes for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.
- each block in the schematic flowchart diagrams and/or schematic block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions of the code for implementing the specified logical function(s).
- An apparatus in one embodiment, includes a processor and a memory that stores code executable by the processor to determine a field of view for a user relative to a display of a head-mounted display unit, detect that the user is attempting to look at content that is presented on the display of the head-mounted display unit but is out of the user's field of view, and adjust the content that is out of the user's field of view to make the content visible to the user.
- the code is executable by the processor to detect that the user is attempting to look at content that is not within the user's field of view by detecting eye movement within an eye-box defined for the head-mounted display unit directed towards the content.
- the code is executable by the processor to detect the eye movement within the eye-box based on input from a camera mounted on the head-mounted display and pointed towards the user's eyes. In various embodiments, the code is executable by the processor to detect the eye movement within the eye-box based on input from a distance sensor mounted on the head-mounted display and pointed towards the user's eyes.
- the code is executable by the processor to detect the eye movement within the eye-box in response to detecting that the user is manually adjusting a position of the head-mounted display unit on the user's head. In some embodiments, the code is executable by the processor to detect that the user is attempting to look at content that is not within the user's field of view by detecting that the user is looking in a direction of the content and detecting that the user is moving the head-mounted display unit in a same direction.
- adjusting the content comprises moving the content into the user's field of view on the display. In some embodiments, adjusting the content comprises resizing the content to be at least one of larger to make the content visible within the user's field of view and smaller to avoid the content overlapping with other content presented on the display.
- adjusting the content comprises adjusting a viewport for the display of the head-mounted display unit until the content that is out of the user's field of view is within the user's field of view.
- the content that is out of the user's field of view comprises tag-along content that is intended to remain within the user's field of view.
- the code is further executable by the processor to trigger re-calibrating the user's field of view relative to the display of the head-mounted display unit in response to detecting that the user is attempting to look at the content that is out of the user's field of view.
- a method for adjusting content of a head mounted display includes determining, by a processor, a field of view for a user relative to a display of a head-mounted display unit, detecting that the user is attempting to look at content that is presented on the display of the head-mounted display unit but is out of the user's field of view, and adjusting the content that is out of the user's field of view to make the content visible to the user.
- detecting that the user is attempting to look at content that is not within the user's field of view comprises detecting eye movement within an eye-box defined for the head-mounted display unit directed towards the content.
- the method includes detecting the eye movement within the eye-box based on input from a camera mounted on the head-mounted display and pointed towards the user's eyes.
- the method includes detecting the eye movement within the eye-box based on input from a distance sensor mounted on the head-mounted display and pointed towards the user's eyes. In one embodiment, the method includes detecting the eye movement within the eye-box in response to detecting that the user is manually adjusting a position of the head-mounted display unit on the user's head.
- adjusting the content comprises moving the content into the user's field of view on the display. In some embodiments, adjusting the content comprises resizing the content to be at least one of larger to make the content visible within the user's field of view and smaller to avoid the content overlapping with other content presented on the display. In various embodiments, adjusting the content comprises adjusting a viewport for the display of the head-mounted display unit until the content that is out of the user's field of view is within the user's field of view.
- a program product for adjusting content of a head mounted display includes computer readable storage medium that stores code executable by a processor.
- the code is executable by the processor to determine a field of view for a user relative to a display of a head-mounted display unit, detect that the user is attempting to look at content that is presented on the display of the head-mounted display unit but is out of the user's field of view, and adjust the content that is out of the user's field of view to make the content visible to the user.
- Head mounted display units and/or headsets may be used for presenting augmented reality and/or virtual reality environments and may operate based on having the wearer's eyes in a known or registered location relative to the display screen.
- the eye-box is the term used to describe a rectangle in space where the display of the headset is visible. Different optical systems can have larger or smaller eye-boxes.
- Registration of the headset to the user's eye might be done through physical adjustments such as knobs, or software mechanisms similar to keystone settings on a projector. Sometimes the registration is not perfect, exact, or accurate (e.g., perhaps the user rushed through or skipped the registration), so part of the display is not visible when the eye is in one position. Additionally, head mounted display units are known to be heavy and lack a well-balanced weight distribution, which may result in the head mounted display unit sliding down the user's face or the user constantly moving the head mounted display unit around to reduce pressure on the nose, ears, back of head, or the like. In these cases, the user may need to move their eye within the eye-box to see certain parts of the display. This kind of motion may be accomplished by moving, holding, readjusting, or the like the head mounted display unit on the user's head.
- augmented reality and virtual reality content There are typically two types of augmented reality and virtual reality content.
- One type sometimes called “tag-along” content, stays in the display all the time, perhaps at a fixed pixel position on the display, or perhaps animated with a kind of rubber-banding effect but still remaining in the user's view.
- the other type sometimes called “pinned” content, is fixed to a position in a three-dimensional space (either the real world or a virtual world), and the head mounted display unit tracking system may be used to determine whether and where to display this content on the display.
- Pinned content does not suffer from the eye-box problem described above—the wearer can just move their head a little bit more to see the content.
- Tag-along content on the other hand, remains in the same place on the display even as the user looks around. Thus, to make the tag-along content visible, the user needs to move the position of their eye in the eye-box.
- FIG. 1 is a schematic block diagram illustrating one embodiment of a system for adjusting content of a head mounted display.
- the system 100 includes a head mounted display (“HMD”) unit 101 .
- HMD head mounted display
- an HMD unit 101 may refer to a display device that is worn on a user's head and has a display positioned in front of the user's eyes.
- a typical HMD unit 101 has one or two displays 102 , with lenses and semi-transparent mirrors embedded in eyeglasses (also termed data glasses), a visor, or a helmet.
- the displays 102 may be miniaturized and may include cathode ray tubes (“CRT”), liquid crystal displays (“LCDs”), liquid crystal on silicon (“LCos”), or organic light-emitting diodes (“OLED”).
- CTR cathode ray tubes
- LCDs liquid crystal displays
- LCos liquid crystal on silicon
- OLED organic light-emitting diodes
- the HMD unit 101 may be configured to mount, hold, or otherwise couple to a smart phone, which becomes the computing device and display for the HMD unit 101 .
- the smart phone, or other mobile device may be operably coupled to the HMD unit 101 to enable a user to interact with the smart phone, receive data from the HMD unit 101 , turn on/off using controls on the HMD unit 101 , and/or the like.
- the display and computing device is integrated into the HMD unit 101 to form a single unit.
- the computing device may comprise various processors or processor cores, memory, storage, network connectivity chips, graphics chips, audio chips, and/or the like.
- the HMD unit 101 may be configured to present a virtual reality or augmented reality environment.
- a virtual reality (“VR”) environment is an interactive, virtual, digital, or otherwise computer generated three-dimensional environment or simulation.
- An augmented reality (“AR”) environment may be considered a form of virtual reality that layers virtual information over a camera feed into an HMD unit 101 or through a smartphone or tablet or other device coupled to the HMD unit 101 giving the user the ability to view three-dimensional images.
- the HMD unit 102 is coupled to a headband 104 that is configured to go over/around the user's head and hold the HMD unit 102 against the user's eyes/face.
- the headband 104 may encompass the user's head using different bands.
- a single headband 104 that goes around the user's head may be used, or other bands that go over the user's head may be used to add additional support for the HMD unit 102 .
- Other embodiments may include arms, similar to arms for eyeglasses, that sit on the user's ears, a chin strap for securing the HMD unit 102 to the user's head, a helmet that totally encloses the user's head, a visor, and/or the like.
- the system 100 includes a content adjustment apparatus 104 .
- the content adjustment apparatus 104 is configured to dynamically adjust content presented on the display 102 of the HMD unit 101 that may be out of the user's field of view, e.g., out of a defined “eyebox” for the user.
- the content adjustment apparatus 104 determines a field of view for a user relative to a display of a head-mounted display unit, detects that the user is attempting to look at content that is presented on the display of the head-mounted display unit but is out of the user's field of view, and adjusts the content that is out of the user's field of view to make the content visible to the user.
- the content adjustment apparatus 104 is described in more detail below with reference to FIGS. 2 and 3 .
- the content adjustment apparatus 104 may include a hardware device such as a secure hardware dongle or other hardware appliance device (e.g., a set-top box, a network appliance, or the like) that attaches to a device such as the HMD unit 101 , a laptop computer, a server 108 , a tablet computer, a smart phone, a security system, a network router or switch, or the like, either by a wired connection (e.g., a universal serial bus (“USB”) connection or other communication bus that is physically attached to the device) or a wireless connection (e.g., Bluetooth®, Wi-Fi, WAN, or the like); that attaches to an electronic display device (e.g., a television or monitor using an HDMI port, a DisplayPort port, a Mini DisplayPort port, VGA port, DVI port, or the like); and/or the like.
- a hardware device such as a secure hardware dongle or other hardware appliance device (e.g., a set-top box, a network appliance
- a hardware appliance of the content adjustment apparatus 104 may include a power interface, a wired and/or wireless network interface, a graphical interface that attaches to a display, and/or a semiconductor integrated circuit device as described below, configured to perform the functions described herein with regard to the content adjustment apparatus 104 .
- the content adjustment apparatus 104 may include a semiconductor integrated circuit device (e.g., one or more chips, die, or other discrete logic hardware), or the like, such as a field-programmable gate array (“FPGA”) or other programmable logic, firmware for an FPGA or other programmable logic, microcode for execution on a microcontroller, an application-specific integrated circuit (“ASIC”), a processor, a processor core, or the like.
- FPGA field-programmable gate array
- ASIC application-specific integrated circuit
- the content adjustment apparatus 104 may be mounted on a printed circuit board with one or more electrical lines or connections (e.g., to volatile memory, a non-volatile storage medium, a network interface, a peripheral device, a graphical/display interface, or the like).
- the hardware appliance may include one or more pins, pads, or other electrical connections configured to send and receive data (e.g., in communication with one or more electrical lines of a printed circuit board or the like), and one or more hardware circuits and/or other electrical circuits configured to perform various functions of the content adjustment apparatus 104 .
- the semiconductor integrated circuit device or other hardware appliance of the content adjustment apparatus 104 includes and/or is communicatively coupled to one or more volatile memory media, which may include but is not limited to random access memory (“RAM”), dynamic RAM (“DRAM”), cache, or the like.
- volatile memory media may include but is not limited to random access memory (“RAM”), dynamic RAM (“DRAM”), cache, or the like.
- the semiconductor integrated circuit device or other hardware appliance of the content adjustment apparatus 104 includes and/or is communicatively coupled to one or more non-volatile memory media, which may include but is not limited to: NAND flash memory, NOR flash memory, nano random access memory (nano RAM or “NRAM”), nanocrystal wire-based memory, silicon-oxide based sub-10 nanometer process memory, graphene memory, Silicon-Oxide-Nitride-Oxide-Silicon (“SONOS”), resistive RAM (“RRAM”), programmable metallization cell (“PMC”), conductive-bridging RAM (“CBRAM”), magneto-resistive RAM (“MRAM”), dynamic RAM (“DRAM”), phase change RAM (“PRAM” or “PCM”), magnetic storage media (e.g., hard disk, tape), optical storage media, or the like.
- non-volatile memory media which may include but is not limited to: NAND flash memory, NOR flash memory, nano random access memory (nano RAM or “NRAM”), nano
- the data network 106 includes a digital communication network that transmits digital communications.
- the data network 106 may include a wireless network, such as a wireless cellular network, a local wireless network, such as a Wi-Fi network, a Bluetooth® network, a near-field communication (“NFC”) network, an ad hoc network, and/or the like.
- the data network 106 may include a wide area network (“WAN”), a storage area network (“SAN”), a local area network (“LAN”) (e.g., a home network), an optical fiber network, the internet, or other digital communication network.
- the data network 106 may include two or more networks.
- the data network 106 may include one or more servers, routers, switches, and/or other networking equipment.
- the data network 106 may also include one or more computer readable storage media, such as a hard disk drive, an optical drive, non-volatile memory, RAM, or the like.
- the network may be configured as a 4G network, a 5G network, a long-term evolution (“LTE”) based network, and/or the like.
- the mobile telephone network may create and manage wireless connections between base access units for the network and user equipment (“UE”) devices for the user.
- the wireless connection may also employ a Wi-Fi network based on any one of the Institute of Electrical and Electronics Engineers (“IEEE”) 802.11 standards.
- IEEE Institute of Electrical and Electronics Engineers
- the one or more servers 108 may be embodied as blade servers, mainframe servers, tower servers, rack servers, and/or the like.
- the one or more servers 108 may be configured as mail servers, web servers, application servers, FTP servers, media servers, data servers, web servers, file servers, virtual servers, and/or the like.
- the servers 108 may store, transmit, and/or the like content to be presented on the HMD unit, such as AR or VR content.
- FIG. 2 is a schematic block diagram illustrating one embodiment of an apparatus 200 for adjusting content of a head mounted display.
- the apparatus 200 includes an instance of a content adjustment apparatus 104 .
- the content adjustment apparatus 104 includes one or more of a FOV module 202 , a gaze module 204 , and an adjustment module 206 , which are described in more detail below.
- the FOV module 202 is configured to determine a field of view for a user relative to a display 102 of a head-mounted display unit, such as the HMD unit 101 described above with reference to FIG. 1 .
- the user's field of view may comprise the area in front of the user's eyes that is visible to the user.
- the user's field of view comprises the area, portion, section, of the like of the display 102 that is visible to the user while the user wears the HMD unit 101 .
- the FOV module 202 may use various sensors such as distance sensors, proximity sensors, light sensors, and/or the like to determine a distance between the user's eyes and the display 102 of the HMD unit 101 to derive the horizontal and vertical field of view angles, dimensions, sizes, and/or the like, which may further be based on the size of the display 102 .
- content on the display 102 may be presented within a viewport, which is the viewable area of the display where content is presented.
- the user's field of view and the viewport may not correspond, however, e.g., due to incorrect calibration, incorrect fit of the HMD unit 101 on the user's head, and/or the like, which may inhibit or prevent the user from viewing all of the available content that is presented on the display.
- the subject matter herein provides a solution for matching or correcting the user's field of view and/or adjusting the content on the display so that the user can view content which may otherwise be at least partially blocked from the user's field of view.
- the gaze module 204 is configured to detect that the user is attempting to look at content that is presented on the display of the head-mounted display unit but is out of the user's field of view. As explained above, some content may not be completely within the user's field of view or may only be partially visible to the user. In such an embodiment, the user may strain or attempt to look at the content, e.g., looking hard right, left, up, and/or down to view content that is not wholly visible to the user.
- the content may include virtual reality or augmented reality content.
- the content may be in a fixed location with the virtual/augmented reality environment or may be tag-along content that changes location on the display 102 to follow the user as the user moves his head around within the virtual/augment reality environment.
- the gaze module 204 detects that the user is attempting to look at content that is not within the user's field of view by detecting eye movement within an eye box defined for the HMD unit 101 unit directed towards the content.
- an eye box may refer to a logical, calculated volume or area between the user's eyes and the display 102 within which users can place their eyes to see content presented on the display 102 .
- the gaze module 204 tracks the user's line-of-sight from the user's eyes to the display 102 .
- the gaze module 204 may use various eye tracking algorithms, image processing algorithms, and/or sensors, e.g., cameras, distance sensors, motion sensors, and/or the like that are mounted to the HMD unit 101 near the display 102 and pointed towards the user's eyes, or the like to detect when a user's moves their eyes, the direction that the user is looking, and a point or area on the display 102 where the user is looking.
- the gaze module 204 based on the user's determined eye gaze and the calculated eye box, detects that the user is attempting to look at content that is not within the user's field of view by detecting eye movement within an eye-box defined for the HMD unit 101 that is directed towards the content. For instance, if the user's eye gaze indicates that the user is attempting to look at content that is on the far left of the display 102 but may be beyond the boundaries of the eye box, the gaze module 204 may determine that the user is attempting to look at content that is not within the user's field of view.
- the gaze module 204 detects the eye movement within the eye-box in response to detecting that the user is manually adjusting a position of the HMD unit 101 on the user's head.
- the gaze module 204 and/or the movement module 302 may detect that the user is manually moving the HMD unit 101 , e.g., left-to-right, up-to-down, or the like, using various sensors in the HMD unit 101 , e.g., accelerometer, gyroscope, and/or the like.
- the gaze module 204 may track the user's eye gaze to determine whether the user's eye gaze is in the same direction as the HMD unit 101 movement. If so, then the gaze module 204 may determine that the user is trying to view content that is not visible to the user.
- the gaze module 204 detects that the user is attempting to look at content that is not within the user's field of view by detecting that the user is looking in a direction of the content and detecting that the user is moving the HMD unit 101 in a same direction. For instance, the gaze module 204 may detect that the user is looking left and is turning his head left, which moves the HMD unit 101 to the left in the same direction. In one embodiment, the gaze module 204 may determine whether the user's eye gaze meets a predefined threshold the defines how much the user is straining or working to look a certain direction.
- the gaze module 204 may determine that the user is trying to view content that is outside not wholly visible on the display 102 .
- the gaze module 204 and/or the movement module 302 may detect in which direction the HMD unit 101 is moving based on feedback from various sensors, e.g., accelerometers, gyroscopes, and/or the like.
- the adjustment module 206 is configured to identify content that is outside the user's field of view that the user is attempting to look at, e.g., tag-along content, and dynamically adjust various characteristics of the content, e.g., size, position, or the like, to make the content visible to the user.
- the adjustment module 206 adjusts the content by moving the content into the user's field of view on the display 102 .
- the adjustment module 206 may dynamically move the content that is not within the user's field of view to a different location on the display 102 that is within the user's field of view and that does not block other content of interest on the display, e.g., is not placed right in the user's direct line of sight.
- the adjustment module 206 adjusts the content by resizing the content to be either larger, to make the content visible within the user's field of view, or smaller, to avoid the content overlapping with other content presented on the display. For example, the adjustment module 206 may make a tag-along content element larger on the display 102 so that it comes into the user's field of view and is visible to the user. In some embodiments, if the content overlaps other content presented on the display 102 , the adjustment module 206 may make the content smaller so that the content does not overlap other content presented on the display 102 .
- the adjustment module 206 adjusts the content comprises by adjusting a viewport for the display of the HMD unit 101 until the content that is out of the user's field of view is within the user's field of view.
- the viewport may refer to the area of the display 102 that presents content, which may not match or correspond to the user's field of view.
- the user's field of view may be smaller or may be offset from the viewport so that the entire viewport, e.g., the entire portion of the display 102 that presents content, is not completely visible to the user.
- the adjustment module 206 modifies the dimensions of the viewport so that it matches or corresponds to the user's field of view.
- the adjustment module 206 may make the viewport smaller in both the horizontal and vertical directions, may only adjust the viewport in one dimension, and/or the like.
- the adjustment module 206 may similarly adjust a resolution of the display 102 , to make content larger or smaller, may adjust a brightness, contrast, or other display setting for the display 102 .
- FIG. 3 is a schematic block diagram illustrating one embodiment of another apparatus 300 for adjusting content of a head mounted display.
- the apparatus 300 includes an instance of a content adjustment apparatus 104 .
- the content adjustment apparatus 104 includes one or more of a FOV module 302 , a gaze module 304 , and an adjustment module 306 , which may be substantially similar to the FOV module 202 , the gaze module 204 , and the adjustment module 206 described above with reference to FIG. 2 .
- the content adjustment apparatus 104 includes a movement module 302 and a calibration module 304 , which are described in more detail below.
- the movement module 302 is configured to detect that the user is manually adjusting a position of the HMD unit 101 on the user's head. In such an embodiment, the movement module 302 may use various sensor data to determine movement of the HMD unit 101 . For instance, the movement module 302 may detect that the user is touching HMD unit 101 , e.g., the display portion 102 , the headband, and/or the like using various touch sensors. In certain embodiments, the movement module 302 may detect that the user is moving the HMD unit 101 using motion sensors, proximity sensors, accelerometers, gyroscopes, and/or the like that may be integrated into or built into the HMD unit 101 .
- the movement module 302 may determine to what degree the HMD unit 101 moves, how far it moves, in which direction it moves, and/or the like based on the sensor data, which may be used to determine whether the user is trying to look at content that is outside the user's field of view and to determine how to adjust the content to make it visible to the user.
- the calibration module 304 is configured to trigger re-calibrating the user's field of view relative to the display 102 of the HMD unit 101 in response to detecting that the user is attempting to look at content that is out of the user's field of view. For instance, the full display area may not be completely visible to the user because of an incorrect, incomplete, or erroneous calibration when the user first uses the HMD unit 101 , which may cause the user's field of view to not correspond or match up with the viewport of the display 102 .
- the calibration module 304 may trigger an automatic recalibration for the user, may prompt the user to confirm that the user would like to trigger a recalibration process, and/or the like.
- FIG. 4 depicts an example system for adjusting content of a head mounted display.
- content 401 is presented on a display 102 of an HMD unit 101 .
- Other content elements may be presented on the display 102 , including tag-along content 404 that “follows” the user, e.g., stays visible on the display 102 , while the user moves his head, and the HMD unit 102 .
- the FOV module 202 may determine the user's field of view 406 , based on the location of the user's eyes 408 relative to the display 102 . If the user's field of view 406 does not match or correspond to the viewport 402 of the display 102 , some content may not be visible to the user. In the depicted embodiment, the tag-along content 404 , e.g., a compass in this example, is not fully visible to the user.
- the adjustment module 206 may perform various actions, described above, to adjust the tag-along content to make it visible to the user, e.g., to make it visible within the user's field of view 406 .
- FIG. 5 is a schematic flow chart diagram illustrating one embodiment of a method 500 for adjusting content of a head mounted display.
- the method 500 begins and determines 502 a field of view for a user relative to a display of a head-mounted display unit.
- the method 500 detects 504 that the user is attempting to look at content that is presented on the display of the head-mounted display unit but is out of the user's field of view.
- the method 500 adjusts 506 the content that is out of the user's field of view to make the content visible to the user, and the method 500 ends.
- the FOV module 202 , the gaze module 204 , and the adjustment module 206 perform the various steps of the method 500 .
- FIG. 6 is a schematic flow chart diagram illustrating one embodiment of another method 600 for adjusting content of a head mounted display.
- the method 600 begins and determines 602 a field of view for a user relative to a display of a head-mounted display unit.
- the method 600 detects 604 that the user is attempting to look at content that is presented on the display of the head-mounted display unit but is out of the user's field of view.
- the method 600 may detect 604 a that the user is attempting to look at content that is not within the user's field of view by detecting eye movement within an eye-box defined for the head-mounted display unit directed towards the content.
- the method 600 may detect 604 b that the user is attempting to look at content that is not within the user's field of view by detecting that the user is looking in a direction of the content and detecting that the user is moving the head-mounted display unit in a same direction.
- the method 600 adjusts 606 the content that is out of the user's field of view to make the content visible to the user.
- the method 600 moves 606 a the content into the user's field of view on the display.
- the method 600 resizes 606 b the content to be at least one of larger to make the content visible within the user's field of view and smaller to avoid the content overlapping with other content presented on the display.
- the method 600 adjusts 606 c a viewport for the display of the head-mounted display unit until the content that is out of the user's field of view is within the user's field of view, and the method 600 ends.
- the FOV module 202 , the gaze module 204 , the adjustment module 206 , and the movement module 302 perform the various steps of the method 600 .
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Optics & Photonics (AREA)
- User Interface Of Digital Computer (AREA)
Abstract
Description
Claims (20)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US17/369,314 US11977676B2 (en) | 2021-07-07 | 2021-07-07 | Adjusting content of a head mounted display |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US17/369,314 US11977676B2 (en) | 2021-07-07 | 2021-07-07 | Adjusting content of a head mounted display |
Publications (2)
Publication Number | Publication Date |
---|---|
US20230008009A1 US20230008009A1 (en) | 2023-01-12 |
US11977676B2 true US11977676B2 (en) | 2024-05-07 |
Family
ID=84799346
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US17/369,314 Active US11977676B2 (en) | 2021-07-07 | 2021-07-07 | Adjusting content of a head mounted display |
Country Status (1)
Country | Link |
---|---|
US (1) | US11977676B2 (en) |
Families Citing this family (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
KR20230057494A (en) * | 2021-10-21 | 2023-05-02 | 삼성디스플레이 주식회사 | Device for providing augmented reality and system for augmented providing augmented reality using the same |
Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20140247286A1 (en) * | 2012-02-20 | 2014-09-04 | Google Inc. | Active Stabilization for Heads-Up Displays |
US20170315612A1 (en) * | 2016-04-27 | 2017-11-02 | Rovi Guides, Inc. | Methods and systems for displaying additional content on a heads up display displaying a virtual reality environment |
US20170354883A1 (en) * | 2016-06-13 | 2017-12-14 | Sony Interactive Entertainment LLC | Method and system for directing user attention to a location based game play companion application |
US20180196505A1 (en) * | 2016-11-30 | 2018-07-12 | Seiko Epson Corporation | Head-mounted display device, computer program, and control method for head-mounted display device |
US10845595B1 (en) * | 2017-12-28 | 2020-11-24 | Facebook Technologies, Llc | Display and manipulation of content items in head-mounted display |
US11269402B1 (en) * | 2018-08-03 | 2022-03-08 | Snap Inc. | User interface interaction paradigms for eyewear device with limited field of view |
-
2021
- 2021-07-07 US US17/369,314 patent/US11977676B2/en active Active
Patent Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20140247286A1 (en) * | 2012-02-20 | 2014-09-04 | Google Inc. | Active Stabilization for Heads-Up Displays |
US20170315612A1 (en) * | 2016-04-27 | 2017-11-02 | Rovi Guides, Inc. | Methods and systems for displaying additional content on a heads up display displaying a virtual reality environment |
US20170354883A1 (en) * | 2016-06-13 | 2017-12-14 | Sony Interactive Entertainment LLC | Method and system for directing user attention to a location based game play companion application |
US20180196505A1 (en) * | 2016-11-30 | 2018-07-12 | Seiko Epson Corporation | Head-mounted display device, computer program, and control method for head-mounted display device |
US10845595B1 (en) * | 2017-12-28 | 2020-11-24 | Facebook Technologies, Llc | Display and manipulation of content items in head-mounted display |
US11269402B1 (en) * | 2018-08-03 | 2022-03-08 | Snap Inc. | User interface interaction paradigms for eyewear device with limited field of view |
Also Published As
Publication number | Publication date |
---|---|
US20230008009A1 (en) | 2023-01-12 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US10591731B2 (en) | Ocular video stabilization | |
JP6619831B2 (en) | Method and apparatus for determining representation of display information based on focus distance | |
US20150170422A1 (en) | Information Display System With See-Through HMD, Display Control Program and Display Control Method | |
EP3552081B1 (en) | Display synchronized image warping | |
US8970495B1 (en) | Image stabilization for color-sequential displays | |
US20150097772A1 (en) | Gaze Signal Based on Physical Characteristics of the Eye | |
US11314323B2 (en) | Position tracking system for head-mounted displays that includes sensor integrated circuits | |
US20150062164A1 (en) | Head mounted display, method of controlling head mounted display, computer program, image display system, and information processing apparatus | |
US10078366B2 (en) | Head-mountable apparatus and system | |
BR112015031234B1 (en) | Method for rendering a virtual object on a head-mounted transparent display and display apparatus | |
JP2015525365A (en) | Method and apparatus for performing focus correction of display information | |
US10114604B2 (en) | Head-mounted display device, control method for head-mounted display device, and computer program | |
CN105319718A (en) | Wearable glasses and method of displaying image via wearable glasses | |
US9442292B1 (en) | Directional array sensing module | |
US9799144B2 (en) | Head mounted display, and control method for head mounted display | |
CN112655202B (en) | Reduced bandwidth stereoscopic distortion correction for fisheye lenses of head-mounted displays | |
WO2019033446A1 (en) | Virtual reality device and content adjusting method therefor | |
US11977676B2 (en) | Adjusting content of a head mounted display | |
US9265415B1 (en) | Input detection | |
KR102650572B1 (en) | Electronic apparatus and method for controlling thereof | |
US20230324988A1 (en) | Display calibration | |
JP2015064476A (en) | Image display device, and method of controlling image display device | |
US20170193701A1 (en) | Display device and method | |
WO2019085109A1 (en) | Image writing control method and apparatus, and electronic device | |
US11900845B2 (en) | System and method for optical calibration of a head-mounted display |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
FEPP | Fee payment procedure |
Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
AS | Assignment |
Owner name: LENOVO (UNITED STATES) INC., NORTH CAROLINA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:NICHOLSON, JOHN W;CROMER, DARYL C;LOCKER, HOWARD;AND OTHERS;SIGNING DATES FROM 20210702 TO 20210706;REEL/FRAME:057216/0833 |
|
AS | Assignment |
Owner name: LENOVO (SINGAPORE) PTE. LTD., SINGAPORE Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:LENOVO (UNITED STATES) INC.;REEL/FRAME:058132/0698 Effective date: 20211111 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: FINAL REJECTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE AFTER FINAL ACTION FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: ADVISORY ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS |
|
ZAAB | Notice of allowance mailed |
Free format text: ORIGINAL CODE: MN/=. |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT VERIFIED |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |