CN118317815A

CN118317815A - Apparatus, method, and graphical user interface for trace mitigation in a three-dimensional environment

Info

Publication number: CN118317815A
Application number: CN202280062757.6A
Authority: CN
Inventors: 李佳宝; K·E·鲍尔利; J·H·林; M·R·卢; J·K·米斯格; F·P·奥利里; C-L·李
Original assignee: Apple Inc
Current assignee: Apple Inc
Priority date: 2021-09-24
Filing date: 2022-09-21
Publication date: 2024-07-09

Abstract

In some embodiments, a computer system displays a representation of a user associated with an external computer system, and the representation includes a visual indication of a portion of the user's body, the visual indication generated based on indirect information of a state of the portion of the user's body. In some embodiments, a computer system displays one or more portions of a representation of a user in a different appearance based on whether the computer system receives information that the one or more portions of the representation are positioned in a predefined area of a physical environment in which the user is located.

Description

Apparatus, method, and graphical user interface for trace mitigation in a three-dimensional environment

Cross Reference to Related Applications

The present application claims priority from U.S. provisional application No. 63/248,036, entitled "DEVICES,METHODS,AND GRAPHICAL USER INTERFACES FOR TRACKING MITIGATION IN THREE-DIMENSIONAL ENVIRONMENTS", filed 24 at 9, 2021, and U.S. patent application No. 17/944,911, entitled "DEVICES,METHODS,AND GRAPHICAL USER INTERFACES FOR TRACKING MITIGATION IN THREE-DIMENSIONAL ENVIRONMENTS", filed 14 at 9, 2022, each of which is hereby incorporated by reference in its entirety.

Technical Field

The present disclosure relates generally to computer systems in communication with display generation components and optionally with external computer systems associated with a first user that provide a computer-generated experience, including, but not limited to, electronic devices that provide virtual reality and mixed reality experiences via a display.

Background

In recent years, the development of computer systems for augmented reality has increased significantly. An example augmented reality environment includes at least some virtual elements that replace or augment the physical world. Input devices (such as cameras, controllers, joysticks, touch-sensitive surfaces, and touch screen displays) for computer systems and other electronic computing devices are used to interact with the virtual/augmented reality environment. Exemplary virtual elements include virtual objects such as digital images, video, text, icons, and control elements (such as buttons and other graphics).

Disclosure of Invention

Some methods and interfaces for displaying representations of users in environments that include at least some virtual elements (e.g., applications, augmented reality environments, mixed reality environments, and virtual reality environments) are cumbersome, inefficient, and limited. For example, systems that track portions of a user in a physical environment and provide inadequate feedback based on an inadequate feedback display representation are complex, tedious, and error-prone, pose a significant cognitive burden on the user, and detract from the experience of the virtual/augmented reality environment. In addition, these methods take longer than necessary, wasting energy from the computer system. This latter consideration is particularly important in battery-powered devices.

Accordingly, there is a need for a computer system with improved methods and interfaces to provide a user with a computer-generated experience that displays a continuous and understandable representation of the user when the feedback received is insufficient, making interactions with the computer system more efficient and intuitive for the user. Such methods and interfaces optionally complement or replace conventional methods for providing an augmented reality experience to a user. Such methods and interfaces reduce the number, extent, and/or nature of inputs from a user by helping the user understand the association between the inputs provided and the response of the device to those inputs, thereby forming a more efficient human-machine interface.

The above-described drawbacks and other problems associated with user interfaces of computer systems are reduced or eliminated by the disclosed systems. In some embodiments, the computer system is a desktop computer with an associated display. In some embodiments, the computer system is a portable device (e.g., a notebook, tablet, or handheld device). In some embodiments, the computer system is a personal electronic device (e.g., a wearable electronic device such as a watch or a head-mounted device). In some embodiments, the computer system has a touch pad. In some embodiments, the computer system has one or more cameras. In some implementations, the computer system has a touch-sensitive display (also referred to as a "touch screen" or "touch screen display"). In some embodiments, the computer system has one or more eye tracking components. In some embodiments, the computer system has one or more hand tracking components. In some embodiments, the computer system has, in addition to the display generating component, one or more output devices including one or more haptic output generators and/or one or more audio output devices. In some embodiments, a computer system has a Graphical User Interface (GUI), one or more processors, memory and one or more modules, a program or set of instructions stored in the memory for performing a plurality of functions. In some embodiments, the user interacts with the GUI through contact and gestures of a stylus and/or finger on the touch-sensitive surface, movements of the user's eyes and hands in space relative to the GUI (and/or computer system) or the user's body (as captured by cameras and other motion sensors), and/or voice input (as captured by one or more audio input devices). In some embodiments, the functions performed by the interactions optionally include image editing, drawing, presentation, word processing, spreadsheet making, game playing, phone calls, video conferencing, email sending and receiving, instant messaging, test support, digital photography, digital video recording, web browsing, digital music playing, notes taking, and/or digital video playing. Executable instructions for performing these functions are optionally included in a transitory and/or non-transitory computer readable storage medium or other computer program product configured for execution by one or more processors.

There is a need for an electronic device with improved methods and interfaces to display a representation of a user upon receiving insufficient feedback regarding the user's status. Such methods and interfaces may supplement or replace conventional methods for communicating with other users in a three-dimensional environment. Such methods and interfaces reduce the amount, degree, and/or nature of input from a user and result in a more efficient human-machine interface. For battery-powered computing devices, such methods and interfaces conserve power and increase the time interval between battery charges.

According to some embodiments, a method is described. The method is performed at a computer system in communication with a display generation component and in communication with an external computer system associated with a first user. The method comprises the following steps: in response to receiving a request to display a representation of the first user in an augmented reality environment: displaying the representation of the first user in the augmented reality environment via the display generating component, wherein the representation of the first user comprises: a visual indication of a portion of the body of the first user in the augmented reality environment, wherein: the visual indication of the portion of the body of the first user has an appearance determined based at least in part on one or more objects in the augmented reality environment; and the visual indication of the portion of the body of the first user represents an estimated state of the portion of the body estimated based on indirect information regarding the state of the portion of the body when the computer system is unable to obtain direct information regarding the state of the portion of the body.

According to some embodiments, a non-transitory computer readable storage medium is described. The non-transitory computer readable storage medium stores one or more programs configured to be executed by one or more processors of a computer system in communication with the display generation component and with an external computer system associated with the first user, the one or more programs comprising instructions for: in response to receiving a request to display a representation of the first user in an augmented reality environment: displaying the representation of the first user in the augmented reality environment via the display generating component, wherein the representation of the first user comprises: a visual indication of a portion of the body of the first user in the augmented reality environment, wherein: the visual indication of the portion of the body of the first user has an appearance determined based at least in part on one or more objects in the augmented reality environment; and the visual indication of the portion of the body of the first user represents an estimated state of the portion of the body estimated based on indirect information regarding the state of the portion of the body when the computer system is unable to obtain direct information regarding the state of the portion of the body.

According to some embodiments, a transitory computer readable storage medium is described. The transitory computer readable storage medium stores one or more programs configured to be executed by one or more processors of a computer system in communication with the display generation component and with an external computer system associated with the first user, the one or more programs comprising instructions for: in response to receiving a request to display a representation of the first user in an augmented reality environment: displaying the representation of the first user in the augmented reality environment via the display generating component, wherein the representation of the first user comprises: a visual indication of a portion of the body of the first user in the augmented reality environment, wherein: the visual indication of the portion of the body of the first user has an appearance determined based at least in part on one or more objects in the augmented reality environment; and the visual indication of the portion of the body of the first user represents an estimated state of the portion of the body estimated based on indirect information regarding the state of the portion of the body when the computer system is unable to obtain direct information regarding the state of the portion of the body.

According to some embodiments, a computer system is described. The computer system communicates with the display generating component and with an external computer system associated with the first user. The computer system includes: one or more processors; and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs comprising instructions for: in response to receiving a request to display a representation of the first user in an augmented reality environment: displaying the representation of the first user in the augmented reality environment via the display generating component, wherein the representation of the first user comprises: a visual indication of a portion of the body of the first user in the augmented reality environment, wherein: the visual indication of the portion of the body of the first user has an appearance determined based at least in part on one or more objects in the augmented reality environment; and the visual indication of the portion of the body of the first user represents an estimated state of the portion of the body estimated based on indirect information regarding the state of the portion of the body when the computer system is unable to obtain direct information regarding the state of the portion of the body.

According to some embodiments, a computer system is described. The computer system communicates with the display generating component and with an external computer system associated with the first user. The computer system includes: in response to receiving a request to display a representation of the first user in an augmented reality environment: means for displaying the representation of the first user in the augmented reality environment via the display generating component, wherein the representation of the first user comprises: a visual indication of a portion of the body of the first user in the augmented reality environment, wherein: the visual indication of the portion of the body of the first user has an appearance determined based at least in part on one or more objects in the augmented reality environment; and the visual indication of the portion of the body of the first user represents an estimated state of the portion of the body estimated based on indirect information regarding the state of the portion of the body when the computer system is unable to obtain direct information regarding the state of the portion of the body.

According to some embodiments, a method is described. The method is performed at a computer system in communication with a display generation component and in communication with an external computer system associated with a first user. The method comprises the following steps: in response to receiving a request to display a representation of the first user in an augmented reality environment: displaying the representation of the first user in the augmented reality environment via the display generating component, wherein displaying the representation of the first user comprises: in accordance with a determination that a first portion of the first user's body is in a first region of a physical environment in which the first user is located, wherein the first region is defined relative to the first user's body, displaying, via the display generating component, a first visual indication of the first portion of the first user's body, wherein the first visual indication of the first portion of the first user's body comprises a first visual fidelity metric; and in accordance with a determination that the first portion of the body of the first user is in a second region of the physical environment, wherein the second region is separate from the first region, displaying, via the display generating component, a second visual indication of the first portion of the body of the first user, wherein the second visual indication of the first portion of the body of the first user comprises a second visual fidelity measure different from the first visual fidelity measure.

According to some embodiments, a non-transitory computer readable storage medium is described. The non-transitory computer readable storage medium stores one or more programs configured to be executed by one or more processors of a computer system in communication with the display generation component and with an external computer system associated with the first user, the one or more programs comprising instructions for: in response to receiving a request to display a representation of the first user in an augmented reality environment: displaying the representation of the first user in the augmented reality environment via the display generating component, wherein displaying the representation of the first user comprises: in accordance with a determination that a first portion of the first user's body is in a first region of a physical environment in which the first user is located, wherein the first region is defined relative to the first user's body, displaying, via the display generating component, a first visual indication of the first portion of the first user's body, wherein the first visual indication of the first portion of the first user's body comprises a first visual fidelity metric; and in accordance with a determination that the first portion of the body of the first user is in a second region of the physical environment, wherein the second region is separate from the first region, displaying, via the display generating component, a second visual indication of the first portion of the body of the first user, wherein the second visual indication of the first portion of the body of the first user comprises a second visual fidelity measure different from the first visual fidelity measure.

According to some embodiments, a transitory computer readable storage medium is described. The transitory computer readable storage medium stores one or more programs configured to be executed by one or more processors of a computer system in communication with the display generation component and with an external computer system associated with the first user, the one or more programs comprising instructions for: in response to receiving a request to display a representation of the first user in an augmented reality environment: displaying the representation of the first user in the augmented reality environment via the display generating component, wherein displaying the representation of the first user comprises: in accordance with a determination that a first portion of the first user's body is in a first region of a physical environment in which the first user is located, wherein the first region is defined relative to the first user's body, displaying, via the display generating component, a first visual indication of the first portion of the first user's body, wherein the first visual indication of the first portion of the first user's body comprises a first visual fidelity metric; and in accordance with a determination that the first portion of the body of the first user is in a second region of the physical environment, wherein the second region is separate from the first region, displaying, via the display generating component, a second visual indication of the first portion of the body of the first user, wherein the second visual indication of the first portion of the body of the first user comprises a second visual fidelity measure different from the first visual fidelity measure.

According to some embodiments, a computer system is described. The computer system communicates with the display generating component and with an external computer system associated with the first user. The computer system includes: one or more processors; and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs comprising instructions for: in response to receiving a request to display a representation of the first user in an augmented reality environment: displaying the representation of the first user in the augmented reality environment via the display generating component, wherein displaying the representation of the first user comprises: in accordance with a determination that a first portion of the first user's body is in a first region of a physical environment in which the first user is located, wherein the first region is defined relative to the first user's body, displaying, via the display generating component, a first visual indication of the first portion of the first user's body, wherein the first visual indication of the first portion of the first user's body comprises a first visual fidelity metric; and in accordance with a determination that the first portion of the body of the first user is in a second region of the physical environment, wherein the second region is separate from the first region, displaying, via the display generating component, a second visual indication of the first portion of the body of the first user, wherein the second visual indication of the first portion of the body of the first user comprises a second visual fidelity measure different from the first visual fidelity measure.

According to some embodiments, a computer system is described. The computer system communicates with the display generating component and with an external computer system associated with the first user. The computer system includes: in response to receiving a request to display a representation of the first user in an augmented reality environment: means for displaying the representation of the first user in the augmented reality environment via the display generating component, wherein displaying the representation of the first user comprises: in accordance with a determination that a first portion of the first user's body is in a first region of a physical environment in which the first user is located, wherein the first region is defined relative to the first user's body, displaying, via the display generating component, a first visual indication of the first portion of the first user's body, wherein the first visual indication of the first portion of the first user's body comprises a first visual fidelity metric; and in accordance with a determination that the first portion of the body of the first user is in a second region of the physical environment, wherein the second region is separate from the first region, displaying, via the display generating component, a second visual indication of the first portion of the body of the first user, wherein the second visual indication of the first portion of the body of the first user comprises a second visual fidelity measure different from the first visual fidelity measure.

It is noted that the various embodiments described above may be combined with any of the other embodiments described herein. The features and advantages described in this specification are not all-inclusive and, in particular, many additional features and advantages will be apparent to one of ordinary skill in the art in view of the drawings, specification, and claims. Furthermore, it should be noted that the language used in the specification has been principally selected for readability and instructional purposes, and may not have been selected to delineate or circumscribe the inventive subject matter.

Drawings

For a better understanding of the various described embodiments, reference should be made to the following detailed description taken in conjunction with the following drawings, in which like reference numerals designate corresponding parts throughout the several views.

FIG. 1 is a block diagram illustrating an operating environment for a computer system for providing an XR experience, according to some embodiments.

FIG. 2 is a block diagram illustrating a controller of a computer system configured to manage and coordinate a user's XR experience, according to some embodiments.

FIG. 3 is a block diagram illustrating a display generation component of a computer system configured to provide a visual component of an XR experience to a user, according to some embodiments.

FIG. 4 is a block diagram illustrating a hand tracking unit of a computer system configured to capture gesture inputs of a user, according to some embodiments.

Fig. 5 is a block diagram illustrating an eye tracking unit of a computer system configured to capture gaze input of a user, according to some embodiments.

Fig. 6 is a flow diagram illustrating a flash-assisted gaze tracking pipeline in accordance with some embodiments.

Fig. 7A-7I illustrate exemplary techniques for displaying a visual indication of a portion of a user, according to some embodiments.

FIG. 8 is a flow chart of a method of displaying a visual indication of a portion of a user, according to various embodiments.

Fig. 9A-9H illustrate exemplary techniques for displaying representations of different portions of a user with different visual fidelity metrics, according to some embodiments.

FIG. 10 is a flowchart of a method of displaying representations of different portions of a user with different visual fidelity metrics, according to various embodiments.

Detailed Description

According to some embodiments, the present disclosure relates to a user interface for providing an augmented reality (XR) experience to a user.

The systems, methods, and GUIs described herein improve user interface interactions with virtual/augmented reality environments in a variety of ways.

In some embodiments, the computer system displays a representation of the user within the augmented reality environment, but the computer system does not receive direct information and/or data indicative of the status of one or more portions of the user's body. Thus, the computer system displays a representation with a visual indication of one or more parts of the user's body that is not an anatomically accurate depiction of one or more parts of the user's body. The computer system estimates a state of one or more body parts of the user's body and displays a visual indication to enable another user viewing the representation to obtain a better understanding of the state of the user's whole body within the augmented reality environment. The visual indication is optionally a shadow indicating an estimated position of a lower part of the user's body. The visual indication is optionally displayed concurrently with a fuzzy extension that provides an additional visual indication of the portion of the user's body that the computer system did not receive direct information and/or data of the state of the portion of the body.

In some embodiments, the computer system displays a representation of the user within the augmented reality environment and displays a visual indication of a portion of the user's body in a different manner based on the location of the portion of the user's body within the physical environment in which the user is located. For example, the computer system determines the location of a portion of the user's body based on indirect and/or direct information about the status of the portion of the user's body within the physical environment in which the user is located. The computer system determines whether the location of the portion of the user's body is inside or outside a predefined area of the physical environment defined relative to the user's body. For example, when the part of the user's body is the user's hand, the predefined area corresponds to an area near the user's pocket. When the body part is the elbow of the user, the predefined area corresponds to an area near the waist and/or buttocks of the user. The computer system applies different visual fidelity metrics to the visual indication based on whether the portion of the user's body is determined to be inside or outside of the predefined region. The computer system optionally defines more than one predefined area and associates each predefined area with a particular portion of the user's body. For example, the computer system may associate a first predefined area with a user's hand and a second predefined area with the user's elbow. The predefined area optionally moves as the user physically moves within the physical environment such that the predefined area remains substantially stationary relative to the user's body.

Fig. 1-6 provide a description of an exemplary computer system for providing an XR experience to a user. Fig. 7A-7I illustrate exemplary techniques for displaying a visual indication of a portion of a user, according to some embodiments. FIG. 8 is a flow chart of a method of displaying a visual indication of a portion of a user, according to various embodiments. The user interfaces in fig. 7A to 7I are used to illustrate the process in fig. 8, respectively. Fig. 9A-9H illustrate exemplary techniques for displaying representations of different portions of a user with different visual fidelity metrics, according to some embodiments. FIG. 10 is a flowchart of a method of displaying representations of different portions of a user with different visual fidelity metrics, according to various embodiments. The user interfaces in fig. 9A to 9H are used to illustrate the process in fig. 10.

The processes described below enhance operability of a device and make a user-device interface more efficient (e.g., by helping a user provide appropriate input and reducing user errors in operating/interacting with the device) through various techniques, including by providing improved visual feedback to the user, reducing the number of inputs required to perform an operation, providing additional control options without cluttering the user interface with additional display controls, performing an operation when a set of conditions has been met without further user input, improving privacy and/or security, providing a richer, more detailed and/or more realistic user experience while conserving storage space, and/or additional techniques. These techniques also reduce power usage and extend battery life of the device by enabling a user to use the device faster and more efficiently. Saving battery power and thus weight, improves the ergonomics of the device. These techniques also enable real-time communication, allowing fewer and/or less accurate sensors to be used, resulting in a more compact, lighter, and cheaper device, and enabling the device to be used under a variety of lighting conditions. These techniques reduce energy usage and thus heat emitted by the device, which is particularly important for wearable devices where devices that fully conform to the operating parameters of the device components may be uncomfortable to wear if the device generates too much heat.

Furthermore, in a method described herein in which one or more steps are dependent on one or more conditions having been met, it should be understood that the method may be repeated in multiple iterations such that during the iteration, all conditions that determine steps in the method have been met in different iterations of the method. For example, if a method requires performing a first step (if a condition is met) and performing a second step (if a condition is not met), one of ordinary skill will know that the stated steps are repeated until both the condition and the condition are not met (not sequentially). Thus, a method described as having one or more steps depending on one or more conditions having been met may be rewritten as a method that repeats until each of the conditions described in the method have been met. However, this does not require the system or computer-readable medium to claim that the system or computer-readable medium contains instructions for performing the contingent operation based on the satisfaction of the corresponding condition or conditions, and thus is able to determine whether the contingent situation has been met without explicitly repeating the steps of the method until all conditions to decide on steps in the method have been met. It will also be appreciated by those of ordinary skill in the art that, similar to a method with optional steps, a system or computer readable storage medium may repeat the steps of the method as many times as necessary to ensure that all optional steps have been performed.

In some embodiments, as shown in fig. 1, an XR experience is provided to a user via an operating environment 100 comprising a computer system 101. The computer system 101 includes a controller 110 (e.g., a processor or remote server of a portable electronic device), a display generation component 120 (e.g., a Head Mounted Device (HMD), a display, a projector, a touch screen, etc.), one or more input devices 125 (e.g., an eye tracking device 130, a hand tracking device 140, other input devices 150), one or more output devices 155 (e.g., a speaker 160, a haptic output generator 170, and other output devices 180), one or more sensors 190 (e.g., an image sensor, a light sensor, a depth sensor, a haptic sensor, an orientation sensor, a proximity sensor, a temperature sensor, a position sensor, a motion sensor, a speed sensor, etc.), and optionally one or more peripheral devices 195 (e.g., a household appliance, a wearable device, etc.). In some implementations, one or more of the input device 125, the output device 155, the sensor 190, and the peripheral device 195 are integrated with the display generating component 120 (e.g., in a head-mounted device or a handheld device).

In describing an XR experience, various terms are used to refer differently to several related but different environments that a user may sense and/or interact with (e.g., interact with inputs detected by computer system 101 that generated the XR experience, such inputs causing the computer system that generated the XR experience to generate audio, visual, and/or tactile feedback corresponding to various inputs provided to computer system 101). The following are a subset of these terms:

Physical environment: a physical environment refers to a physical world in which people can sense and/or interact without the assistance of an electronic system. Physical environments such as physical parks include physical objects such as physical trees, physical buildings, and physical people. People can directly sense and/or interact with a physical environment, such as by visual, tactile, auditory, gustatory, and olfactory.

And (3) augmented reality: conversely, an augmented reality (XR) environment refers to a fully or partially simulated environment in which people sense and/or interact via an electronic system. In XR, a subset of the physical movements of the person, or a representation thereof, is tracked, and in response, one or more characteristics of one or more virtual objects simulated in the XR environment are adjusted in a manner consistent with at least one physical law. For example, an XR system may detect a person's head rotation and, in response, adjust the graphical content and sound field presented to the person in a manner similar to the manner in which such views and sounds change in a physical environment. In some cases (e.g., for reachability reasons), the adjustment of the characteristics of the virtual object in the XR environment may be made in response to a representation of the physical motion (e.g., a voice command). A person may utilize any of his senses to sense and/or interact with XR objects, including vision, hearing, touch, taste, and smell. For example, a person may sense and/or interact with audio objects that create a 3D or spatial audio environment that provides perception of a point audio source in 3D space. As another example, an audio object may enable audio transparency that selectively introduces environmental sounds from a physical environment with or without computer generated audio. In some XR environments, a person may sense and/or interact with only audio objects.

Examples of XRs include virtual reality and mixed reality.

Virtual reality: a Virtual Reality (VR) environment refers to a simulated environment designed to be based entirely on computer-generated sensory input for one or more senses. The VR environment includes a plurality of virtual objects that a person can sense and/or interact with. For example, computer-generated images of trees, buildings, and avatars representing people are examples of virtual objects. A person may sense and/or interact with virtual objects in the VR environment through a simulation of the presence of the person within the computer-generated environment and/or through a simulation of a subset of the physical movements of the person within the computer-generated environment.

Mixed reality: in contrast to VR environments designed to be based entirely on computer-generated sensory input, a Mixed Reality (MR) environment refers to a simulated environment designed to introduce sensory input from a physical environment or a representation thereof in addition to including computer-generated sensory input (e.g., virtual objects). On a virtual continuum, a mixed reality environment is any condition between, but not including, a full physical environment as one end and a virtual reality environment as the other end. In some MR environments, the computer-generated sensory input may be responsive to changes in sensory input from the physical environment. In addition, some electronic systems for rendering MR environments may track the position and/or orientation relative to the physical environment to enable virtual objects to interact with real objects (i.e., physical objects or representations thereof from the physical environment). For example, the system may cause the motion such that the virtual tree appears to be stationary relative to the physical ground.

Examples of mixed reality include augmented reality and augmented virtualization.

Augmented reality: an Augmented Reality (AR) environment refers to a simulated environment in which one or more virtual objects are superimposed over a physical environment or a representation of a physical environment. For example, an electronic system for presenting an AR environment may have a transparent or translucent display through which a person may directly view the physical environment. The system may be configured to present the virtual object on a transparent or semi-transparent display such that a person perceives the virtual object superimposed over the physical environment with the system. Alternatively, the system may have an opaque display and one or more imaging sensors that capture images or videos of the physical environment, which are representations of the physical environment. The system combines the image or video with the virtual object and presents the composition on an opaque display. A person utilizes the system to indirectly view the physical environment via an image or video of the physical environment and perceive a virtual object superimposed over the physical environment. As used herein, video of a physical environment displayed on an opaque display is referred to as "pass-through video," meaning that the system captures images of the physical environment using one or more image sensors and uses those images when rendering an AR environment on the opaque display. Further alternatively, the system may have a projection system that projects the virtual object into the physical environment, for example as a hologram or on a physical surface, such that a person perceives the virtual object superimposed on top of the physical environment with the system. An augmented reality environment also refers to a simulated environment in which a representation of a physical environment is transformed by computer-generated sensory information. For example, in providing a passthrough video, the system may transform one or more sensor images to apply a selected viewing angle (e.g., a viewpoint) that is different from the viewing angle captured by the imaging sensor. As another example, the representation of the physical environment may be transformed by graphically modifying (e.g., magnifying) portions thereof such that the modified portions may be representative but not real versions of the original captured image. For another example, the representation of the physical environment may be transformed by graphically eliminating or blurring portions thereof.

Enhanced virtualization: enhanced virtual (AV) environment refers to a simulated environment in which a virtual environment or computer-generated environment incorporates one or more sensory inputs from a physical environment. The sensory input may be a representation of one or more characteristics of the physical environment. For example, an AV park may have virtual trees and virtual buildings, but the face of a person is realistically reproduced from an image taken of a physical person. As another example, the virtual object may take the shape or color of a physical object imaged by one or more imaging sensors. For another example, the virtual object may employ shadows that conform to the positioning of the sun in the physical environment.

Viewpoint-locked virtual object: when the computer system displays the virtual object at the same location and/or position in the user's viewpoint, the virtual object is viewpoint-locked even if the user's viewpoint is offset (e.g., changed). In embodiments in which the computer system is a head-mounted device, the user's point of view is locked to the forward direction of the user's head (e.g., when the user looks directly in front, the user's point of view is at least a portion of the user's field of view); thus, the user's point of view remains fixed without moving the user's head, even when the user's gaze is offset. In embodiments in which the computer system has a display generating component (e.g., a display screen) that is repositionable relative to the user's head, the user's point of view is an augmented reality view presented to the user on the display generating component of the computer system. For example, a viewpoint-locked virtual object displayed in the upper left corner of the user's viewpoint continues to be displayed in the upper left corner of the user's viewpoint when the user's viewpoint is in a first orientation (e.g., the user's head faces north), even when the user's viewpoint changes to a second orientation (e.g., the user's head faces west). In other words, the position and/or orientation of the virtual object in which the viewpoint lock is displayed in the viewpoint of the user is independent of the position and/or orientation of the user in the physical environment. In embodiments in which the computer system is a head-mounted device, the user's point of view is locked to the orientation of the user's head, such that the virtual object is also referred to as a "head-locked virtual object.

Environment-locked visual object: when a computer system displays a virtual object at a location and/or position in a user's point of view, the virtual object is environment-locked (alternatively, "world-locked"), the location and/or position being based on (e.g., selected and/or anchored to) a location and/or object in a three-dimensional environment (e.g., a physical environment or virtual environment) with reference to the location and/or object. As the user's point of view moves, the position and/or object in the environment relative to the user's point of view changes, which results in the environment-locked virtual object being displayed at a different position and/or location in the user's point of view. For example, an environmentally locked virtual object that locks onto a tree immediately in front of the user is displayed at the center of the user's viewpoint. When the user's viewpoint is shifted to the right (e.g., the user's head is turned to the right) such that the tree is now to the left of center in the user's viewpoint (e.g., the tree positioning in the user's viewpoint is shifted), the environmentally locked virtual object that is locked onto the tree is displayed to the left of center in the user's viewpoint. In other words, the position and/or orientation at which the environment-locked virtual object is displayed in the user's viewpoint depends on the position and/or orientation of the object and/or the position at which the virtual object is locked in the environment. In some embodiments, the computer system uses a stationary frame of reference (e.g., a coordinate system anchored to a fixed location and/or object in the physical environment) in order to determine the location of the virtual object that displays the environmental lock in the viewpoint of the user. The environment-locked virtual object may be locked to a stationary portion of the environment (e.g., a floor, wall, table, or other stationary object), or may be locked to a movable portion of the environment (e.g., a representation of a vehicle, animal, person, or even a portion of a user's body such as a user's hand, wrist, arm, or foot that moves independent of the user's point of view) such that the virtual object moves as the point of view or the portion of the environment moves to maintain a fixed relationship between the virtual object and the portion of the environment.

In some implementations, the environmentally or view-locked virtual object exhibits an inert follow-up behavior that reduces or delays movement of the environmentally or view-locked virtual object relative to movement of a reference point that the virtual object follows. In some embodiments, the computer system intentionally delays movement of the virtual object when detecting movement of a reference point (e.g., a portion of the environment, a viewpoint, or a point fixed relative to the viewpoint, such as a point between 5cm and 300cm from the viewpoint) that the virtual object is following while exhibiting inert follow-up behavior. For example, when a reference point (e.g., the portion or viewpoint of the environment) moves at a first speed, the virtual object is moved by the device to remain locked to the reference point, but moves at a second speed that is slower than the first speed (e.g., until the reference point stops moving or slows down, at which point the virtual object begins to catch up with the reference point). In some embodiments, when the virtual object exhibits inert follow-up behavior, the device ignores small movements of the reference point (e.g., ignores movements of the reference point below a threshold amount of movement, such as movements of 0 to 5 degrees or movements of 0 to 50 cm). For example, when a reference point (e.g., the portion or point of view of the environment to which the virtual object is locked) moves a first amount, the distance between the reference point and the virtual object increases (e.g., because the virtual object is being displayed so as to maintain a fixed or substantially fixed position relative to the portion of the environment or point of view other than the reference point to which the virtual object is locked), and when the reference point (e.g., the portion or point of view of the environment to which the virtual object is locked) moves a second amount that is greater than the first amount, the distance between the reference point and the virtual object initially increases (e.g., because the virtual object is being displayed so as to maintain a fixed or substantially fixed position relative to the portion of the environment other than the point of view or point to which the virtual object is locked), and then decreases as the amount of movement of the reference point increases above a threshold (e.g., an "inertia following" threshold) because the virtual object is moved by the computer system to maintain a fixed or substantially fixed position relative to the reference point. In some embodiments, maintaining a substantially fixed position of the virtual object relative to the reference point includes the virtual object being displayed within a threshold distance (e.g., 1cm, 2cm, 3cm, 5cm, 15cm, 20cm, 50 cm) of the reference point in one or more dimensions (e.g., up/down, left/right, and/or forward/backward relative to the position of the reference point).

Hardware: there are many different types of electronic systems that enable a person to sense and/or interact with various XR environments. Examples include head-mounted systems, projection-based systems, head-up displays (HUDs), vehicle windshields integrated with display capabilities, windows integrated with display capabilities, displays formed as lenses designed for placement on a person's eyes (e.g., similar to contact lenses), headphones/earphones, speaker arrays, input systems (e.g., wearable or handheld controllers with or without haptic feedback), smartphones, tablets, and desktop/laptop computers. The head-mounted system may include speakers and/or other audio output devices integrated into the head-mounted system for providing audio output. The head-mounted system may have one or more speakers and an integrated opaque display. Alternatively, the head-mounted system may be configured to accept an external opaque display (e.g., a smart phone). The head-mounted system may incorporate one or more imaging sensors for capturing images or video of the physical environment and/or one or more microphones for capturing audio of the physical environment. The head-mounted system may have a transparent or translucent display instead of an opaque display. The transparent or translucent display may have a medium through which light representing an image is directed to the eyes of a person. The display may utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light sources, or any combination of these techniques. The medium may be an optical waveguide, a holographic medium, an optical combiner, an optical reflector, or any combination thereof. In one embodiment, the transparent or translucent display may be configured to selectively become opaque. Projection-based systems may employ retinal projection techniques that project a graphical image onto a person's retina. The projection system may also be configured to project the virtual object into the physical environment, e.g. as a hologram, or onto a physical surface. In some embodiments, the controller 110 is configured to manage and coordinate the XR experience of the user. In some embodiments, controller 110 includes suitable combinations of software, firmware, and/or hardware. The controller 110 is described in more detail below with reference to fig. 2. In some implementations, the controller 110 is a computing device that is in a local or remote location relative to the scene 105 (e.g., physical environment). For example, the controller 110 is a local server located within the scene 105. As another example, the controller 110 is a remote server (e.g., cloud server, central server, etc.) located outside of the scene 105. In some implementations, the controller 110 is communicatively coupled with the display generation component 120 (e.g., HMD, display, projector, touch-screen, etc.) via one or more wired or wireless communication channels 144 (e.g., bluetooth, IEEE 802.11x, IEEE 802.16x, IEEE 802.3x, etc.). In another example, the controller 110 is included within a housing (e.g., a physical enclosure) of the display generation component 120 (e.g., an HMD or portable electronic device including a display and one or more processors, etc.), one or more of the input devices 125, one or more of the output devices 155, one or more of the sensors 190, and/or one or more of the peripheral devices 195, or shares the same physical housing or support structure with one or more of the above.

In some embodiments, display generation component 120 is configured to provide an XR experience (e.g., at least a visual component of the XR experience) to a user. In some embodiments, display generation component 120 includes suitable combinations of software, firmware, and/or hardware. The display generating section 120 is described in more detail below with respect to fig. 3. In some embodiments, the functionality of the controller 110 is provided by and/or combined with the display generating component 120.

According to some embodiments, display generation component 120 provides an XR experience to a user when the user is virtually and/or physically present within scene 105.

In some embodiments, the display generating component is worn on a portion of the user's body (e.g., on his/her head, on his/her hand, etc.). As such, display generation component 120 includes one or more XR displays provided for displaying XR content. For example, in various embodiments, the display generation component 120 encloses a field of view of a user. In some embodiments, display generation component 120 is a handheld device (such as a smart phone or tablet computer) configured to present XR content, and the user holds the device with a display facing the user's field of view and a camera facing scene 105. In some embodiments, the handheld device is optionally placed within a housing that is worn on the head of the user. In some embodiments, the handheld device is optionally placed on a support (e.g., tripod) in front of the user. In some embodiments, display generation component 120 is an XR room, housing, or room configured to present XR content, wherein the user does not wear or hold display generation component 120. Many of the user interfaces described with reference to one type of hardware for displaying XR content (e.g., a handheld device or a device on a tripod) may be implemented on another type of hardware for displaying XR content (e.g., an HMD or other wearable computing device). For example, a user interface showing interactions with XR content triggered based on interactions occurring in a space in front of a handheld device or a tripod-mounted device may similarly be implemented with an HMD, where the interactions occur in the space in front of the HMD and responses to the XR content are displayed via the HMD. Similarly, a user interface showing interaction with XR content triggered based on movement of a handheld device or tripod-mounted device relative to a physical environment (e.g., a scene 105 or a portion of a user's body (e.g., a user's eye, head, or hand)) may similarly be implemented with an HMD, where the movement is caused by movement of the HMD relative to the physical environment (e.g., the scene 105 or a portion of the user's body (e.g., a user's eye, head, or hand)).

While relevant features of the operating environment 100 are shown in fig. 1, those of ordinary skill in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the exemplary embodiments disclosed herein.

Fig. 2 is a block diagram of an example of a controller 110 according to some embodiments. While certain specific features are shown, those of ordinary skill in the art will appreciate from the disclosure that various other features are not shown for the sake of brevity and so as not to obscure more pertinent aspects of the embodiments disclosed herein. To this end, as a non-limiting example, in some embodiments, the controller 110 includes one or more processing units 202 (e.g., microprocessors, application Specific Integrated Circuits (ASICs), field Programmable Gate Arrays (FPGAs), graphics Processing Units (GPUs), central Processing Units (CPUs), processing cores, etc.), one or more input/output (I/O) devices 206, one or more communication interfaces 208 (e.g., universal Serial Bus (USB), IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, global system for mobile communications (GSM), code Division Multiple Access (CDMA), time Division Multiple Access (TDMA), global Positioning System (GPS), infrared (IR), bluetooth, ZIGBEE, and/or similar types of interfaces), one or more programming (e.g., I/O) interfaces 210, memory 220, and one or more communication buses 204 for interconnecting these components and various other components.

In some embodiments, one or more of the communication buses 204 include circuitry that interconnects and controls communications between system components. In some embodiments, the one or more I/O devices 206 include at least one of a keyboard, a mouse, a touchpad, a joystick, one or more microphones, one or more speakers, one or more image sensors, one or more displays, and the like.

Memory 220 includes high-speed random access memory such as Dynamic Random Access Memory (DRAM), static Random Access Memory (SRAM), double data rate random access memory (DDR RAM), or other random access solid state memory devices. In some embodiments, memory 220 includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid state storage devices. Memory 220 optionally includes one or more storage devices located remotely from the one or more processing units 202. Memory 220 includes a non-transitory computer-readable storage medium. In some embodiments, memory 220 or a non-transitory computer readable storage medium of memory 220 stores the following programs, modules, and data structures, or a subset thereof, including optional operating system 230 and XR experience module 240.

Operating system 230 includes instructions for handling various basic system services and for performing hardware-related tasks. In some embodiments, XR experience module 240 is configured to manage and coordinate single or multiple XR experiences of one or more users (e.g., single XR experiences of one or more users, or multiple XR experiences of a respective group of one or more users). To this end, in various embodiments, the XR experience module 240 includes a data acquisition unit 241, a tracking unit 242, a coordination unit 246, and a data transmission unit 248.

In some embodiments, the data acquisition unit 241 is configured to acquire data (e.g., presentation data, interaction data, sensor data, location data, etc.) from at least the display generation component 120 of fig. 1, and optionally from one or more of the input device 125, the output device 155, the sensor 190, and/or the peripheral device 195. To this end, in various embodiments, the data acquisition unit 241 includes instructions and/or logic for instructions as well as heuristics and metadata for heuristics.

In some embodiments, tracking unit 242 is configured to map scene 105 and track at least the location/position of display generation component 120 relative to scene 105 of fig. 1, and optionally relative to one or more of tracking input device 125, output device 155, sensor 190, and/or peripheral device 195. To this end, in various embodiments, the tracking unit 242 includes instructions and/or logic for instructions as well as heuristics and metadata for heuristics. In some embodiments, tracking unit 242 includes a hand tracking unit 244 and/or an eye tracking unit 243. In some embodiments, the hand tracking unit 244 is configured to track the location/position of one or more portions of the user's hand, and/or the motion of one or more portions of the user's hand relative to the scene 105 of fig. 1, relative to the display generating component 120, and/or relative to a coordinate system defined relative to the user's hand. The hand tracking unit 244 is described in more detail below with respect to fig. 4. In some embodiments, the eye tracking unit 243 is configured to track the positioning or movement of the user gaze (or more generally, the user's eyes, face, or head) relative to the scene 105 (e.g., relative to the physical environment and/or relative to the user (e.g., the user's hand)) or relative to XR content displayed via the display generating component 120. The eye tracking unit 243 is described in more detail below with respect to fig. 5.

In some embodiments, coordination unit 246 is configured to manage and coordinate XR experiences presented to a user by display generation component 120, and optionally by one or more of output device 155 and/or peripheral device 195. For this purpose, in various embodiments, coordination unit 246 includes instructions and/or logic for instructions as well as heuristics and metadata for heuristics.

In some embodiments, the data transmission unit 248 is configured to transmit data (e.g., presentation data, location data, etc.) to at least the display generation component 120, and optionally to one or more of the input device 125, the output device 155, the sensor 190, and/or the peripheral device 195. For this purpose, in various embodiments, the data transmission unit 248 includes instructions and/or logic for instructions as well as heuristics and metadata for heuristics.

While the data acquisition unit 241, tracking unit 242 (e.g., including eye tracking unit 243 and hand tracking unit 244), coordination unit 246, and data transmission unit 248 are shown as residing on a single device (e.g., controller 110), it should be understood that in other embodiments, any combination of the data acquisition unit 241, tracking unit 242 (e.g., including eye tracking unit 243 and hand tracking unit 244), coordination unit 246, and data transmission unit 248 may reside in a single computing device.

Furthermore, FIG. 2 is a functional description of various features that may be present in a particular implementation, as opposed to a schematic of the embodiments described herein. As will be appreciated by one of ordinary skill in the art, the individually displayed items may be combined and some items may be separated. For example, some of the functional blocks shown separately in fig. 2 may be implemented in a single block, and the various functions of a single functional block may be implemented by one or more functional blocks in various embodiments. The actual number of modules and the division of particular functions, and how features are allocated among them, will vary depending upon the particular implementation, and in some embodiments, depend in part on the particular combination of hardware, software, and/or firmware selected for a particular implementation.

Fig. 3 is a block diagram of an example of display generation component 120 according to some embodiments. While certain specific features are shown, those of ordinary skill in the art will appreciate from the disclosure that various other features are not shown for the sake of brevity and so as not to obscure more pertinent aspects of the embodiments disclosed herein. For this purpose, as a non-limiting example, in some embodiments, display generation component 120 (e.g., HMD) includes one or more processing units 302 (e.g., microprocessors, ASIC, FPGA, GPU, CPU, processing cores, etc.), one or more input/output (I/O) devices and sensors 306, one or more communication interfaces 308 (e.g., ,USB、FIREWIRE、THUNDERBOLT、IEEE 802.3x、IEEE 802.11x、IEEE 802.16x、GSM、CDMA、TDMA、GPS、IR、BLUETOOTH、ZIGBEE and/or similar types of interfaces), one or more programming (e.g., I/O) interfaces 310, one or more XR displays 312, one or more optional internally and/or externally facing image sensors 314, memory 320, and one or more communication buses 304 for interconnecting these components and various other components.

In some embodiments, one or more communication buses 304 include circuitry for interconnecting and controlling communications between various system components. In some embodiments, the one or more I/O devices and sensors 306 include an Inertial Measurement Unit (IMU), an accelerometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., blood pressure monitor, heart rate monitor, blood oxygen sensor, blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptic engine, and/or one or more depth sensors (e.g., structured light, time of flight, etc.), and/or the like.

In some embodiments, one or more XR displays 312 are configured to provide an XR experience to a user. In some embodiments, one or more XR displays 312 correspond to holographic, digital Light Processing (DLP), liquid Crystal Displays (LCD), liquid crystal on silicon (LCoS), organic light emitting field effect transistors (OLET), organic Light Emitting Diodes (OLED), surface conduction electron emitting displays (SED), field Emission Displays (FED), quantum dot light emitting diodes (QD-LED), microelectromechanical systems (MEMS), and/or similar display types. In some embodiments, one or more XR displays 312 correspond to diffractive, reflective, polarizing, holographic, etc. waveguide displays. For example, the display generation component 120 (e.g., HMD) includes a single XR display. In another example, display generation component 120 includes an XR display for each eye of the user. In some embodiments, one or more XR displays 312 are capable of presenting MR and VR content. In some implementations, one or more XR displays 312 can present MR or VR content.

In some embodiments, the one or more image sensors 314 are configured to acquire image data corresponding to at least a portion of the user's face including the user's eyes (and may be referred to as an eye tracking camera). In some embodiments, the one or more image sensors 314 are configured to acquire image data corresponding to at least a portion of the user's hand and optionally the user's arm (and may be referred to as a hand tracking camera). In some implementations, the one or more image sensors 314 are configured to face forward in order to acquire image data corresponding to a scene that a user would see in the absence of the display generating component 120 (e.g., HMD) (and may be referred to as a scene camera). The one or more optional image sensors 314 may include one or more RGB cameras (e.g., with Complementary Metal Oxide Semiconductor (CMOS) image sensors or Charge Coupled Device (CCD) image sensors), one or more Infrared (IR) cameras, and/or one or more event-based cameras, etc.

Memory 320 includes high-speed random access memory such as DRAM, SRAM, DDR RAM or other random access solid state memory devices. In some embodiments, memory 320 includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid state storage devices. Memory 320 optionally includes one or more storage devices located remotely from the one or more processing units 302. Memory 320 includes a non-transitory computer-readable storage medium. In some embodiments, memory 320 or a non-transitory computer readable storage medium of memory 320 stores the following programs, modules, and data structures, or a subset thereof, including optional operating system 330 and XR presentation module 340.

Operating system 330 includes processes for handling various basic system services and for performing hardware-related tasks. In some embodiments, XR presentation module 340 is configured to present XR content to a user via one or more XR displays 312. To this end, in various embodiments, the XR presentation module 340 includes a data acquisition unit 342, an XR presentation unit 344, an XR map generation unit 346, and a data transmission unit 348.

In some embodiments, the data acquisition unit 342 is configured to at least acquire data (e.g., presentation data, interaction data, sensor data, location data, etc.) from the controller 110 of fig. 1. For this purpose, in various embodiments, the data acquisition unit 342 includes instructions and/or logic for instructions and heuristics and metadata for heuristics.

In some embodiments, XR presentation unit 344 is configured to present XR content via one or more XR displays 312. For this purpose, in various embodiments, XR presentation unit 344 includes instructions and/or logic for instructions and heuristics and metadata for heuristics.

In some embodiments, XR map generation unit 346 is configured to generate an XR map based on the media content data (e.g., a 3D map of a mixed reality scene or a map of a physical environment in which computer-generated objects may be placed to generate an augmented reality). For this purpose, in various embodiments, XR map generation unit 346 includes instructions and/or logic for the instructions as well as heuristics and metadata for the heuristics.

In some embodiments, the data transmission unit 348 is configured to transmit data (e.g., presentation data, location data, etc.) to at least the controller 110, and optionally one or more of the input device 125, the output device 155, the sensor 190, and/or the peripheral device 195. For this purpose, in various embodiments, the data transmission unit 348 includes instructions and/or logic for instructions and heuristics and metadata for heuristics.

Although the data acquisition unit 342, the XR presentation unit 344, the XR map generation unit 346, and the data transmission unit 348 are shown as residing on a single device (e.g., the display generation component 120 of fig. 1), it should be understood that in other embodiments, any combination of the data acquisition unit 342, the XR presentation unit 344, the XR map generation unit 346, and the data transmission unit 348 may be located in separate computing devices.

Furthermore, fig. 3 is used more as a functional description of various features that may be present in a particular embodiment, as opposed to a schematic of the embodiments described herein. As will be appreciated by one of ordinary skill in the art, the individually displayed items may be combined and some items may be separated. For example, some of the functional blocks shown separately in fig. 3 may be implemented in a single block, and the various functions of a single functional block may be implemented by one or more functional blocks in various embodiments. The actual number of modules and the division of particular functions, and how features are allocated among them, will vary depending upon the particular implementation, and in some embodiments, depend in part on the particular combination of hardware, software, and/or firmware selected for a particular implementation.

Fig. 4 is a schematic illustration of an exemplary embodiment of a hand tracking device 140. In some embodiments, the hand tracking device 140 (fig. 1) is controlled by the hand tracking unit 244 (fig. 2) to track the position/location of one or more portions of the user's hand, and/or the movement of one or more portions of the user's hand relative to the scene 105 of fig. 1 (e.g., relative to a portion of the physical environment surrounding the user, relative to the display generating component 120, or relative to a portion of the user (e.g., the user's face, eyes, or head), and/or relative to a coordinate system defined relative to the user's hand). In some implementations, the hand tracking device 140 is part of the display generation component 120 (e.g., embedded in or attached to a head-mounted device). In some embodiments, the hand tracking device 140 is separate from the display generation component 120 (e.g., in a separate housing or attached to a separate physical support structure).

In some implementations, the hand tracking device 140 includes an image sensor 404 (e.g., one or more IR cameras, 3D cameras, depth cameras, and/or color cameras, etc.) that captures three-dimensional scene information including at least a human user's hand 406. The image sensor 404 captures the hand image with sufficient resolution to enable the fingers and their respective locations to be distinguished. The image sensor 404 typically captures images of other parts of the user's body, and possibly also all parts of the body, and may have a zoom capability or a dedicated sensor with increased magnification to capture images of the hand with a desired resolution. In some implementations, the image sensor 404 also captures 2D color video images of the hand 406 and other elements of the scene. In some implementations, the image sensor 404 is used in conjunction with other image sensors to capture the physical environment of the scene 105, or as an image sensor that captures the physical environment of the scene 105. In some embodiments, the image sensor 404, or a portion thereof, is positioned relative to the user or the user's environment in a manner that uses the field of view of the image sensor to define an interaction space in which hand movements captured by the image sensor are considered input to the controller 110.

In some embodiments, the image sensor 404 outputs a sequence of frames containing 3D mapping data (and, in addition, possible color image data) to the controller 110, which extracts high-level information from the mapping data. This high-level information is typically provided via an Application Program Interface (API) to an application program running on the controller, which drives the display generating component 120 accordingly. For example, a user may interact with software running on the controller 110 by moving his hand 406 and changing his hand pose.

In some implementations, the image sensor 404 projects a speckle pattern onto a scene that includes the hand 406 and captures an image of the projected pattern. In some implementations, the controller 110 calculates 3D coordinates of points in the scene (including points on the surface of the user's hand) by triangulation based on lateral offsets of the blobs in the pattern. This approach is advantageous because it does not require the user to hold or wear any kind of beacon, sensor or other marker. The method gives the depth coordinates of points in the scene relative to a predetermined reference plane at a specific distance from the image sensor 404. In this disclosure, it is assumed that the image sensor 404 defines an orthogonal set of x-axis, y-axis, z-axis such that the depth coordinates of points in the scene correspond to the z-component measured by the image sensor. Alternatively, the image sensor 404 (e.g., a hand tracking device) may use other 3D mapping methods, such as stereoscopic imaging or time-of-flight measurements, based on single or multiple cameras or other types of sensors.

In some implementations, the hand tracking device 140 captures and processes a time series containing a depth map of the user's hand as the user moves his hand (e.g., the entire hand or one or more fingers). Software running on the image sensor 404 and/or a processor in the controller 110 processes the 3D mapping data to extract image block descriptors of the hand in these depth maps. The software may match these descriptors with image block descriptors stored in database 408 based on previous learning processes in order to estimate the pose of the hand in each frame. The pose typically includes the 3D position of the user's hand joints and finger tips.

The software may also analyze the trajectory of the hand and/or finger over multiple frames in the sequence to identify gestures. The pose estimation functions described herein may alternate with motion tracking functions such that image block-based pose estimation is performed only once every two (or more) frames while tracking changes used to find poses that occur on the remaining frames. Pose, motion, and gesture information are provided to an application running on the controller 110 via the APIs described above. The program may move and modify images presented on the display generation component 120, for example, in response to pose and/or gesture information, or perform other functions.

In some implementations, the gesture includes an air gesture. An air gesture is a motion of a portion of a user's body (e.g., a head, one or more arms, one or more hands, one or more fingers, and/or one or more legs) through the air that is detected without the user touching an input element (or being independent of an input element that is part of a device) that is part of a device (e.g., computer system 101, one or more input devices 125, and/or hand tracking device 140) (including a motion of the user's body relative to an absolute reference (e.g., angle of the user's arm relative to the ground or distance of the user's hand relative to the ground), movement relative to another portion of the user's body (e.g., movement of the user's hand relative to the user's shoulder, movement of one hand of the user relative to the other hand of the user, and/or movement of the user's finger relative to the other finger or portion of the hand of the user), and/or absolute movement of a portion of the user's body (e.g., a flick gesture comprising a predetermined amount and/or speed of movement of the hand in a predetermined gesture, or a shake gesture comprising a predetermined speed or amount of rotation of a portion of the user's body)).

In some embodiments, according to some embodiments, the input gestures used in the various examples and embodiments described herein include air gestures performed by movement of a user's finger relative to other fingers or portions of the user's hand for interacting with an XR environment (e.g., a virtual or mixed reality environment). In some embodiments, the air gesture is a gesture that is detected without the user touching an input element that is part of the device (or independent of an input element that is part of the device) and based on a detected movement of a portion of the user's body through the air, including a movement of the user's body relative to an absolute reference (e.g., an angle of the user's arm relative to the ground or a distance of the user's hand relative to the ground), a movement relative to another portion of the user's body (e.g., a movement of the user's hand relative to the user's shoulder, a movement of the user's hand relative to the other hand of the user, and/or a movement of the user's finger relative to the other finger or part of the hand of the user), and/or an absolute movement of a portion of the user's body (e.g., a flick gesture that includes a predetermined amount and/or speed of movement of the hand in a predetermined gesture that includes a predetermined gesture of the hand, or a shake gesture that includes a predetermined speed or amount of rotation of a portion of the user's body).

In some embodiments where the input gesture is an air gesture (e.g., in the absence of physical contact with the input device, the input device provides information to the computer system as to which user interface element is the target of the user input, such as contact with a user interface element displayed on a touch screen, or contact with a mouse or touchpad to move a cursor to the user interface element), the gesture takes into account the user's attention (e.g., gaze) to determine the target of the user input (e.g., for direct input, as described below). Thus, in embodiments involving air gestures, for example, an input gesture in combination (e.g., simultaneously) with movement of a user's finger and/or hand detects an attention (e.g., gaze) toward a user interface element to perform pinch and/or tap inputs, as described below.

In some implementations, an input gesture directed to a user interface object is performed with direct or indirect reference to the user interface object. For example, user input is performed directly on a user interface object according to performing input with a user's hand at a location corresponding to the location of the user interface object in a three-dimensional environment (e.g., as determined based on the user's current viewpoint). In some implementations, upon detecting a user's attention (e.g., gaze) to a user interface object, an input gesture is performed indirectly on the user interface object in accordance with a position of a user's hand not being at the position corresponding to the position of the user interface object in the three-dimensional environment while the user is performing the input gesture. For example, for a direct input gesture, the user can direct the user's input to the user interface object by initiating the gesture at or near a location corresponding to the display location of the user interface object (e.g., within 0.5cm, 1cm, 5cm, or within a distance between 0 and 5cm measured from the outer edge of the option or the center portion of the option). For indirect input gestures, a user can direct the user's input to a user interface object by focusing on the user interface object (e.g., by looking at the user interface object), and while focusing on an option, the user initiates an input gesture (e.g., at any location detectable by the computer system) (e.g., at a location that does not correspond to the display location of the user interface object).

In some embodiments, according to some embodiments, the input gestures (e.g., air gestures) used in the various examples and embodiments described herein include pinch inputs and tap inputs for interacting with a virtual or mixed reality environment. For example, pinch and tap inputs described below are performed as air gestures.

In some implementations, the pinch input is part of an air gesture that includes one or more of: pinch gestures, long pinch gestures, pinch and drag gestures, or double pinch gestures. For example, pinch gestures as air gestures include movements of two or more fingers of a hand to contact each other, i.e., optionally, immediately followed by interruption of contact with each other (e.g., within 0 to 1 second). A long pinch gesture, which is an air gesture, includes movement of two or more fingers of a hand into contact with each other for at least a threshold amount of time (e.g., at least 1 second) before a break in contact with each other is detected. For example, a long pinch gesture includes a user holding a pinch gesture (e.g., where two or more fingers make contact), and the long pinch gesture continues until a break in contact between the two or more fingers is detected. In some implementations, the double pinch gesture as an air gesture includes two (e.g., or more) pinch inputs (e.g., performed by the same hand) that are detected in succession with each other immediately (e.g., within a predefined period of time). For example, the user performs a first pinch input (e.g., a pinch input or a long pinch input), releases the first pinch input (e.g., breaks contact between two or more fingers), and performs a second pinch input within a predefined period of time (e.g., within 1 second or within 2 seconds) after releasing the first pinch input.

In some implementations, the pinch-and-drag gesture as an air gesture includes a pinch gesture (e.g., a pinch gesture or a long pinch gesture) that is performed in conjunction with (e.g., follows) a drag input that changes a position of a user's hand from a first position (e.g., a start position of the drag) to a second position (e.g., an end position of the drag). In some implementations, the user holds the pinch gesture while the drag input is performed, and releases the pinch gesture (e.g., opens their two or more fingers) to end the drag gesture (e.g., at the second location). In some implementations, pinch input and drag input are performed by the same hand (e.g., a user pinch two or more fingers to contact each other and move the same hand to a second position in the air with a drag gesture). In some embodiments, the input gesture as an over-the-air gesture includes an input (e.g., pinch and/or tap input) performed using two hands of the user, e.g., the input gesture includes two (e.g., or more) inputs performed in conjunction with each other (e.g., simultaneously or within a predefined time period).

In some implementations, the tap input (e.g., pointing to the user interface element) performed as an air gesture includes movement of a user's finger toward the user interface element, movement of a user's hand toward the user interface element (optionally, the user's finger extends toward the user interface element), downward movement of the user's finger (e.g., mimicking a mouse click motion or a tap on a touch screen), or other predefined movement of the user's hand. In some embodiments, a flick input performed as an air gesture is detected based on a movement characteristic of a finger or hand performing a flick gesture movement of the finger or hand away from a user's point of view and/or toward an object that is a target of the flick input, followed by an end of the movement. In some embodiments, the end of movement is detected based on a change in movement characteristics of the finger or hand performing the flick gesture (e.g., the end of movement away from the user's point of view and/or toward an object that is the target of the flick input, the reversal of the direction of movement of the finger or hand, and/or the reversal of the acceleration direction of movement of the finger or hand).

In some embodiments, the determination that the user's attention is directed to a portion of the three-dimensional environment is based on detection of gaze directed to that portion (optionally, without other conditions). In some embodiments, the portion of the three-dimensional environment to which the user's attention is directed is determined based on detecting a gaze directed to the portion of the three-dimensional environment with one or more additional conditions, such as requiring the gaze to be directed to the portion of the three-dimensional environment for at least a threshold duration (e.g., dwell duration) and/or requiring the gaze to be directed to the portion of the three-dimensional environment when the point of view of the user is within a distance threshold from the portion of the three-dimensional environment, such that the device determines the portion of the three-dimensional environment to which the user's attention is directed, wherein if one of the additional conditions is not met, the device determines that the attention is not directed to the portion of the three-dimensional environment to which the gaze is directed (e.g., until the one or more additional conditions are met).

In some embodiments, detection of the ready state configuration of the user or a portion of the user is detected by the computer system. Detection of a ready state configuration of a hand is used by a computer system as an indication that a user may be ready to interact with the computer system using one or more air gesture inputs (e.g., pinch, tap, pinch and drag, double pinch, long pinch, or other air gestures described herein) performed by the hand. For example, the ready state of the hand is determined based on whether the hand has a predetermined hand shape (e.g., a pre-pinch shape in which the thumb and one or more fingers extend and are spaced apart in preparation for making a pinch or grasp gesture, or a pre-flick in which the one or more fingers extend and the palm faces away from the user), based on whether the hand is in a predetermined position relative to the user's point of view (e.g., below the user's head and above the user's waist and extending at least 15cm, 20cm, 25cm, 30cm, or 50cm from the body), and/or based on whether the hand has moved in a particular manner (e.g., toward an area above the user's waist and in front of the user's head or away from the user's body or legs). In some implementations, the ready state is used to determine whether an interactive element of the user interface is responsive to an attention (e.g., gaze) input.

In some embodiments, the software may be downloaded to the controller 110 in electronic form, over a network, for example, or may alternatively be provided on tangible non-transitory media, such as optical, magnetic, or electronic memory media. In some embodiments, database 408 is also stored in a memory associated with controller 110. Alternatively or in addition, some or all of the described functions of the computer may be implemented in dedicated hardware, such as a custom or semi-custom integrated circuit or a programmable Digital Signal Processor (DSP). Although the controller 110 is shown in fig. 4, for example, as a separate unit from the image sensor 404, some or all of the processing functions of the controller may be performed by a suitable microprocessor and software or by dedicated circuitry within the housing of the image sensor 404 (e.g., a hand tracking device) or other devices associated with the image sensor 404. In some embodiments, at least some of these processing functions may be performed by a suitable processor integrated with display generation component 120 (e.g., in a television receiver, handheld device, or head mounted device) or with any other suitable computerized device (such as a game console or media player). The sensing functionality of the image sensor 404 may likewise be integrated into a computer or other computerized device to be controlled by the sensor output.

Fig. 4 also includes a schematic diagram of a depth map 410 captured by the image sensor 404, according to some embodiments. As described above, the depth map comprises a matrix of pixels having corresponding depth values. Pixels 412 corresponding to the hand 406 have been segmented from the background and wrist in the map. The brightness of each pixel within the depth map 410 is inversely proportional to its depth value (i.e., the measured z-distance from the image sensor 404), where the gray shade becomes darker with increasing depth. The controller 110 processes these depth values to identify and segment components of the image (i.e., a set of adjacent pixels) that have human hand features. These features may include, for example, overall size, shape, and frame-to-frame motion from a sequence of depth maps.

Fig. 4 also schematically illustrates the hand bones 414 that the controller 110 eventually extracts from the depth map 410 of the hand 406, according to some embodiments. In fig. 4, the hand skeleton 414 is superimposed over the hand background 416 that has been segmented from the original depth map. In some embodiments, key feature points of the hand and optionally on the wrist or arm connected to the hand (e.g., points corresponding to knuckles, finger tips, palm centers, ends of the hand connected to the wrist, etc.) are identified and located on the hand bones 414. In some embodiments, the controller 110 uses the positions and movements of these key feature points on the plurality of image frames to determine a gesture performed by the hand or a current state of the hand according to some embodiments.

Fig. 5 illustrates an exemplary embodiment of the eye tracking device 130 (fig. 1). In some embodiments, eye tracking device 130 is controlled by eye tracking unit 243 (fig. 2) to track the positioning and movement of the user gaze relative to scene 105 or relative to XR content displayed via display generation component 120. In some embodiments, the eye tracking device 130 is integrated with the display generation component 120. For example, in some embodiments, when display generating component 120 is a head-mounted device (such as a headset, helmet, goggles, or glasses) or a handheld device placed in a wearable frame, the head-mounted device includes both components that generate XR content for viewing by a user and components for tracking the user's gaze with respect to the XR content. In some embodiments, the eye tracking device 130 is separate from the display generation component 120. For example, when the display generating component is a handheld device or an XR chamber, the eye tracking device 130 is optionally a device separate from the handheld device or XR chamber. In some embodiments, the eye tracking device 130 is a head mounted device or a portion of a head mounted device. In some embodiments, the head-mounted eye tracking device 130 is optionally used in combination with a display generating component that is also head-mounted or a display generating component that is not head-mounted. In some embodiments, the eye tracking device 130 is not a head mounted device and is optionally used in conjunction with a head mounted display generating component. In some embodiments, the eye tracking device 130 is not a head mounted device and optionally is part of a non-head mounted display generating component.

In some embodiments, the display generation component 120 uses a display mechanism (e.g., a left near-eye display panel and a right near-eye display panel) to display frames including left and right images in front of the user's eyes, thereby providing a 3D virtual view to the user. For example, the head mounted display generating component may include left and right optical lenses (referred to herein as eye lenses) located between the display and the user's eyes. In some embodiments, the display generation component may include or be coupled to one or more external cameras that capture video of the user's environment for display. In some embodiments, the head mounted display generating component may have a transparent or translucent display and the virtual object is displayed on the transparent or translucent display through which the user may directly view the physical environment. In some embodiments, the display generation component projects the virtual object into the physical environment. The virtual object may be projected, for example, on a physical surface or as a hologram, such that an individual uses the system to observe the virtual object superimposed over the physical environment. In this case, separate display panels and image frames for the left and right eyes may not be required.

As shown in fig. 5, in some embodiments, the eye tracking device 130 (e.g., a gaze tracking device) includes at least one eye tracking camera (e.g., an Infrared (IR) or Near Infrared (NIR) camera) and an illumination source (e.g., an IR or NIR light source, such as an array or ring of LEDs) that emits light (e.g., IR or NIR light) toward the user's eye. The eye-tracking camera may be directed toward the user's eye to receive IR or NIR light reflected directly from the eye by the light source, or alternatively may be directed toward "hot" mirrors located between the user's eye and the display panel that reflect IR or NIR light from the eye to the eye-tracking camera while allowing visible light to pass through. The eye tracking device 130 optionally captures images of the user's eyes (e.g., as a video stream captured at 60-120 frames per second (fps)), analyzes the images to generate gaze tracking information, and communicates the gaze tracking information to the controller 110. In some embodiments, both eyes of the user are tracked separately by the respective eye tracking camera and illumination source. In some embodiments, only one eye of the user is tracked by the respective eye tracking camera and illumination source.

In some embodiments, the eye tracking device 130 is calibrated using a device-specific calibration process to determine parameters of the eye tracking device for the particular operating environment 100, such as 3D geometry and parameters of LEDs, cameras, hot mirrors (if present), eye lenses, and display screens. The device-specific calibration procedure may be performed at the factory or another facility prior to delivering the AR/VR equipment to the end user. The device-specific calibration process may be an automatic calibration process or a manual calibration process. According to some embodiments, the user-specific calibration process may include an estimation of eye parameters of a specific user, such as pupil position, foveal position, optical axis, visual axis, eye distance, etc. According to some embodiments, once the device-specific parameters and the user-specific parameters are determined for the eye-tracking device 130, the images captured by the eye-tracking camera may be processed using a flash-assist method to determine the current visual axis and gaze point of the user relative to the display.

As shown in fig. 5, the eye tracking device 130 (e.g., 130A or 130B) includes an eye lens 520 and a gaze tracking system including at least one eye tracking camera 540 (e.g., an Infrared (IR) or Near Infrared (NIR) camera) positioned on a side of the user's face on which eye tracking is performed, and an illumination source 530 (e.g., an IR or NIR light source such as an array or ring of NIR Light Emitting Diodes (LEDs)) that emits light (e.g., IR or NIR light) toward the user's eyes 592. The eye-tracking camera 540 may be directed toward a mirror 550 (which reflects IR or NIR light from the eye 592 while allowing visible light to pass) located between the user's eye 592 and the display 510 (e.g., left or right display panel of a head-mounted display, or display of a handheld device, projector, etc.) (e.g., as shown in the top portion of fig. 5), or alternatively may be directed toward the user's eye 592 to receive reflected IR or NIR light from the eye 592 (e.g., as shown in the bottom portion of fig. 5).

In some implementations, the controller 110 renders AR or VR frames 562 (e.g., left and right frames for left and right display panels) and provides the frames 562 to the display 510. The controller 110 uses the gaze tracking input 542 from the eye tracking camera 540 for various purposes, such as for processing the frames 562 for display. The controller 110 optionally estimates the gaze point of the user on the display 510 based on gaze tracking input 542 acquired from the eye tracking camera 540 using a flash assist method or other suitable method. The gaze point estimated from the gaze tracking input 542 is optionally used to determine the direction in which the user is currently looking.

Several possible use cases of the current gaze direction of the user are described below and are not intended to be limiting. As an exemplary use case, the controller 110 may render virtual content differently based on the determined direction of the user's gaze. For example, the controller 110 may generate virtual content in a foveal region determined according to a current gaze direction of the user at a higher resolution than in a peripheral region. As another example, the controller may position or move virtual content in the view based at least in part on the user's current gaze direction. As another example, the controller may display particular virtual content in the view based at least in part on the user's current gaze direction. As another exemplary use case in an AR application, the controller 110 may direct an external camera used to capture the physical environment of the XR experience to focus in the determined direction. The autofocus mechanism of the external camera may then focus on an object or surface in the environment that the user is currently looking at on display 510. As another example use case, the eye lens 520 may be a focusable lens, and the controller uses the gaze tracking information to adjust the focus of the eye lens 520 such that the virtual object the user is currently looking at has the appropriate vergence to match the convergence of the user's eyes 592. The controller 110 may utilize the gaze tracking information to direct the eye lens 520 to adjust the focus such that the approaching object the user is looking at appears at the correct distance.

In some embodiments, the eye tracking device is part of a head mounted device that includes a display (e.g., display 510), two eye lenses (e.g., eye lens 520), an eye tracking camera (e.g., eye tracking camera 540), and a light source (e.g., light source 530 (e.g., IR or NIR LED)) mounted in a wearable housing. The light source emits light (e.g., IR or NIR light) toward the user's eye 592. In some embodiments, the light sources may be arranged in a ring or circle around each of the lenses, as shown in fig. 5. In some embodiments, for example, eight light sources 530 (e.g., LEDs) are arranged around each lens 520. However, more or fewer light sources 530 may be used, and other arrangements and locations of light sources 530 may be used.

In some implementations, the display 510 emits light in the visible range and does not emit light in the IR or NIR range, and thus does not introduce noise in the gaze tracking system. Note that the position and angle of the eye tracking camera 540 is given by way of example and is not intended to be limiting. In some implementations, a single eye tracking camera 540 is located on each side of the user's face. In some implementations, two or more NIR cameras 540 may be used on each side of the user's face. In some implementations, a camera 540 with a wider field of view (FOV) and a camera 540 with a narrower FOV may be used on each side of the user's face. In some implementations, a camera 540 operating at one wavelength (e.g., 850 nm) and a camera 540 operating at a different wavelength (e.g., 940 nm) may be used on each side of the user's face.

The embodiment of the gaze tracking system as shown in fig. 5 may be used, for example, in computer-generated reality, virtual reality, and/or mixed reality applications to provide a user with a computer-generated reality, virtual reality, augmented reality, and/or augmented virtual experience.

Fig. 6 illustrates a flash-assisted gaze tracking pipeline in accordance with some embodiments. In some embodiments, the gaze tracking pipeline is implemented by a glint-assisted gaze tracking system (e.g., eye tracking device 130 as shown in fig. 1 and 5). The flash-assisted gaze tracking system may maintain a tracking state. Initially, the tracking state is off or "no". When in the tracking state, the glint-assisted gaze tracking system uses previous information from a previous frame when analyzing the current frame to track pupil contours and glints in the current frame. When not in the tracking state, the glint-assisted gaze tracking system attempts to detect pupils and glints in the current frame and, if successful, initializes the tracking state to "yes" and continues with the next frame in the tracking state.

As shown in fig. 6, the gaze tracking camera may capture left and right images of the left and right eyes of the user. The captured image is then input to the gaze tracking pipeline for processing beginning at 610. As indicated by the arrow returning to element 600, the gaze tracking system may continue to capture images of the user's eyes, for example, at a rate of 60 to 120 frames per second. In some embodiments, each set of captured images may be input to a pipeline for processing. However, in some embodiments or under some conditions, not all captured frames are pipelined.

At 610, for the currently captured image, if the tracking state is yes, the method proceeds to element 640. At 610, if the tracking state is no, the image is analyzed to detect a user's pupil and glints in the image, as indicated at 620. At 630, if the pupil and glints are successfully detected, the method proceeds to element 640. Otherwise, the method returns to element 610 to process the next image of the user's eye.

At 640, if proceeding from element 610, the current frame is analyzed to track pupils and glints based in part on previous information from the previous frame. At 640, if proceeding from element 630, a tracking state is initialized based on the pupil and flash detected in the current frame. The results of the processing at element 640 are checked to verify that the results of the tracking or detection may be trusted. For example, the results may be checked to determine if the pupil and a sufficient number of flashes for performing gaze estimation are successfully tracked or detected in the current frame. If the result is unlikely to be authentic at 650, then the tracking state is set to no at element 660 and the method returns to element 610 to process the next image of the user's eye. At 650, if the result is trusted, the method proceeds to element 670. At 670, the tracking state is set to YES (if not already YES), and pupil and glint information is passed to element 680 to estimate the gaze point of the user.

Fig. 6 is intended to serve as one example of an eye tracking technique that may be used in a particular implementation. As will be appreciated by one of ordinary skill in the art, other eye tracking techniques, currently existing or developed in the future, may be used in place of or in combination with the glint-assisted eye tracking techniques described herein in computer system 101 for providing an XR experience to a user, according to various embodiments.

In this disclosure, various input methods are described with respect to interactions with a computer system. When one input device or input method is used to provide an example and another input device or input method is used to provide another example, it should be understood that each example may be compatible with and optionally utilize the input device or input method described with respect to the other example. Similarly, various output methods are described with respect to interactions with a computer system. When one output device or output method is used to provide an example and another output device or output method is used to provide another example, it should be understood that each example may be compatible with and optionally utilize the output device or output method described with respect to the other example. Similarly, the various methods are described with respect to interactions with a virtual environment or mixed reality environment through a computer system. When examples are provided using interactions with a virtual environment, and another example is provided using a mixed reality environment, it should be understood that each example may be compatible with and optionally utilize the methods described with respect to the other example. Thus, the present disclosure discloses embodiments that are combinations of features of multiple examples, without the need to list all features of the embodiments in detail in the description of each example embodiment.

User interface and associated process

Attention is now directed to embodiments of a user interface ("UI") and associated processes that may be implemented on a computer system, such as a portable multifunction device or a head-mounted device, that communicates with a display generating component and with an external computer system associated with a first user.

Fig. 7A to 7I illustrate examples of displaying a visual indication of a portion of a user. FIG. 8 is a flow chart of an exemplary method 800 for displaying a visual indication of a portion of a user. The user interfaces in fig. 7A-7I are used to illustrate the processes described below, including the process in fig. 8.

Fig. 7A shows a first electronic device 700 (e.g., "John's tablet") displaying a communication user interface 702 on a display 704a and a second electronic device 704 (e.g., "Jane's tablet") displaying a communication user interface 706 on the display 700 a. Further, fig. 7A illustrates a first physical environment 708 that is using a first electronic device 700 and/or a first user 710 associated with the first electronic device (e.g., "John") and a second physical environment 712 that is using a second electronic device 704 and/or a second user 714 associated with the second electronic device (e.g., "Jane"). The first user 710 is at a first location 709a within the first physical environment 708 and the second user 714 is at a first location 713a within the second physical environment 712. Further, at fig. 7A, a first electronic device 700 and a second electronic device 704 communicate with each other. In particular, the first user 710 and the second user 714 are participating in a real-time communication session (e.g., a video call, a virtual video call, a video conference, and/or a virtual video conference) via the first electronic device 700 and the second electronic device 704. In some embodiments, the real-time communication session includes actual image and/or video data transferred between electronic devices 700 and 704. In some implementations, the real-time communication session includes virtual representations of users 710 and 714 (e.g., avatars of users 710 and 714) that are generated and displayed based on data captured via one or more sensors (e.g., sensors 716a-716 c) in communication with electronic devices 700 and 704.

At fig. 7A, the communication user interface 702 includes: a first participant zone 702a comprising a first representation 718 corresponding to a first user 710; and a second participant zone 702b that includes an augmented reality environment 715 (e.g., a virtual reality environment, an augmented reality environment, and/or a mixed reality environment) and/or a second representation 730 corresponding to a second user 714. Similarly, the communication user interface 706 includes: a third participant zone 706a comprising an augmented reality environment 715 and a first representation 718 corresponding to the first user 710; and a fourth participant zone 706b that includes a second representation 730 corresponding to the second user 714. The first electronic device 700 and the second electronic device 704 are configured to direct (e.g., transmit) sensor data, audio data, image data, and/or video data between each other such that the first user 710 and the second user 714 can communicate with each other via the first electronic device 700 and the second electronic device 704, respectively (e.g., the first physical environment 708 in which the first user 710 is located is remote from the second physical environment 712 in which the second user 714 is located). As described below, the first electronic device 700 and the second electronic device 704 display representations of the first user 710 and the second user 714 on the communication user interfaces 702 and 706, respectively. The displayed representations of the first user 710 and the second user 714 may include avatars of the first user 710 and the second user 714 generated by the first electronic device 700 and/or the second electronic device 704. In other words, the representations of the first user 710 and the second user 714 displayed by the first electronic device 700 and the second electronic device 704 are not actual images of the first user 710 and the second user 714 captured via the camera. Rather, the representations are virtual avatars generated based on data captured via one or more sensors (e.g., sensors 716a-716 c) in communication with the first electronic device 700 and/or the second electronic device 704. Thus, the representation of the user in the augmented reality environment optionally appears different from the appearance of the user in the physical environment.

At fig. 7A, the first physical environment 708 includes a first sensor 716a and a second sensor 716b, and the second physical environment 712 includes a third sensor 716c. The sensors 716a and 716b are in communication (e.g., wired and/or wireless communication) with the first electronic device 700, and the third sensor 716c is in communication (e.g., wired and/or wireless communication) with the second electronic device 704. In some implementations, the sensors 716a-716c include cameras, image sensors, light sensors, depth sensors, haptic sensors, orientation sensors, proximity sensors, temperature sensors, position sensors, motion sensors, and/or speed sensors. Sensors 716a and 716b are configured to capture data related to a state (e.g., position, location, orientation, posture, and/or position) of a first user 710 (e.g., the body of first user 710) within first physical environment 708, and sensor 716c is configured to capture data related to a state (e.g., position, location, orientation, posture, and/or position) of a second user 714 (e.g., the body of second user 714) within second physical environment 712.

Fig. 7A-7I illustrate exemplary user interfaces displayed on electronic devices 700 and 704 during a real-time communication session. While both electronic devices 700 and 704 are shown, the described examples primarily relate to sensors 716a and 716b capturing data indicative of a state of the first user 710 and the second electronic device 704 displaying and/or updating a first representation 718 of the first user 710 on the communication user interface 706 based on received information based on the data. It should be appreciated that in some examples, the first electronic device 700 operates in a similar manner as the second electronic device 704 based on received information based on data captured via the sensor 716c during a real-time communication session. Thus, in some examples, the first electronic device 700 displays representations of the second user 714 on the communication user interface 702 that are similar to those described below with reference to the second electronic device 704.

At fig. 7A, a first electronic device 700 receives data from sensors 716a and 716b and directs (e.g., transmits) information to a second electronic device 704 based on the data. The second electronic device 704 receives the information and generates a first representation 718 of the first user 710, which is displayed via the display 704a in the augmented reality environment 715 on the communication user interface 706. The second electronic device 704 generates a first representation 718 based on the received information. The data captured by the sensors 716a and 716b includes data related to the status of one or more body parts of the first user 710 within the first physical environment 708. In some embodiments, the data captured by the sensors 716a and 716b also includes data related to characteristics and/or physical characteristics of one or more body parts of the first user 710 (e.g., facial characteristics, hair color, eye color, clothing, size of body parts, and/or position of one or more body parts relative to each other). The second electronic device 704 uses the received information (which is based on the data) and generates a first representation 718 to include, at least in part, an appearance that mimics the actual state of the first user 710 within the first physical environment 708. Similarly, the first electronic device 700 receives information based on data captured via the sensor 716c (e.g., from the second electronic device 704 and/or via an external device such as a server), and generates a second representation 730 corresponding to the second user 714 based on the received information.

At fig. 7A, a first user 710 is located at a first location 709a in a first physical environment 708, where a first hand 710a and a second hand 710b are adjacent to a waist 710c of the body of the first user 710. Accordingly, the second electronic device 704 receives information indicative of the state of the first user 710 and generates and displays a first representation 718 at a first location 719a in the augmented reality environment 715 based on the information. In some embodiments, the first location 719a corresponds to the first location 709a. At fig. 7A, the first representation 718 includes a first hand 718a (e.g., a representation of the first hand 710 a) and a second hand 718b (e.g., a representation of the second hand 710 b) adjacent to the waist 718c (e.g., a representation of the waist 710 c). Thus, the second electronic device 704 displays the first representation 718 in the augmented reality environment 715 to mimic the state of the first user 710 in the first physical environment 708.

In some cases, the information received by the second electronic device 704 is based on indirect data indicative of the state of the one or more body parts of the first user 710 (e.g., data that does not correspond to the actual and/or accurate state of the one or more body parts of the first user 710). For example, in some embodiments, the sensors 716a and 716b capture direct data indicative of the state of the upper portion 711a of the body of the first user 710, but capture indirect data indicative of the state of the lower portion 711b of the body of the first user 710 (e.g., when the sensors 716a and/or 716b are not directed toward the lower portion 711b and/or are otherwise configured to capture data regarding the lower portion). In some embodiments, the indirect data includes extrapolated and/or inferred states of one or more portions of the body of first user 710, while the direct data includes captured and/or sensed states of one or more portions of the body of first user 710 (e.g., the direct data does not include extrapolated and/or inferred states of one or more portions of the body of first user 710). When the information received by the second electronic device 704 is based on indirect data, the second electronic device 704 displays the first visual indication 720 as part of the first representation 718 as an indication of a lower portion 711b of the body of the first user 710. At fig. 7A, the first visual indication 720 includes a shadow 720a that is displayed on a ground 722 of the augmented reality environment 715. In some implementations, the second electronic device 704 estimates (e.g., predicts based on the received information, extrapolates the received information, and/or approximates based on the received information) the state of the lower portion 711b and displays a visual indication 720 in the location of the augmented reality environment 715 based on the estimate. In some embodiments, the first electronic device 700 estimates the state of the lower portion 711b based on data captured via the sensor 716a and/or the sensor 716b, and directs (e.g., transmits) information including the estimated state of the lower portion 711b to the second electronic device 704. Thus, the second electronic device 704 displays the visual indication 720 at an estimated location based on indirect data indicative of the state of the upper portion 711b of the body of the first user 710 and/or based on direct data indicative of the state of the upper portion 711a of the body of the first user 710.

At fig. 7A, a first user 710 has a first leg 710d, a second leg 710e, a first foot 710f, and a second foot 710g included in a lower portion 711b of the body of the first user 710. In some implementations, when the information received by the second electronic device 704 is based on indirect data indicating the status of the first leg 710d, the second leg 710e, the first foot 710f, and the second foot 710g, the second electronic device 704 displays the visual indication 720 to represent these body parts of the first user 710 in the augmented reality environment 715. In some embodiments, the indirect data regarding the status of the first leg 710d, the second leg 710e, the first foot 710f, and the second foot 710g includes an estimate, approximation, and/or inference regarding the position and/or orientation of the first leg 710d, the second leg 710e, the first foot 710f, and the second foot 710g within the physical environment 708. In some such embodiments, the indirect data of the status of the first leg 710d, the second leg 710e, the first foot 710f, and the second foot 710g does not include direct sensor data that captures the actual positions and/or orientations of the first leg 710d, the second leg 710e, the first foot 710f, and the second foot 710g within the physical environment 708. Thus, the second user 714 can view the communication user interface 706 and obtain an understanding of where these body parts can be positioned relative to other portions of the first representation 718 (e.g., the first hand 718a, the second hand 718b, and/or the waist 718 c) based on the location of the visual indication 720. In some implementations, the visual indication 720 represents a portion of the first representation 718 that is not clearly displayed by the second electronic device 704. In some implementations, the visual indication 720 represents a portion of the first representation 718 that is not displayed via the second electronic device 704.

In some implementations, when the second electronic device 704 receives information based on indirect data indicative of a state of one or more body parts of the first user 710, the second electronic device 704 displays (e.g., in addition to and/or in lieu of the visual indication 720) a second visual indication 724 of one or more portions of the body of the first user 710 as part of the first representation 718. For example, at fig. 7A, the second electronic device 704 displays a second visual indication 724 (e.g., indicated by the dashed line at fig. 7A) representing the first leg 710d, the second leg 710e, the first foot 710f, and the second foot 710 g. As shown in fig. 7A, the second visual indication 724 includes dashed lines to represent the first leg 718d, the second leg 718e, the first foot 718f, and the second foot 718g of the first representation 718. In some embodiments, the second visual indication 724 does not include an anatomically accurate depiction of one or more parts of the body of the first user 710 (e.g., the second visual indication 724 does not include a clear representation of the first leg 718d, the second leg 718e, the first foot 718f, and the second foot 718 g). In some implementations, the second visual indication 724 includes a blur extension of the first representation 718. For example, the second visual indication 724 may include a spherical object including a non-zero amount of blur, the spherical object representing one or more of the first leg 710d, the second leg 710e, the first foot 710f, and/or the second foot 710g of the first user 710. … in some implementations, the second electronic device 704 estimates (e.g., predicts based on, extrapolates, and/or approximates) the location of the second visual indication 724 to display the second visual indication 724 at a location in the augmented reality environment 715. In some embodiments, the first electronic device 700 estimates the state of the lower portion 711b based on data captured via the sensor 716a and/or the sensor 716b, and directs (e.g., transmits) information including the estimated state of the lower portion 711b to the second electronic device 704. Thus, the second user 714 can view the communication user interface 706 and obtain an understanding of where the one or more body parts of the first user 710 can be positioned relative to the rest of the first representation 718 based on the location of the second visual indication 724.

In some implementations, the augmented reality environment 715 is a virtual environment that is different from and/or not based on captured data indicative of features of the first physical environment 708. In some such implementations, the ground 722 is a virtual ground that is different from and/or not based on captured data indicative of characteristics of the ground of the first physical environment 708. In some implementations, the augmented reality environment 715 includes one or more objects (e.g., virtual objects, representations of physical objects in the first physical environment 708, and/or images of physical objects in the first physical environment 708). The second electronic device 704 is configured to display a visual indication 720 and/or a second visual indication 724 in association with one or more objects within the augmented reality environment 715. In other words, the second electronic device 704 may adjust the size, shape, and/or location of the visual indication 720 and/or the second visual indication 724 based on the respective locations of one or more objects displayed in the augmented reality environment 715.

Further, the second electronic device 704 is configured to adjust and/or modify the display of the first representation 718 including the visual indication 720 and/or the second visual indication 724 based on information indicative of the status of the first user 710 (e.g., information received from the first electronic device 700 based on data captured via the sensors 716a and 716 b). For example, in response to detecting movement of the first user 710, the first electronic device 700 directs (e.g., transmits) information to the second electronic device 704 indicating movement and/or new locations of the first user 710 within the first physical environment 708. In response to receiving the information, the second electronic device 704 displays the first representation 718, the visual indication 720, and/or the second visual indication 724 at a second location 719B on the third participant region 706a, as shown in fig. 7B.

At fig. 7B, first user 710 has moved (e.g., walked) to a second location 709B within first physical environment 708, where second location 709B is different from first location 709a. In response to receiving the information indicative of the movement (e.g., from the first electronic device 700 and/or via an external device such as a server), the second electronic device 704 displays a first representation 718 at a second location 719b within the augmented reality environment 708 to reflect the actual movement of the first user 710 within the first physical environment 715. Further, at fig. 7B, the second electronic device 704 displays a visual indication 720 to also reflect the movement of the first user 710. For example, at fig. 7B, the visual indication 720 is positioned partially between the waist 718c of the first representation 718 and the ground 722 of the augmented reality environment 715. Accordingly, the second electronic device 704 displays a visual indication 720 at an updated location within the augmented reality environment 715 in response to receiving information indicating that the first user 710 moved and/or has changed location within the first physical environment 708.

Further, as shown in fig. 7A and 7B, the second user 714 does not move within the second physical environment 712, but remains at the location 713 a. Thus, the first electronic device 700 receives information indicative of the status of the second user 714 (e.g., from the second electronic device 704 and/or via an external device such as a server), and in response to receiving the information, maintains a display of the second representation 730 having the same status (e.g., the same status shown in fig. 7A).

At fig. 7B, a first portion 720B of the visual indication 720 is located between a waist 718c of the first representation 718 and a ground 722 of the augmented reality environment 715. The second portion 720c of the visual indication 720 extends beyond the second location 709b (e.g., appears to be behind the second location 709 b) in the augmented reality environment 715 to reflect the movement of the first user 710. For example, the second electronic device 704 displays the second portion 720c as a shadow of the first representation 718 within the third participant region 706 a.

As described below, in some embodiments, the second electronic device 704 displays the second portion 720c as extending beyond the second location 709b based on illumination (e.g., a representation of virtual illumination and/or actual illumination in the first physical environment 708 and/or the second physical environment 712) included and/or displayed in the augmented reality environment 715. For example, returning to fig. 7A, illumination of the augmented reality environment 715 is emitted from a source located at a position above the first representation 718 relative to the ground 722. Thus, when the first representation 718 is displayed at the first position 709a, the visual indication 720 includes an elliptical shape under the first representation 718. When the second electronic device 704 displays the first representation 718 at the second location 709b, the second electronic device 704 displays the visual indication 720 with the second portion 720c to reflect that the first representation 718 is now displayed at the second location 709b, which is no longer under the light source of the augmented reality environment 715 (e.g., the light source emits light that is blocked by at least a portion of the first representation 718 such that the second portion 720c of the visual indication 720 extends beyond the second location 709 b).

At fig. 7B, the second portion 720c includes a shape and/or appearance that mimics the state of the first representation 718. For example, the second portion 720c includes a generally linear protrusion 720c extending from the first portion 720b that reflects that the first hand 718a (e.g., a representation of the first hand 710 a) and the second hand 718b (e.g., a representation of the second hand 710 b) are positioned adjacent to the waist 718c (e.g., a representation of the waist 710 c) of the first representation 718. In response to receiving the information indicative of the movement of the body part of the first user 710, the second electronic device 704 is configured to display the visual indication 720 and/or portions 720b and 720c of the visual indication 720 to include an appearance (e.g., shape) determined based on the information.

At fig. 7C, second electronic device 704 receives information indicating a change in the location of one or more body parts of first user 710 within first physical environment 708. For example, at fig. 7C, the first user has moved the first hand 710a and the second hand 710b to a position away from and above the waist 710C of the first user 710 (e.g., relative to the ground of the first physical environment 708). Further, the first user 710 has bent the first and second legs 710d, 710e to squat slightly (e.g., as compared to the position and/or posture of the first user 710 shown in fig. 7B). The second electronic device 704 receives information indicating movement of the first hand 710a, the second hand 710b, the first leg 710d, and the second leg 710e and displays a first representation 718 having an appearance shown in fig. 7C. At fig. 7C, the first representation 718 includes a first hand 718a and a second hand 718b that are positioned away from and above the waist 718C. In addition, the first representation 718 includes a first leg 718d and a second leg 718e that are curved and in a slightly squat position. In some implementations, the second electronic device 704 does not display the first leg 718d and/or the second leg 718e of the first representation 718, but rather displays a visual indication 720 and/or a second visual indication 724 to indicate the first leg 718d and/or the second leg 718e. Further, the position, orientation, and/or pose of the first leg 718d and/or the second leg 718e may be inferred based on the position, orientation, and/or pose of the remainder of the first representation 718. Thus, the second electronic device 704 is configured to display the first representation 718 to mimic movement of portions of the body of the first user 710 even when the first user 710 remains substantially stationary relative to the ground of the first physical environment 708 (e.g., the first user 710 remains at the location 709 b).

At fig. 7C, the second electronic device 704 also displays a visual indication 720 to include an updated appearance based on information indicating movement of the first hand 710a, the second hand 710b, the first leg 710d, and the second leg 710e of the first user 710. As shown in fig. 7C, the visual indication 720 includes a third portion 720d to represent a change in the position of the first hand 710a that has moved away from and over the waist 710C. The third portion 720d includes an appearance and shape that reflects the shape and appearance of the first hand 710a (and arm 710 h). Further, the second electronic device 704 enlarges the second portion 720c of the visual representation 720 to indicate and/or infer that the first and second legs 710d, 710e are curved (e.g., when compared to the second portion 720B shown in fig. 7B, the first and second legs 710d, 710e are substantially straight).

At fig. 7C, the second electronic device 704 also displays a table 726 (e.g., a first object) and a chair 728 (e.g., a second object) in the augmented reality environment 715. In some implementations, the second electronic device 704 displays the table 726 and the chair 728 in response to receiving information indicating that the first user 710 has approached (e.g., moved to) a physical object within the first physical environment 708. In some implementations, the second electronic device 704 displays the table 726 and the chair 728 in response to one or more user inputs requesting the second electronic device 704 to display one or more objects in the augmented reality environment 715 (e.g., virtual objects that are not based on physical objects in the first physical environment 708). In some implementations, the table 726 and/or chair 728 are representations of physical objects in the first physical environment 708 (e.g., representations based on actual image and/or video data). In some implementations, the table 726 and/or chair 728 are virtual representations of physical objects in the first physical environment 708 (e.g., virtual representations that are not based on actual image and/or video data). In some implementations, the table 726 and/or the chair 728 are virtual objects that are not based on physical objects in the first physical environment 708, but are based on only a portion of the augmented reality environment 715.

At fig. 7C, the second electronic device 704 displays a table 726 including a visual indication 726a (e.g., a shadow) and a chair 728 including a visual indication 728a (e.g., a shadow). In some embodiments, the second electronic device 704 does not display the visual indication 726a and/or the visual indication 728a.

As described above, the second electronic device 704 determines the shape and/or appearance of the visual indication 720 based on illumination of the augmented reality environment 715, which may include a representation of physical illumination (e.g., a representation of light emitted from actual light sources located in the first physical environment 708 and/or the second physical environment 712) and/or virtual illumination (e.g., light that does not represent light emitted from actual light sources located in the first physical environment 708 and/or the second physical environment 712). At fig. 7C, the augmented reality environment 715 includes an indication of light 732 directed from a light source at an angle 734 relative to the ground 722 of the augmented reality environment 715. The second electronic device 704 determines the shape and/or appearance of the visual indication 720, the visual indication 726a, and/or the visual indication 728a based on the light 732 and/or the angle 734 of the light 732. Thus, the second electronic device 704 displays the visual indication 720, the visual indication 726a, and/or the visual indication 728a to appear as shadows of the first representation 718, the table 726, and the chair 728, respectively, caused by and/or to be caused by the light 732.

At fig. 7D, the second electronic device 704 determines that the angle at which the light 732 emitted from the light source has changed in the augmented reality environment 715. As shown in fig. 7D, light 732 is emitted at an angle 736 relative to the ground 722 of the augmented reality environment 715, and the angle 736 is different from the angle 734. In some implementations, the second electronic device 704 determines that the angle has changed based on information received from the first electronic device 700 indicating light emitted by a physical light source within the first physical environment 708. In some implementations, the second electronic device 704 determines that the angle has changed based on the passage of time (e.g., the second electronic device 704 displays the light 732 as virtual light that changes over time during the real-time communication session).

At fig. 7D, the second electronic device 704 adjusts the appearance of the visual indication 720 based on determining that the angle of the emitted light 732 has changed (e.g., from angle 734 to angle 736). For example, the visual indication 720 includes a longer second portion 720c extending from the first portion 720b based on determining that the angle 736 is less than the angle 734 (e.g., relative to the ground 722 of the augmented reality environment 715). Further, the second electronic device 704 does not display the third portion 720d of the visual indication 720 because the third portion 720d extends beyond the boundary (e.g., edge and/or frame) of the third participant zone 706 a. The second electronic device 704 also displays a fourth portion 720e of the visual indication 720 that represents the second hand 718b of the first representation 718 (and/or the second hand 710b of the first user 710). At fig. 7C, the second electronic device 704 does not display a fourth portion 720e of the visual indication 720 because the fourth portion 720e is positioned behind the first representation 718 and thus is not visible within the augmented reality environment 715 (e.g., from the perspective of the second user 714 viewing the second electronic device 704). At fig. 7D, the second electronic device 704 displays a fourth portion 720e of the visual indication 720 based at least in part on the determination of the angle 736. Accordingly, the second electronic device 704 is configured to adjust the appearance, shape, and/or location of the visual indication 720 based on the light 732 included and/or displayed in the augmented reality environment 715.

At fig. 7D, the second electronic device 704 also adjusts the appearance of the visual indication 726a of the table 726 and the visual indication 728a of the chair 728 based on the angle 736 of the light 732. The visual indication 726a extends further to the left of the table 726 (e.g., relative to the ground 722 of the augmented reality environment 715) and the visual indication 728a extends further to the left of the chair 728 (e.g., relative to the ground 722 of the augmented reality environment 715). Thus, in some implementations, the second electronic device 704 also adjusts the appearance of the visual indication 726a and the visual indication 728a based on the light 732 included and/or displayed in the augmented reality environment 715.

The second electronic device 704 is further configured to adjust the appearance of the visual indication 720 and/or display additional visual indications in response to receiving an indication that the first user 710 interacted with one or more objects in the first physical environment 708. For example, at fig. 7E, the second electronic device 704 receives an indication that the first user 710 is in a sitting position (e.g., receives information from the first electronic device 700 based on data captured via the sensors 716a and/or 716 b). In response to receiving the indication that the first user 710 is in the sitting position, the second electronic device 704 adjusts the appearance of the visual indication 720 to include a chair 738, as shown in fig. 7E. At fig. 7E, the visual indication 720 includes a chair 738 and does not include the shadows 720a shown in fig. 7A-7D. In some embodiments, the visual indication 720 includes both a chair 738 and a shadow 720a.

In some embodiments, the chair 738 includes a representation (e.g., a representation based on actual image and/or video data) of a physical chair and/or other object supporting the first user 710 in a seated position in the first physical environment 708. In some embodiments, the chair 738 includes a virtual representation (e.g., a virtual representation that is not based on actual image and/or video data) of a physical chair and/or other objects supporting the first user 710 in a seated position in the first physical environment 708. In some embodiments, the chair 738 is a virtual object that is not based on a physical chair and/or other object that supports the first user 710 in a seated position in the first physical environment 708, but rather is part of the augmented reality environment 715 instead of the physical environment 708. In some embodiments, the chair 738 is a parameterized chair that the second electronic device 704 displays as a two-dimensional virtual object having a chair-like appearance, where the parameterized chair is not based on physical chairs and/or other objects in the first physical environment 708.

At fig. 7F, the second electronic device 704 receives an indication (e.g., receives information from the first electronic device 700 based on data captured via the sensors 716a and/or 716 b) that the first user 710 has contacted (e.g., touched and/or picked up with the second hand 710 b) a water bottle 740 (e.g., a physical water bottle) located in the first physical environment 708. In response to receiving an indication that the first user 710 has contacted the water bottle 740 (e.g., the body part of the first user 710 is within a predetermined distance of the water bottle 740), the second electronic device 704 displays a water bottle representation 742 corresponding to the water bottle in the augmented reality environment 715. In some embodiments, the water bottle representation 742 is based on actual image and/or video data of the water bottle 740 in the first physical environment 708. In some embodiments, the water bottle representation 742 is not based on actual image and/or video data of the water bottle 740 in the first physical environment 708. In some embodiments, the water bottle representation 742 is a virtual object that does not represent the water bottle 740 (e.g., the water bottle representation 742 includes a cup instead of a water bottle).

Further, the second electronic device 704 displays a visual indication 742a in the augmented reality environment 715 in response to receiving an indication that the first user 710 has contacted the water bottle 740. At fig. 7F, the visual indication 742a includes highlighting and/or shading around the perimeter of the water bottle representation 742 to indicate to the second user 714 (e.g., a user viewing the second electronic device 704) that the first user 710 is contacting an object in the first physical environment 708.

At fig. 7G, the second electronic device 704 receives an indication (e.g., information received from the first electronic device 700 based on data captured via the sensors 716a and/or 716 b) that the first user 710 has contacted a table 744 (e.g., a physical table) located in the first physical environment 708 (e.g., touching the table and/or placing the first hand 710a on the table). In response to receiving an indication that first user 710 has contacted table 744 (e.g., a body part of first user 710 is within a predetermined distance of the table), second electronic device 704 displays a table 726 corresponding to table 744 in augmented reality environment 715. In some embodiments, table 726 is based on actual image and/or video data of table 744 in first physical environment 708. In some embodiments, table 726 is not based on actual image and/or video data of table 744 in first physical environment 708. In some embodiments, table 726 is a virtual object that does not represent table 744 (e.g., table 726 includes a square surface instead of a circular surface).

Further, the second electronic device 704 displays a visual indication 726b in the augmented reality environment 715 in response to receiving an indication that the first user 710 has contacted the table 744. At fig. 7F, the visual indication 726b includes a highlighting and/or shading display around a portion of the surface of the table 726 that the first representation 718 is contacting (e.g., the first hand 718a is contacting) such that the second electronic device 704 displays a visual indication to the second user 714 (e.g., a user viewing the second electronic device 704) that the first user 710 is contacting an object in the first physical environment 708.

At fig. 7H, the first user 710 has returned to a first location 709a within the first physical environment 708, and the second electronic device 704 displays a first representation 718 and a visual indication 720 based on information indicating the current state of the first user 710, as discussed above. The first electronic device 700 is further configured to adjust an appearance of a second representation 730 of a second user 714 within the augmented reality environment 715 displayed on the second participant region 702b of the first electronic device 700. At fig. 7H, the first user 714 is in position 713a and in a standing position such that the head 714a of the first user 714 is in a first position 746a relative to the ground 712a of the second physical environment 712. In response to receiving the information corresponding to the status of the second user 714, the first electronic device 700 displays a head 730a of the second representation 730 at a first location 748a relative to the ground 722 of the augmented reality environment 715 to reflect that the second user 714 is in a standing position in the second physical environment 712. At fig. 7H, the first electronic device 700 displays a second representation 730 of a second user 714 having a visual indication 754 (e.g., shading). In response to receiving the information indicating that the state of the second user 714 has changed within the second physical environment 712, the first electronic device 700 adjusts the appearance of the second representation 730 and/or the visual indication 754 displayed in the augmented reality environment 715.

At fig. 7I, the first electronic device 700 receives information indicating that the second user 714 has moved from the standing position to the bow-step position. For example, the first user 714 is in a bow-step position and squats down and/or is closer to the ground 712a of the second physical environment 712 than in a standing position. Thus, the head 714a of the first user 714 is in the second position 746b relative to the ground 712a of the second physical environment 712. Further, the first hand 714b and the second hand 714c of the second user 714 are lifted and unfolded from the waist 714d of the second user 714. The first leg 714e of the second user 714 also bends and the second leg 714f of the second user 714 extends away from the waist 714d to form an arcade pose.

In response to receiving the information indicating that the second user 714 has moved from the standing position to the bow-step position, the first electronic device 700 adjusts the appearance of the second representation 730 in the augmented reality environment 715. At fig. 7I, the second representation 730 includes a head 730b at a second position 748b relative to the ground 722 of the augmented reality environment 715, wherein the second position 748b is lower than the first position 748a relative to the ground 722 to reflect that the second user 714 is in the bow step position. The first hand 730b and the second hand 730c are lifted up and spread from the waist portion 730d of the second image 730. In some implementations, the first electronic device 700 receives information based on direct data corresponding to the state of the head 714a, the first arm 714b, the second arm 714c, and/or the waist 714d of the second user 714. Accordingly, the first electronic device 700 displays a second representation 730 in which the locations of the head 730a, the first hand 730b, the second hand 730c, and the waist 730d within the augmented reality environment 715 mimic the locations of the head 714a, the first hand 714b, the second hand 714c, and the waist 714d of the second user 714 within the second physical environment 712.

In addition, the first electronic device 700 adjusts the appearance of the visual indication 754 to include a first foot representation 754a and a second foot representation 754b, as shown in fig. 7I. In some implementations, the first electronic device 700 receives information based on indirect data corresponding to the state of the first leg 714e (e.g., including the first foot 714 g) and the second leg 714f (e.g., including the second foot 714 h) of the second user 714. In some implementations, when the first electronic device 700 receives information based on indirect data corresponding to the states of the first leg 714e and the second leg 714f, the first electronic device 700 is configured to estimate the location of the first foot representation 754a and the second foot representation 754b within the augmented reality environment 715 based at least in part on the information. In some implementations, the second electronic device 704 receives indirect data (e.g., via the sensor 716 c) corresponding to the states of the first leg 714e and the second leg 714f, estimates the locations of the first foot representation 754a and the second foot representation 754b within the augmented reality environment 715, and directs (e.g., transmits) information to the first electronic device 700 (e.g., the information includes the estimated locations of the first foot representation 754a and the second foot representation 754 b) based on the indirect data. At fig. 7I, the first electronic device 700 displays a first foot representation 754a and a second foot representation 754b at respective locations in the augmented reality environment 715 that are different from the actual locations of the first foot 714g and the second foot 714h of the second user 714 within the second physical environment 712. However, the estimated positions of the first and second foot representations 754a, 754b are configured to provide an indication of movement of the second user 714 between the standing position and the bow-step position to the first user 710 (e.g., a user viewing and/or using the first electronic device 700).

Additional description regarding fig. 7A-7I is provided below with reference to the method 800 described with respect to fig. 7A-7I.

Fig. 8 is a flowchart of an exemplary method 800 for displaying a visual indication of a portion of a user, according to some embodiments. In some embodiments, the method 800 is performed at a computer system (e.g., 101, 700, 704, and/or 944) that includes a display generating component (e.g., 120, 700a, 704a, and/or 944 a) (e.g., a visual output device, a 3D display, a display having at least a transparent or translucent portion on which an image may be projected (e.g., a see-through display), a projector, a heads-up display, a display controller) and an external computer system (e.g., 101, 700, 704, and/or 944) associated with a first user (e.g., being operated by the first user (e.g., a user that is conducting a communication session (e.g., augmented reality and/or video conferencing) with a user of the computer system). In some embodiments, the method 800 is managed by instructions stored in a non-transitory (or transitory) computer-readable storage medium and executed by one or more processors of a computer system (such as the one or more processors 202 of the computer system 101) (e.g., the control 110 in fig. 1). Some of the operations in method 800 are optionally combined and/or the order of some of the operations are optionally changed.

As described below, the method 800 provides an intuitive way for displaying a visual indication of a portion of a user. The method reduces the cognitive burden of users participating in real-time communication sessions, thereby creating a more efficient human-machine interface. For battery-powered computing devices, enabling a user to communicate with another participant of a real-time communication session faster and more efficiently saves power and increases the time interval between battery charges.

In response to receiving a request (802) (in some embodiments, to display a virtual avatar in the augmented reality environment in place of the first user) to display a representation (e.g., 718 and/or 730) (e.g., an avatar and/or virtual representation of at least a portion of the first user) of a first user (e.g., 710 and/or 714) (e.g., a user of an external computer system) in the augmented reality environment (e.g., such as the augmented reality environment) in a manner that tracks a subset of physical movements of the person or representations thereof and in response adjusts one or more characteristics of one or more virtual objects simulated in the augmented reality environment in accordance with at least one laws of physics) in place of the first user (e.g., a fully or partially simulated environment in which the person senses and/or interacts via an electronic system), the computer system (e.g., 101, 700, 704, and/or 944) displays (804) the representation (e.g., 710 and/or 714) (e.g., a user in the physical environment) (e.g., 718 and/or 730) (e.g., a virtual avatar of at least a portion of the first user) in the augmented reality environment via a display generating component (e.g., 120, 700a, 704 a).

The representation of the first user includes (in some embodiments, instead of the first user displaying a virtual avatar in the augmented reality environment) a visual indication (806) (e.g., 720, 726b, 738, 742a, 754a, and/or 754 b) of a portion of the first user's body (e.g., 710 and/or 714) in the augmented reality environment (e.g., 715) (e.g., a visual indication of at least one body part of the first user that is not an anatomically accurate representation of the at least one body part of the first user) (e.g., a body part of the first user below the circumference of the first user and/or other portions of the first user's body and/or body parts of the first user that are not directly tracked and/or detected by an external computer system and/or computer system).

The visual indication (808) (e.g., 720, 726b, 738, 742a, 754a, and/or 754 b) of the portion (e.g., 711 b) of the body of the first user (e.g., 710 and/or 714) has an appearance (e.g., displayed on, overlaid on, obscured by, moved relative to, and/or changed in size and/or appearance relative to one or more objects) (e.g., based on movement of the first user, movement of a computer system and/or display generating component, and/or a change in the augmented reality environment, the visual indication of the portion of the body changing relative to at least one visual element (e.g., a surface, a visual representation of an object, an avatar, and/or a representation of an entity and/or furniture) included and/or displayed within the augmented reality environment (e.g., a surface, a floor, a ceiling, a surface, a movement, a change in the position, a change in the shape, a change in the position, a change in the shape, a blur, a change in the position, a change in the shape, a change in the size, a change in the shape, a change in the size, and/or a change in the shape) determined based at least in the visual indication of the visual indication (e.g., the visual indication of the visual indication and/representation.

The visual indication (810) (e.g., 720, 726b, 738, 742a, 754a, and/or 754 b) of the portion (e.g., 711 b) of the body of the first user (e.g., 710 and/or 714) represents an estimated state (e.g., position, orientation, and/or pose) of the portion (e.g., 711 b) of the body (e.g., a position relative to a second portion of the body of the first user, a position relative to an augmented reality environment, and/or a position relative to a physical environment surrounding the first user) estimated based on indirect information about the state of the portion (e.g., 711 b) when the computer system (e.g., 101, 700, 704, and/or 944) is unable to obtain direct information about the state of the portion (e.g., 711 b) of the body. In some embodiments, the visual indication (e.g., 720, 726b, 738, 742a, 754a, and/or 754 b) of the portion (e.g., 711 b) of the body of the first user (e.g., 710 and/or 714) comprises a blurred representation of the portion (e.g., 711 b) of the body of the first user (e.g., 710 and/or 714), wherein the blurred representation is not an anatomically accurate depiction of the portion (e.g., 711 b) of the body of the first user (e.g., 710 and/or 714). In some implementations, the visual indication (e.g., 720, 726b, 738, 742a, 754a, and/or 754 b) of the portion (e.g., 711 b) of the body of the user (e.g., 710 and/or 714) is a shadow (e.g., 720 a) and/or a blurred visual element that is displayed in the augmented reality environment (e.g., 715) and that represents the portion (e.g., 711 b) of the body of the first user (e.g., 710 and/or 714). In some embodiments, the portion of the first user's body (e.g., 711 b) includes at least a lower portion of the first user's body (e.g., 711 b), such as a body part (e.g., leg, knee, ankle, foot, and/or hip) below the waist circumference (e.g., 710 c) of the first user (e.g., 710 and/or 714). In some implementations, the portion (e.g., 711 b) of the body of the first user (e.g., 710 and/or 714) includes a body part of the first user that is below a predetermined distance from a surface (e.g., 722) (e.g., floor) of the augmented reality environment (e.g., 715).

In some embodiments, the representation (e.g., 718 and/or 730) of the first user further comprises a visual representation (e.g., virtual representation of at least one body part of the first user, such as a head, face, shoulders, hands, arms, and/or torso) of the second portion (e.g., 711 a) of the body of the first user (e.g., 710 and/or 714), wherein the visual representation of the second portion of the body of the first user is generated based on data indicative of a position (e.g., relative to an external computer system and/or relative to a computer system) of the second portion of the body of the first user (e.g., 710 and/or 714). In some embodiments, the external computer system (e.g., 700 and/or 704) includes one or more sensors (e.g., 716a-716 c) configured to detect and/or track one or more body parts of the first user (e.g., 710 and/or 714). The external computer system (e.g., 700 and/or 704) generates and/or collects data indicative of the location of the second portion (e.g., 711 a) of the body of the first user (e.g., 710 and/or 714), and transmits the data (e.g., directly and/or indirectly via another external computer system such as a server) to the computer system (e.g., 101, 700, 704 and/or 944) such that the computer system (e.g., 101, 700, 704 and/or 944) generates a visual representation of the second portion (e.g., 711 a) of the body of the first user (e.g., 710 and/or 714) based on the data. In some embodiments, the visual representation of the second portion (e.g., 711 a) of the first user (e.g., 710 and/or 714) body includes a virtual representation of the physical features of the first user (e.g., 710 and/or 714), such as the facial features, head features, upper body features, clothing, and/or dimensions of the various body parts of the first user (e.g., 710 and/or 714). In some embodiments, the visual representation of the second portion (e.g., 711 a) of the body of the first user (e.g., 710 and/or 714) is configured to move as the first user (e.g., 710 and/or 714) moves one or more body parts (e.g., head, shoulders, arms, and/or hands) included in the second portion (e.g., 711 a) of the body. In some embodiments, the visual representation of the second portion (e.g., 711 a) of the body of the first user (e.g., 710 and/or 714) is an anatomically accurate representation and/or depiction of the body part of the second portion (e.g., 711 a) of the body of the first user (e.g., 710 and/or 714).

In some embodiments, while displaying a visual indication (e.g., 720, 726b, 738, 742a, 754a, and/or 754 b) of a representation (e.g., 718 and/or 730) of a first user (e.g., 710 and/or 714) including a movement of a portion of the first user's body (e.g., 710 and/or 714) including a movement of the portion of the first user's body (e.g., 711 b), a movement of the computer system (e.g., 720, 726b, 738, 742a, 754a, and/or 754 b) that causes a change in the position of the first user on the display generating component (e.g., a second user of the system to adjust the position of the display generating component (e.g., move and/or tilt the display generating component), causes the position of the first user to be adjusted relative to the display generating component) and/or a change in characteristics (e.g., change in the physical environment, e.g., the AR, the real-world, and/or the changing (e.g., 101 and/or the changing virtual environment) of the virtual environment (e.g., the virtual environment) and/or the changing (e.g., 101, the real-world environment) of the computer system (e.g., 700, the changing and/or the changing conditions of the virtual environment) and/or the changing (e.g., the virtual environment) of the portions (e.g., 101 and/or the changing conditions) of the computing system, 711b) Is a visual indication of movement (e.g., 720, 726b, 738, 742a, 754a, and/or 754 b) in an augmented reality environment (e.g., 715). In some implementations, movement of the visual indication (e.g., 720, 726b, 738, 742a, 754a, and/or 754 b) of the portion (e.g., 711 b) of the body of the first user (e.g., 710 and/or 714) in the augmented reality environment (e.g., 715) includes a transition (e.g., 720, 726b, 738, 742a, 754a, and/or 754 b) from displaying the visual indication (e.g., 720, 726b, 738, 742a, 754 b) of the portion (e.g., 711 b) of the body of the first user (e.g., 710 and/or 714) at a first location (e.g., 719a) in the augmented reality environment (e.g., 715) relative to a first location (e.g., 719 a) of one or more visual elements of the augmented reality environment to a second location (e.g., 720, 726b, 754a, 754b, and/or 754 b) of the body of the first user (e.g., 754 b) in the augmented reality environment.

Displaying a visual indication of a portion of the first user's body that represents an estimated state of the portion of the first user's body enables the computer system to provide a more complete state of the first user's body even when direct information about the portion of the body is not available, which provides improved visual feedback.

In some embodiments, the representation (e.g., 718 and/or 730) of the first user (e.g., 710 and/or 714) further includes a second visual indication (e.g., 724) of a second portion (e.g., 710d and/or 710 e) of the body of the first user (e.g., 710 and/or 714) (e.g., a second visual indication different from the visual indication and/or a second visual indication applies a first visual effect (e.g., a blur amount) to the first portion of the representation and a visual indication applies a second visual effect (e.g., a shadow) to the second portion of the representation) (e.g., a second portion of the same body as the body portion, a second portion of the body at least partially different from the body portion, and/or a second portion of the body that is entirely different from the body portion), and the second visual indication (e.g., 724) of the second portion (e.g., 710d and/or 710 e) of the body of the first user (e.g., 710 and/or 714) includes a second portion (e.g., 710d and/or 710 e) of the body of the first user (e) that is to blur the first portion of the body (e.g., 710 and/or 710 e) by a blur amount of the first user) compared to the first portion of the body. In some embodiments, the computer system (e.g., 101, 700, 704, and/or 944) does not receive direct information regarding the state of the second portion (e.g., 710d and/or 710 e) of the body of the first user (e.g., 710 and/or 714) such that the second visual indication (e.g., 724) includes an estimated state of the second portion (e.g., 710d and/or 710 e) of the body of the first user (e.g., 710 and/or 714).

Displaying the second visual indication of the second portion of the first user's body enables the computer system to provide a more complete state of the first user's body, which provides improved visual feedback.

In some implementations, the visual indication (e.g., 720, 726b, 738, 742a, 754a, and/or 754 b) of the portion (e.g., 711 b) of the body of the first user (e.g., 710 and/or 714) includes a shadow (e.g., 720 a) (e.g., a projection within the augmented reality environment that visually indicates a state of at least the portion of the body of the first user).

Displaying the visual indication of the portion of the first user's body as including shadows enables the computer system to provide a more complete state of the first user's body in a familiar manner, which provides improved visual feedback.

In some implementations, the shadow (e.g., 720 a) represents a portion of the representation (e.g., 718 and/or 730) of the first user (e.g., 710 and/or 714) that is displayed with a visual fidelity (e.g., the precision and/or accuracy of the information received by the computer system to generate the representation) that is below a visual fidelity threshold amount (e.g., the shadow represents a portion of the first user for which the computer system does not have sufficient data to generate an accurate representation of the portion of the first user for which the precision and/or accuracy exceeds a threshold amount of data received by the computer system). In some implementations, a second portion of the representation (e.g., 718 and/or 730) of the first user (e.g., 710 and/or 714) that is displayed with visual fidelity that is above the visual fidelity threshold amount is not represented by shading (e.g., 720 a).

Displaying the shadow as a portion of the representation representing the first user that is displayed with visual fidelity that is below the threshold amount of visual fidelity enables the computer system to provide a more complete state of the first user's body, regardless of whether the portion of the user's representation is displayed in a less clear manner, which provides improved visual feedback.

In some embodiments, the shadow (e.g., 720 a) represents a portion of the first user's (e.g., 710 and/or 714) representation (e.g., 718 and/or 730) that is not currently displayed via the display generating component (e.g., 120, 700a, 704a and/or 944 a) of the computer system (e.g., 101, 700, 704 and/or 944 a) (e.g., a portion of the first user's body that is not included and/or otherwise indicated by the representation). In some implementations, the currently undisplayed portion of the representation (e.g., 718 and/or 730) of the first user (e.g., 710 and/or 714) includes a leg (e.g., 710d and/or 710 e) of the user (e.g., 710 and/or 714), and the shadow (e.g., 720 a) is optionally displayed to approximate and/or estimate a shadow that would occur when the leg (e.g., 710d and/or 710 e) of the user is included and displayed in the augmented reality environment (e.g., 715).

Displaying the shadow as a currently undisplayed portion of the representation representing the first user enables the computer system to provide a more complete state of the first user's body, regardless of the portion of the user's representation not being displayed, which provides improved visual feedback.

In some embodiments, a computer system (e.g., 101, 700, 704, and/or 944) displays a shadow (e.g., 720 a) in an augmented reality environment (e.g., 715) at a first location (e.g., a location of shadow 720a shown in fig. 7A) relative to one or more objects (e.g., 722) in the augmented reality environment (e.g., 120, 700a, 704a, and/or 944 a) via a display generating component (e.g., 120, 700a, 704a, and/or 944 a) (e.g., when the shadow is displayed on and/or relative to one or more objects in the augmented reality environment, the first location causes the shadow to have a first shape). In response to detecting movement of the first user (e.g., 710 and/or 714) (e.g., movement detected via one or more sensors in communication with the computer system), the computer system (e.g., 101, 700, 704, and/or 944) displays a shadow (e.g., 720 a) in the augmented reality environment (e.g., 715) via the display generating component (e.g., 120, 700a, 704a, and/or 944 a) at a second location (e.g., a location of shadow 720a shown in fig. 7B) different from the first location relative to one or more objects (e.g., 715) in the augmented reality environment (e.g., the second location causes the shadow to have a second shape when the shadow is displayed on and/or relative to one or more objects in the augmented reality environment). In some implementations, displaying the shadow (e.g., 720 a) at the second location includes displaying the shadow (e.g., 720 a) in the augmented reality environment (e.g., 715) with an appearance and/or location based on movement of the user (e.g., 710 and/or 714) and/or a location of the user (e.g., 710 and/or 714) in the physical environment (e.g., 708 and/or 712) (e.g., a location of the user in the physical environment after movement).

Displaying shadows at the second location in response to detecting movement of the first user as the first user moves in the physical environment provides improved visual feedback regarding the state of the first user's body, which provides improved visual feedback.

In some implementations, a computer system (e.g., 101, 700, 704, and/or 944) displays, via a display generating component (e.g., 120, 700a, 704a, and/or 944 a), a shadow (e.g., 720 a) in an augmented reality environment (e.g., 715) having a first shape (e.g., the shape of shadow 720a shown in fig. 7A) in the augmented reality environment (e.g., 715) (e.g., the computer system determines the first shape based at least in part on information indicative of a state of one or more body parts of a first user). In response to detecting a change in the location of the first user (e.g., 710 and/or 714) (e.g., a change in the location of at least one body part of the first user detected via one or more sensors in communication with the computer system), the computer system displays, via the display generating component (e.g., 120, 700a, 704a, and/or 944 a), a shadow (e.g., 720 a) in the augmented reality environment (e.g., 715) having a second shape (e.g., the shape of the shadow shown in fig. 7B) different from the first shape in the augmented reality environment (e.g., 715) in response to detecting the change in the location of the at least one body part of the first user) (e.g., the computer system determines the second shape based at least in part on information indicative of a state of the one or more body parts of the first user (e.g., a change in the location of the first user).

Displaying the shadow as having the second shape in response to detecting the change in position of the first user when the first user changes position in the physical environment provides improved visual feedback regarding the state of the first user's body, which provides improved visual feedback.

In some implementations, the augmented reality environment (e.g., 715) includes a representation of a ground (e.g., 722) of a physical environment (e.g., 708 and/or 712) of the first user (e.g., 710 and/or 714) (e.g., the computer system displays the augmented reality environment with a representation of a physical ground and/or floor included within the physical environment in which the first user is located), and the shadow (e.g., 720 a) is displayed on the ground (e.g., 722) of the physical environment (e.g., 708 and/or 712) (e.g., at least partially displayed on the first ground, such as overlaid on the first ground).

Displaying shadows on the ground of the physical environment provides improved visual feedback regarding the state of the first user's body relative to the physical environment, which provides improved visual feedback.

In some implementations, the augmented reality environment (e.g., 715) includes a representation of a ground (e.g., 722) of the virtual environment (e.g., the computer system utilizes the representation of the virtual ground to display the augmented reality environment that is not a representation and/or depiction of a physical ground in a physical environment in which the first user and/or another user is located), and the shadow (e.g., 720 a) is displayed on the ground (e.g., 722) of the virtual environment (e.g., at least partially displayed on a second ground, such as overlaid on the second ground).

Displaying shadows on the ground of the virtual environment provides improved visual feedback regarding the state of the first user's body relative to the augmented reality environment, which provides improved visual feedback.

In some implementations, the augmented reality environment (e.g., 715) includes a representation of illumination (e.g., 732) from a light source that illuminates at least a portion of the physical environment (e.g., 708 and/or 712) corresponding to the augmented reality environment (e.g., 715) (e.g., the computer system displays the augmented reality environment with a representation of actual illumination from an actual light source (e.g., a lamp, bulb, and/or sun) within the physical environment in which the first user is located), and the shadow (e.g., 720 a) has an appearance (e.g., the computer system determines a shape, size, and/or other appearance of the shadow based at least in part on the direction and/or angle of light emission from the light source (e.g., 732) that illuminates the portion of the physical environment (e.g., 708 and/or 712) in the physical environment in which the first user is located).

Displaying the shadow as having an appearance determined based at least in part on illumination from the light source illuminating the portion of the physical environment enables the computer system to provide a more complete state of the first user's body in a familiar manner, which provides improved visual feedback.

In some implementations, the augmented reality environment (e.g., 715) includes a representation of virtual light (e.g., 732) that is illuminating at least a portion of the augmented reality environment (e.g., 715) (e.g., a computer system displays the augmented reality environment with a representation of virtual illumination from a virtual light source (e.g., a lamp, bulb, and/or sun) that is not within a physical environment in which the first user and/or another user is located), and the shadow (e.g., 720 a) has an appearance (e.g., the computer system determines the shape, size, and/or other appearance of the shadow based at least in part on the representation of virtual light (e.g., 732) that is illuminating the portion of the augmented reality environment (e.g., 715) based at least in part on the direction and/or angle in which the light is emitted via the virtual light source).

Displaying the shadow as having an appearance determined based at least in part on the representation of the virtual light illuminating the portion of the augmented reality environment enables the computer system to provide a more complete state of the first user's body in a familiar manner, which provides improved visual feedback.

In some embodiments, the representation (e.g., 718 and/or 730) of the first user (e.g., 710 and/or 714) includes a third visual indication (e.g., 754a and/or 754 b) of the foot (e.g., 714g and/or 714 h) of the body of the first user (e.g., 710 and/or 714) (e.g., a representation of the foot and/or shoe that is not an anatomically accurate representation of the foot of the first user). While displaying the representation (e.g., 718 and/or 730) of the first user (e.g., 710 and/or 714) at a third location (e.g., 748 a) in the augmented reality environment (e.g., 715) (e.g., a third location determined based at least in part on information indicative of a state of the first user), the computer system (e.g., 101, 700, 704 and/or 944) displays the foot (e.g., 710 and/or 714) of the body of the first user (e.g., 710 and/or 714) in the augmented reality environment (e.g., 715) via the display generating component (e.g., 120, 700a, 704a and/or 944 a), 714g and/or 714H) at a fourth location (e.g., the location of representation 730 shown in fig. 7H) in the augmented reality environment (e.g., 754a and/or 754 b) (e.g., the fourth location of the third visual indication of the foot is determined based at least in part on the state of the physical foot of the first user and/or the state of another portion of the body of the first user, rather than based on the state of the physical foot of the first user). in response to detecting movement of the first user (e.g., 710 and/or 714) (e.g., upward and/or downward movement of at least one body part (e.g., head, eyes, and/or shoulders) of the first user via one or more sensors in communication with the computer system), the computer system (e.g., 101, 700, 704, and/or 944) generates, via the display generating component (e.g., 120, 700a, 704a, and/or 944 a), a fifth location (e.g., 748 b) (e.g., A representation (e.g., 718 and/or 730) of the first user (e.g., 710 and/or 714) is displayed at a fifth location determined based at least in part on the information indicative of the state of the first user, wherein the fifth location is determined based at least in part on the movement of the first user (e.g., 710 and/or 714), and wherein the fifth location (e.g., 748 b) is above or below the third location (e.g., 748 a) (e.g., relative to one or more objects in the augmented reality environment). In response to detecting movement of the first user (e.g., 710 and/or 714) (e.g., upward and/or downward movement of at least one body part (e.g., head, eyes, and/or shoulders) of the first user via one or more sensors in communication with the computer system), the computer system (e.g., 101, 700, 704, and/or 944) displays the first user (e.g., via the display generating component (e.g., 120, 700a, 704a, and/or 944 a) in the augmented reality environment (e.g., 715) at a sixth location (e.g., the location of representation 730 shown in figure 7I) of the augmented reality environment (e.g., 715), 710 and/or 714) and/or a third visual indication (e.g., 754a and/or 754 b) of the foot (e.g., 714g and/or 714 h) of the body (e.g., the sixth location of the third visual indication of the foot is determined based at least in part on the detected movement of the first user, the state of the physical foot of the first user, and/or the state of another portion of the body of the first user, rather than based on the state of the physical foot of the first user).

Displaying a third visual indication of the foot of the body of the first user at the sixth location in response to detecting movement of the first user as the first user moves within the physical environment provides improved visual feedback regarding the state of the body of the first user, which provides improved visual feedback.

In some embodiments, a computer system (e.g., 101, 700, 704, and/or 944) displays a representation (e.g., 718 and/or 730) of a first user (e.g., 710 and/or 714) in an augmented reality environment (e.g., 715), the representation comprising: in accordance with a determination that a set of one or more criteria is met (e.g., the first user is determined to be in a sitting position and/or at least one body part of the first user is determined to be touching and/or contacting a piece of furniture (e.g., a chair, a table, and/or a desk)), a visual indication (e.g., 720, 726b, 738, 742a, 754a, and/or 754 b) of a portion (e.g., 711 b) of the body of the first user (e.g., 710 and/or 714) includes a piece of furniture (e.g., 738) in an augmented reality environment (e.g., a representation of a piece of furniture positioned in a physical environment in which the first user is located and/or a piece of virtual furniture that is not based on a piece of physical furniture in a physical environment in which the first user is located) (e.g., displaying the virtual representation of the portion of the first user's body including a piece of furniture includes displaying a piece of furniture having a position and/or orientation based on the position and/or orientation of the first user's representation in the augmented reality environment, and in accordance with a determination that the set of one or more criteria is not met (e.g., the first user is determined not to be sitting and/or at least one body part of the first user is not determined to be touching and/or contacting a piece of furniture (e.g., a chair, table, and/or desk)), a visual indication (e.g., 720, 726b, 738, 742a, 754a, and/or 754 b) of the portion of the first user (e.g., 710 and/or 714) is in the augmented reality environment (e.g., 715 No piece of furniture (e.g., 738) (e.g., maintaining a representation of the first user displayed with a visual indication).

Displaying the visual indication as including a piece of furniture in accordance with the determination that the set of one or more criteria is met provides improved visual feedback when the first user is in a sitting position.

In some embodiments, the set of one or more criteria includes criteria that are met when the computer system (e.g., 101, 700, 704, and/or 944) receives an indication that the first user (e.g., 710 and/or 714) is in a sitting position (e.g., the computer system detects and/or receives an indication that one or more portions of the first user's body (e.g., legs, knees, feet, and/or torso) are in a position that indicates the first user is sitting), and the piece of furniture (e.g., 738) in the augmented reality is a representation of a chair (e.g., 738) (e.g., a representation of a chair positioned in a physical environment in which the first user is located (e.g., the representation of a chair includes an appearance that is similar to that of a chair in the physical environment) and/or a representation of a virtual chair having an appearance that is not based on the piece of physical furniture in the physical environment in which the first user is located).

When the computer system receives an indication that the first user is in a sitting position, the visual indication is displayed as including a representation of the chair, which provides improved visual feedback when the first user is in a sitting position, which provides improved visual feedback.

In some embodiments, the representation of the chair (e.g., 738) includes a parameterized chair (e.g., a two-dimensional virtual object comprising a chair-like appearance and positioned below the first user relative to the ground of the augmented reality environment to indicate that the parameterized chair is supporting the first user in a sitting position) having one or more properties (e.g., color, number of legs, simulated material, and/or shape) independent of (e.g., not determined based on) the physical object supporting the first user (e.g., 710 and/or 714) in the physical environment (e.g., 708 and/or 712) in which the first user (e.g., 710 and/or 714) is located (e.g., the appearance of the parameterized chair is not determined based on actual image data of the chair and/or other object on which the first user is sitting).

Displaying the representation of the chair to include parameterizing the chair provides improved visual feedback regarding the state of the first user's body in a familiar manner, which provides improved visual feedback.

In some implementations, when a computer system (e.g., 101, 700, 704, and/or 944) displays a representation (e.g., 718 and/or 730) of a first user (e.g., 710 and/or 714) in an augmented reality environment (e.g., 715), the computer system (e.g., 101, 700, 704, and/or 944) detects that the first user (e.g., 710 and/or 714) is touching a surface (e.g., 744) of an object (e.g., 708 and/or 712) in a physical environment (e.g., 710 and/or 714) in which the first user (e.g., 710 and/or 714) is located within a predetermined distance (e.g., less than 5 centimeters (cm), less than 3cm, less than 2cm, or less than 1 cm) of a physical surface of at least a portion (e.g., a hand) of a body of the first user in the physical environment. In response to detecting that the first user (e.g., 710 and/or 714) is touching a surface of an object (e.g., 744) in the physical environment (e.g., 708 and/or 712), the computer system (e.g., 101, 700, 704, and/or 944) displays (e.g., displays concurrently with the representation of the first user) a representation of the surface (e.g., 726 and/or 726 b) in the augmented reality environment (e.g., 715) (e.g., a representation of the surface positioned in the physical environment in which the first user is located (e.g., a representation of the surface includes a representation of a virtual surface based on an appearance of the surface positioned in the physical environment) and/or having an appearance that is not based on the physical surface in the physical environment in which the first user is located).

Displaying a representation of a surface in response to detecting that the first user is touching the surface of the object provides improved visual feedback when the first user contacts the surface in the physical environment, which provides improved visual feedback.

In some implementations, when a computer system (e.g., 101, 700, 704, and/or 944) displays a representation (e.g., 718 and/or 730) of a first user (e.g., 710 and/or 714) in an augmented reality environment (e.g., 715), the computer system (e.g., 101, 700, 704, and/or 944) detects that the first user (e.g., 710 and/or 714) is touching an object (e.g., 740) in a physical environment (e.g., 708 and/or 712) in which the first user (e.g., 710 and/or 714) is located (e.g., detects that at least a portion of the first user's body (e.g., a hand) is within a predetermined distance (e.g., less than 5 centimeters (cm), less than 3cm, less than 2cm, or less than 1 cm) of the physical object in the physical environment in which the user is located. In response to detecting that the first user (e.g., 710 and/or 714) is touching an object (e.g., 740) in the physical environment (e.g., 708 and/or 712), the computer system (e.g., 101, 700, 704, and/or 944) displays (e.g., displays concurrently with the representation of the first user) a representation of the object (e.g., 742 and/or 742 b) in the augmented reality environment (e.g., representation of the first user) (e.g., representation of the object includes a representation of a virtual object based on an appearance of the object positioned in the physical environment) and/or having an appearance that is not based on the physical object positioned in the physical environment in which the first user is positioned).

Displaying a representation of an object in a physical environment in response to detecting that the first user is touching the object provides improved visual feedback when the first user contacts the object, which provides improved visual feedback.

In some embodiments, when a visual indication (e.g., 720, 726b, 738, 742a, 754a, and/or 754 b) of a first user (e.g., 710 and/or 714) is displayed in an augmented reality environment (e.g., 715) at a seventh location (e.g., a location of visual indication 720 shown in fig. 7A) relative to one or more objects (e.g., 722) in the computer system (e.g., the computer system determines the seventh location based at least in part on information indicative of a state of the portion of the first user's body), the computer system (e.g., 101, 700, 704, and/or 944) detects movement of a second portion of the first user's body (e.g., 710 and/or 714) (e.g., movement of the second portion of the first user's body (e.g., the portion and/or a different portion of the first user's body) via one or more sensors in communication with the computer system). In response to detecting movement of the second portion of the body of the first user (e.g., 710 and/or 714), the computer system (e.g., 101, 700, 704, and/or 944) displays a visual indication (e.g., 720, 726B, 738, 742a, 754a, and/or 754B) of the first user (e.g., 710 and/or 714) at an eighth location (e.g., a location of visual indication 720 shown in fig. 7B) that is different from the seventh location relative to the one or more objects (e.g., 722) in the augmented reality environment (e.g., 715) due to movement of the portion of the body of the first user such that the visual indication changes appearance of the eighth location relative to the one or more objects in the augmented reality environment.

Displaying a visual indication at an eighth location in response to detecting movement of the first user as the first user moves in the physical environment provides improved visual feedback regarding the state of the first user's body, which provides improved visual feedback.

In some embodiments, aspects/operations of methods 800 and 1000 may be interchanged, substituted, and/or added between the methods. For example, the representation displayed by the computer system performing method 1000 may include a visual indication of the portion of the first user's body. For the sake of brevity, these details are not repeated here.

Fig. 9A-9H illustrate examples of representations of different portions of a display user with different visual fidelity metrics. FIG. 10 is a flow chart of an exemplary method 1000 for displaying representations of different portions of a user with different visual fidelity metrics. The user interfaces in fig. 9A-9H are used to illustrate the processes described below, including the process in fig. 10.

Fig. 9A-9H illustrate examples of an electronic device 944 displaying a representation of one or more portions of a user 900 in different appearances based on a position and/or location of the one or more portions of the user 900 relative to one or more regions defined with respect to the user 900's body. When a portion of the body of the user 900 (e.g., one or more body parts and/or one or more particular body parts) is positioned in a region (e.g., a region corresponding to the portion of the body of the user 900), the electronic device 944 displays a representation of the user 900 with a first appearance. For example, the electronic device 944 displays a portion of the representation corresponding to the portion of the body of the user 900 with a reduced visual fidelity measure (e.g., lower accuracy, lower definition, lower visibility, and/or increased amount of blurring) based on the portion of the body of the user 900 being positioned within the region. When the portion of the user 900's body is positioned outside the region, the electronic device 944 displays a representation of the user 900 with a second appearance (e.g., an increased amount of visual fidelity and/or a decreased amount of blur) that is different from the first appearance.

Fig. 9A and 9B show examples of a first region 902 and a second region 904, respectively, within a physical environment 901 and as defined with respect to a body of a user 900. The electronic device 944 associates the first region 902 with the hand 900b and/or hand 900c of the user 900. For example, the electronic device 944 receives information indicative of a state (e.g., position, orientation, shape, and/or pose) of the hand 900b and/or hand 900c of the user 900 within the physical environment 901. In some embodiments, the information indicative of the status of the hand 900b and/or the hand 900c is indicative of whether the hand 900b and/or the hand 900c is positioned within the first region 902. The electronic device 944 displays a representation of the hand 900b and/or hand 900c having a first appearance, such as a first visual fidelity measure (e.g., a reduced visual fidelity measure (e.g., lower accuracy and/or lower sharpness) and/or an increased amount of blur), based on information indicating that the hand 900b and/or hand 900c is within the first region 902. The electronic device 944 displays a representation of the hand 900b and/or the hand 900c having a second appearance, such as a second visual fidelity measure (e.g., an increased visual fidelity measure (e.g., greater accuracy and/or greater sharpness) and/or a decreased amount of blur as compared to the first visual fidelity measure) based on information indicating that the hand 900b and/or the hand 900c is positioned outside the first region 902.

Similarly, the electronic device 944 associates the second region 904 with the elbow 900g and/or the elbow 900h of the user 900. For example, the electronic device 944 receives information indicating the state (e.g., position, orientation, shape, and/or pose) of the elbow 900h and/or the elbow 900h of the user 900 within the physical environment 901. In some embodiments, the information indicative of the state of the elbow 900h and/or the elbow 900h indicates whether the elbow 900h and/or the elbow 900h is positioned within the second zone 904. The electronic device 944 displays a representation of the elbow 900h and/or elbow 900h having a first appearance, such as a first visual fidelity measure (e.g., a reduced visual fidelity measure (e.g., lower accuracy and/or lower sharpness) and/or an increased amount of blur), based on information indicating that the elbow 900h and/or elbow 900h is positioned within the second region 904. The electronic device 944 displays a representation of the elbow 900h and/or elbow 900h having a second appearance, such as a second visual fidelity measure (e.g., an increased visual fidelity measure (e.g., greater accuracy and/or greater sharpness) and/or a decreased amount of blur as compared to the first visual fidelity measure) based on information indicating that the elbow 900h and/or elbow 900h is positioned outside the second region 904.

In some embodiments, the electronic device 944 does not display a representation of other body parts than the hand 900b and/or the hand 900b, which has a first appearance within the first region 902 based on the other body parts. Similarly, in some embodiments, the electronic device 944 does not display a representation of the other body than the elbow 900g and/or the elbow 900h that has a first appearance based on the other body parts within the second region 904. Thus, the electronic device 944 does not adjust the appearance of the representation of the body part that does not correspond to the first region 902 and the second region 904 and/or otherwise display a different appearance of the representation.

At fig. 9A, a first view 906 of the user 900 includes a side view of the body of the user 900 and a first region 902, shown, for example, as a box extending a depth 902a and a height 902b relative to the body of the user 900 (e.g., a waist 900a of the user 900). In particular, the depth 902a extends from a first location 908a across the waist 900a of the user 900 to a second location 908b, wherein the first location 908a and the second location 908b are not located on the physical body of the user 900 (e.g., the first location 908a and the second location 908b are located in an area of the physical environment 901 that is determined based on the location of the body of the user 900 (e.g., the waist 900 a)). Further, the height 902b extends from a third location 908c to a fourth location 908d. In some implementations, the height 902b includes a distance determined based on the height 910 of the user 900 and/or the size 912 of the hand 900b of the user 900. In some implementations, the depth 902a includes a distance determined based on a width 914 of the user 900 (e.g., a width of the waist 900a of the user 900). In some embodiments, the position and/or location of the first region 902 relative to the body of the first user 900 is determined based on the position of the pocket and/or other feature of the article of clothing worn by the user 900 such that the hand 900b and/or hand 900c is within the first region 902 when the hand 900b and/or hand 900c is proximate the pocket and/or other feature.

At fig. 9A, a second perspective 916 of user 900 includes a front view of user 900, and first region 902 is shown as a box extending, for example, height 902b and length 902 c. The length 902c extends from the fifth location 908e through the hand 900b, waist 900a, and hand 900c of the user 900 to the sixth location 908f. In some embodiments, the length 902c includes a distance equal to the depth 902 a. In some embodiments, the length 902c includes a distance determined based on the width 918 of the user 900 (e.g., the width 918 is the distance between the hand 900b and the hand 900c and/or the distance between the shoulder 900d and the shoulder 900e of the user 900 when the hand 900b and the hand 900c are placed on the sides of the user 900). As shown in fig. 9A, the depth 902a, the height 902b, and the length 902c each extend along three different axes within the physical environment 901 such that the first region 902 extends in three dimensions around (e.g., at least partially around) the body of the user 900.

For example, the third perspective 920 of the user 900 includes a bird's eye view and/or a top view of the user 900, and the first region 902 is shown as, for example, an area between the first circle 902d and the second circle 902e (e.g., the first region 902 does not include an area of the physical environment 901 that is inside the first circle 902d and outside the second circle 902 e). Although fig. 9A shows the first region 902 as the region between the first circle 902d and the circle 902e, in some implementations, the first region 902 is defined as the region between two non-circular shapes extending outward from the center 922 of the user 900. At fig. 9A, the diameter 924 of the second circle 902e includes substantially the same length as the depth 902a and length 902 c. Furthermore, the first circle 902d and the second circle 902e do not extend the entire height of the body of the user 900, but extend a distance substantially equal to the height 902 b. In some embodiments, the diameter 926 of the first circle 902d includes a distance based on the width 914 and/or width 918 of the user 900. In some implementations, the distance 928 between the first circle 902d and the second circle 902e is based on the position of the hand 900b and/or the position of the hand 900c when placed at and/or near the waist 900a of the user 900.

Similarly, at fig. 9B, a first perspective 930 of the user 900 includes a side view of the body of the user 900, and the second region 904 is shown as a box extending the depth 904a and the height 904B, for example, relative to the body of the user 900 (e.g., the waist 900a and/or the back 900f of the user 900). In particular, the depth 904a extends from a first location 932a across the waist 900a of the user 900 to a second location 932b, wherein the first location 932a and the second location 932b are not located on the physical body of the user 900 (e.g., the first location 932a and the second location 932b are located in an area of the physical environment 901 that is determined based on a location of the body (e.g., the waist 900a and/or the back 900 f) of the user 900). Further, the height 904b extends from the third position 932c to the fourth position 932d. In some implementations, the height 904b includes a distance determined based on the height 910 of the user 900. In some implementations, the depth 904a includes a distance determined based on the width 914 of the user 900. In some embodiments, the depth 904a and the height 904b are less than the depth 902a and the height 902b.

At fig. 9B, a second view 934 of user 900 includes a front view of user 900, and second region 904 is shown as a box extending, for example, height 904B and length 904 c. The length 904c extends from the fifth location 932e through the elbow 900g, the waist 900a, and the elbow 900h of the user 900 to the sixth location 932f. In some embodiments, the length 904c includes a distance equal to the depth 904 a. In some embodiments, the length 904c includes a distance determined based on the width 918 of the user 900 (e.g., the width 918 is the distance between the hand 900b and the hand 900c and/or the distance between the shoulder 900d and the shoulder 900e of the user 900 when the hand 900b and the hand 900c are placed on the sides of the user 900). As shown in fig. 9B, the depth 904a, the height 904B, and the length 904c each extend along three different axes within the physical environment 901 such that the second region 904 extends in three dimensions around (e.g., at least partially around) the body of the user 900.

For example, the third viewing angle 936 of the user 900 includes a bird's eye view and/or a top view of the user 900, and the second region 904 is shown as, for example, an area between the first circle 904d and the second circle 904e (e.g., the second region 904 does not include an area of the physical environment 901 that is inside the first circle 904d and outside the second circle 904 e). Although fig. 9B shows the second region 904 as the region between the first circle 904d and the circle 904e, in some embodiments the second region 904 is defined as the region between two non-circular shapes extending outward from the center 922 of the user 900. At fig. 9B, the diameter 938 of the second circle 904e includes a length that is substantially the same as the depth 904a and the length 904 c. Furthermore, the first circle 904d and the second circle 904e do not extend the entire height of the body of the user 900, but extend a distance substantially equal to the height 904 b. In some embodiments, the diameter 940 of the first circle 904d includes a distance based on the width 914 and/or width 918 of the user 900. In some embodiments, the distance 942 between the first circle 904d and the second circle 904e is based on the position of the elbow 900g and/or the position of the elbow 900h when placed at and/or near the body side of the user 900.

As discussed below, the first region 902 and the second region 904 are substantially fixed relative to one or more portions of the body of the user 900 (e.g., the waist 900a, the shoulder 900d, the shoulder 900e, and/or the back 900 f) such that corresponding portions of the body of the user 900 are determined to be within the first region 902 (e.g., the hand 900b and/or the hand 900 c) and/or the second region 904 (e.g., the elbow 900g and/or the elbow 900 h) regardless of movement of the user 900.

Fig. 9C-9H illustrate examples of electronic device 944 displaying, via display 944a, a communication interface 946 that includes a first participant region 946a that corresponds to user 900 and a second participant region 946b that corresponds to a second user (e.g., a second user associated with and/or using electronic device 944). At fig. 9C, the first participant region 946a includes an augmented reality environment 948 and a first representation 950 and a table representation 952 of the user 900 within the augmented reality environment 948 (e.g., an image representing a virtual table and/or a table 958 within the physical environment 901). In addition, the second participant region 946b includes a second representation 954 of the second user (e.g., representing an avatar and/or image of the second user).

Fig. 9C-9H also illustrate a user 900 within a physical environment 901 (e.g., an actual environment in which the user 900 is physically located), where the physical environment 901 includes the user 900 and a table 958 (e.g., a physical table). Electronic device 944 communicates with sensor 960a and sensor 960b that are positioned within physical environment 901 (e.g., wirelessly via external electronic devices associated and/or used by user 900). In some implementations, the sensors 960a and 960b include cameras, image sensors, light sensors, depth sensors, haptic sensors, orientation sensors, proximity sensors, temperature sensors, position sensors, motion sensors, and/or speed sensors. The sensors 960a and 960b are configured to capture data and/or information related to a state (e.g., position, orientation, pose, and/or position) of the user 900 within the physical environment 901. For example, sensors 960a and 960b are configured to detect and capture information related to the positioning and/or movement of various body parts of user 900 within physical environment 901. Although fig. 9C-9H illustrate the electronic device 944 communicating with two sensors (e.g., sensor 960a and sensor 960 b), in some embodiments the electronic device 944 communicates with any suitable number of sensors (e.g., via external electronic devices associated with the user 900).

At fig. 9C, electronic device 944 receives information (e.g., via sensors 960a and/or 960b and/or via an external device) that indicates a state of one or more body parts of user 900 within physical environment 901. In response to receiving the information, the electronic device 944 displays a first representation 950 within the augmented reality environment 948 of the first participant region 946 a. As shown in fig. 9C, the first representation 950 includes an appearance that mimics the physical appearance of the user 900 in the physical environment 901. For example, the first representation 950 includes waists 900a, hands 900b, hands 900c, shoulders 900d, shoulders 900e, elbows 900g, and elbows 900h corresponding to the waists 950a, hands 950b, hands 950c, shoulders 950d, shoulders 950e, elbows 950g, and elbows 950h of the user 900. Specifically, the hand 950b of the first representation 950 is lifted above the waist 950a within the augmented reality environment 948, similar to the hand 900b of the user 900 in the physical environment 901. The hand 950c of the first representation 950 is positioned on and/or near the table representation 952 within the augmented reality environment 948, similar to the hand 900c of the user 900 being positioned on and/or near the table 958 in the physical environment 901.

At fig. 9C, a first region 902 and a second region 904 are shown within a physical environment 901 in which a user 900 is located. The first region 902 is represented by a semicircle surrounding the area of the physical environment 901 near the waist 900a of the user 900 and the second region 904 is represented by a hatched line surrounding the area near the stomach 900i of the user 900. Although the first region 902 and the second region 904 are shown within the physical environment 901, the first region 902 and the second region 904 are not physically visible to and/or indistinguishable within the physical environment to the user 900. Further, the electronic device 944 does not display the first region 902 and/or the second region 904 on the communication user interface 946 and/or in the augmented reality environment 948 via the display 944 a.

At fig. 9C, both the hand 900b and the hand 900C of the user 900 are positioned outside of a first region 904 within the physical environment 901. Similarly, elbow 900g and elbow 900h are both positioned outside of second region 906 within physical environment 901. The electronic device 944 receives information indicating the state (e.g., position, orientation, posture, and/or pose) of the hand 900b and hand 900c outside the first region 902 and the elbow 900g and elbow 900h outside the second region 904. Based on the received information of the hands 900b and 900C outside the first region 902, the electronic device 944 displays the hands 950b and 950C of the first representation 950 as having a first appearance within the augmented reality environment 948 (e.g., as indicated by solid lines shown in fig. 9C). At fig. 9C, the electronic device 944 displays the hand 950b and the hand 950C of the first representation 950 with a first visual fidelity measure and/or without applying blur to the hand 950b and the hand 950C. In some embodiments, the electronic device 944 displays the hand 950b and the hand 950c as anatomically accurate representations of the hand 900b and the hand 900c without applying any amount of blurring to the hand 900b and the hand 900c based on information indicating that the hand 950b and the hand 950c are outside of the first region 902. In some implementations, the electronic device 944 displays the hand 950b and the hand 950c in a first appearance (e.g., a first visual fidelity measure) because the hand 900b and the hand 900c of the user 900 are outside the first region 902, thereby indicating that the user 900 is using the hand 900b and/or the hand 900c to communicate with a second user.

Similarly, based on the received information indicating that the elbow 900g and the elbow 900h are outside the second region 904, the electronic device 944 displays the elbow 950g and the elbow 950h of the first representation 950 as having a first appearance within the augmented reality environment 948 (e.g., as indicated by solid lines shown in fig. 9C). At fig. 9C, the electronic device 944 displays the elbow 950g and the elbow 950h of the first representation 950 with a first visual fidelity measure and/or without applying blur to the elbow 950g and/or the elbow 950h. In some embodiments, the electronic device 944 displays the elbows 950g and 950h as anatomically accurate representations of the elbows 900g and 900h without applying any amount of blurring to the elbows 950g and 950h based on information indicating that the elbows 900g and 900h are outside the second region 904.

At fig. 9D, user 900 has moved toward table 958 in physical environment 901. For example, user 900 is positioned closer to table 958 in fig. 9D than user 900 is in fig. 9C. Specifically, the waist 900a of the user 900 moves toward the table 958 while the hand 900c of the user 900 remains located on and/or near the table 958. At fig. 9D, the electronic device 944 receives information indicating a state of the user 900 within the physical environment 901, including the locations of the waist 900a and the hand 900c. Information indicating the status of the user 900 indicates that the hand 900c is inside the first region 902, as shown in fig. 9D. Based on the received information indicating that the hand 900c is inside the first region 902, the electronic device 944 displays the hand 950c of the first representation 950 as having a second appearance. For example, at fig. 9D, hand 950c is shown displayed by electronic device 944 in dashed lines to indicate that the electronic device is displaying hand 950c in a second appearance. In some implementations, the second appearance includes displaying the hand 950c with a second visual fidelity measure (e.g., precision and/or sharpness) and/or with an increased amount of blur. In some embodiments, the second appearance includes a blurred sphere and/or other non-anatomically accurate representation of the hand 950c displayed as the hand 900c of the user 900. In some implementations, the electronic device 944 displays the hand 950c in a second appearance (e.g., a second visual fidelity measure) because the hand 900c of the user 900 is inside the first region 902, thereby indicating that the user 900 is not using the hand 900c to communicate with a second user. In some embodiments, the electronic device 944 displays the hand 900c in a second appearance even when the electronic device 944 receives information (e.g., direct data captured and/or detected via sensors 960a and/or 960 b) that indicates a state of the hand 950c within the physical environment 901.

As shown in fig. 9D, a first region 902 and a second region 904 are shown relative to the user 900. In contrast to the position and/or location shown in fig. 9C, the first region 902 and the second region 904 include substantially the same position and/or location relative to the waist 900a and/or stomach 900i of the user 900, although the user 900 is moving toward the table 958. Thus, even when the user 900 moves in the physical environment 901, the position and/or location of the first region 902 and the second region 904 within the physical environment 901 is maintained relative to at least a portion of the body (e.g., the waist 900 a) of the user 900.

At fig. 9E, the user 900 has moved off the table 958 (e.g., as compared to the position of the user 900 at fig. 9D) and has moved the hand 900b and the hand 900c (e.g., relative to the waist 900a of the user 900). The user 900 has moved the hand 900b to a position and/or location near the waist 900a of the user 900 and/or near a pocket of a jacket worn by the user 900. The user 900 also moves the hand 900c to a location and/or position near the stomach 900i of the user 900. Thus, hand 900b is within first region 902 and hand 900c is within second region 904 but outside of first region 902. At fig. 9E, the electronic device 900 receives information indicating the status of the user 900 within the physical environment 901, including the locations of the waist 900a, hand 900b, and hand 900 c. The received information indicates that hand 900b is inside first region 902 and hand 900c is outside first region 902, as shown at fig. 9E. Based on the received information, the electronic device 944 displays the hand 950b of the first representation 950 as having a second appearance. For example, at fig. 9E, hand 950b is shown displayed by electronic device 944 in dashed lines to indicate that the electronic device is displaying hand 950b in a second appearance. In some embodiments, the second appearance includes displaying the hand 950b with a second visual fidelity measure (e.g., precision and/or sharpness) and/or with an increased amount of blur. In some embodiments, the second appearance includes a blurred sphere and/or other non-anatomically accurate representation of the hand 950b displayed as the hand 900b of the user 900. In some implementations, the electronic device 944 displays the hand 950b in a second appearance (e.g., a second visual fidelity measure) because the hand 900b of the user 900 is inside the first region 902, thereby indicating that the user 900 is not using the hand 900b to communicate with a second user.

At fig. 9E, the electronic device 944 displays the hand 900c of the first representation 950 in a first appearance (e.g., indicated by the solid line shown in fig. 9E) based on the received information indicating that the hand 950c of the user 900 is outside the first region 902. Even though hand 900c is within second region 904, electronic device 944 displays hand 950c in a first appearance because the received information indicates that hand 900c is outside of first region 902. Thus, the electronic device 944 is configured to display the hand 900b and/or the hand 900c in a first appearance and/or a second appearance based on information indicating whether the hand 950b and/or the hand 950c is inside and/or outside the first region 902 and not inside and/or outside the second region 904. In other words, the electronic device 944 does not modify the appearance of the hand 900b and/or hand 900c based on information indicating that the hand 950b and/or hand 950c is positioned inside the second region 904.

As shown in fig. 9E, a first region 902 and a second region 904 are shown relative to the user 900. In contrast to the positions and/or orientations shown in fig. 9C and 9D, the first region 902 and the second region 904 include substantially the same positions and/or orientations relative to the waist 900a and/or stomach 900i of the user 900, although the user 900 is moving toward the table 958. Thus, even when the user 900 moves in the physical environment 901, the position and/or location of the first region 902 and the second region 904 within the physical environment 901 is maintained relative to at least a portion of the body (e.g., the waist 900 a) of the user 900.

At fig. 9F, user 900 has moved hand 900b and hand 900c over shoulder 900d and shoulder 900e, respectively. Thus, both hand 900b and hand 900c are positioned outside of first region 902. At fig. 9F, the electronic device 900 receives information indicating the status of the user 900 within the physical environment 901, including the positions of the hand 900b, hand 900c, elbow 900g, and elbow 900 h. The received information indicates that both hand 900b and hand 900c are positioned outside of the first region 902. Based on information that both the indication 900b and the hand 900c are outside the first region 902, the electronic device 944 displays the hand 950b and the hand 950c of the first representation 950 as having a first appearance. Further, the received information indicates that both the elbow 900g and the elbow 900h are positioned outside the second zone 904. Based on the information indicating that both the elbow 900g and the elbow 900h are outside the second region 904, the electronic device 944 displays the elbow 950g and the elbow 950h of the first representation 950 as having a first appearance.

At fig. 9G, the user 900 has moved the hand 900b, hand 900c, elbow 900G, and elbow 900h to the sides of the user 900. Thus, hand 900b and hand 900c are both positioned inside first region 902, and elbow 900g and elbow 900h are both positioned inside second region 904. At fig. 9F, the electronic device 900 receives information indicating the status of the user 900 within the physical environment 901, including the positions of the hand 900b, hand 900c, elbow 900g, and elbow 900 h. The received information indicates that both hand 900b and hand 900c are positioned inside the first region 902. Based on the received information indicating that both hand 900b and hand 900c are inside first region 902, electronic device 944 displays hand 950b and hand 950c of first representation 950 as having a second appearance (e.g., as indicated by hand 950b and hand 950c having dashed lines). Further, the received information indicates that both the elbow 900g and the elbow 900h are positioned inside the second zone 904. Based on the information indicating that both the elbow 900g and the elbow 900h are inside the second region 904, the electronic device 944 displays the elbow 950g and the elbow 950h of the first representation 950 as having a second appearance (e.g., as indicated by the elbow 950g and the elbow 950h having dashed lines).

As described above, the electronic device 944 is configured to adjust the appearance of the hand 900b and/or hand 900c based on whether the received information indicates that the hand 950b and/or hand 950c is inside and/or outside the first region 902. Further, the electronic device 944 is configured to adjust the appearance of the elbows 900g and 900h based on whether the received information indicates that the elbows 950g and 950h are positioned inside and/or outside the second region 904. When the received information indicates that the hand 900b and/or the hand 950c is positioned inside the second region 904, the electronic device 944 does not adjust and/or modify the appearance of the hand 950b and/or the hand 900 c. Similarly, when the received information indicates that the elbow 900g and/or the elbow 950h is positioned inside the first zone 902, the electronic device 944 does not adjust and/or modify the appearance of the elbow 950g and/or the elbow 900 h.

At fig. 9H, user 900 has moved in physical environment 901 toward table 958. For example, user 900 is positioned closer to table 958 in fig. 9H than user 900 is in fig. 9G. Specifically, the waist 900a of the user 900 moves toward the table 958 while the hands 900b and 900c of the user 900 remain located near the sides (e.g., waist 900 a) of the user 900. In addition, elbows 900g and 900H of user 900 remain located near the sides of user 900, as shown in fig. 9H. The electronic device 944 receives information indicative of the status of the user 900 within the physical environment 901, including the locations of the waist 900a, the hands 900b, the hands 900c, the elbows 900g, and/or the elbows 900 h. The received information indicates that both hand 900b and hand 900b are inside first region 902. Based on the received information indicating that the hands 900b and 900c are inside the first region 902, the electronic device 944 displays the hands 950b and 950c of the first representation 950 as having a second appearance. The received information also indicates that both the elbow 900g and the elbow 900h are inside the second zone 904. Based on the received information indicating that the elbows 900g and 900h are inside the second zone 904, the electronic device 944 displays the elbows 950g and 950h in a second appearance.

At fig. 9H, a first region 902 and a second region 904 are shown relative to user 900. In contrast to the positions and/or orientations shown in fig. 9C-9G, the first region 902 and the second region 904 include substantially the same positions and/or orientations relative to the waist 900a of the user 900, although the user 900 is moving toward the table 958. Thus, even when the user 900 moves in the physical environment 901, the position and/or location of the first region 902 and the second region 904 within the physical environment 901 is maintained relative to at least a portion of the body (e.g., the waist 900 a) of the user 900.

Additional description regarding fig. 9A-9H is provided below with reference to the method 1000 described with respect to fig. 9A-9H.

Fig. 10 is a flowchart of an exemplary method 1000 for displaying representations of different portions of a user with different visual fidelity metrics, according to some embodiments. In some embodiments, the method 1000 is performed at a computer system (e.g., computer system 101 in fig. 1) of an external computer system that includes a display generating component (e.g., display generating component 120 in fig. 1,3, and 4) (e.g., a visual output device, a 3D display, a display having at least a transparent or translucent portion on which an image may be projected (e.g., see-through display), a projector, a heads-up display, a display controller) and a communication session (e.g., augmented reality and/or video conferencing) being operated by a first user (e.g., a user with whom the computer system is being engaged). In some embodiments, method 1000 is managed by instructions stored in a non-transitory (or transitory) computer-readable storage medium and executed by one or more processors of a computer system (such as one or more processors 202 of computer system 101) (e.g., control 110 in fig. 1). Some operations in method 1000 are optionally combined and/or the order of some operations is optionally changed.

As described below, the method 1000 provides an intuitive way for displaying representations of different portions of a user with different visual fidelity metrics. The method reduces the cognitive burden of users participating in real-time communication sessions, thereby creating a more efficient human-machine interface. For battery-powered computing devices, enabling a user to communicate with another participant of a real-time communication session faster and more efficiently saves power and increases the time interval between battery charges.

The computer system (e.g., 101, 700, 704, and/or 944) displays (1004) a representation (e.g., 950) (e.g., an avatar and/or virtual representation of at least a portion of the first user) of the first user (e.g., a user in a physical environment) in the augmented reality environment (e.g., 948) in response to receiving a request (e.g., based on user input at the computer system) to display (e.g., display) a representation (e.g., 950) (e.g., an avatar; a virtual avatar (e.g., a virtual representation of at least a portion of the first user) in the augmented reality environment (e.g., 948) via the display generating component (e.g., 120, 700a, 704a, and/or 944 a).

Displaying a representation (e.g., 950) of the first user (e.g., 900) includes (in some embodiments, displaying a virtual avatar in place of the first user in an augmented reality environment): in accordance with a determination that a first portion (e.g., 900b, 900c, 900g, and/or 900 h) of a body of a first user (e.g., 900) is in a first region (e.g., 902 and/or 904) of a physical environment (e.g., 901) in which the first user (e.g., 900) is located, wherein the first region (e.g., 902 and/or 904) is defined relative to the body of the first user (e.g., the first portion of the body of the first user is positioned within a predetermined area relative to a second portion (e.g., hip, waist, torso, and/or abdomen region) of the body of the first user, the computer system (e.g., 101, 700, 704, and/or 944) displays (1006), via a display generation component (e.g., 120, 700a, 704a, and/or 944 a), a first visual indication (e.g., 950b, 950c, 950g, and/or 950 h) of a first portion (e.g., 900b, 900c, 900g, and/or 900 h) of a body of a first user (e.g., a blurred representation of the first portion of the body of the first user, wherein the blurred representation is not an anatomically accurate depiction and/or representation of the first portion of the body of the first user, wherein the first visual indication (e.g., 950b, 950c, 900g, and/or 900 h) of the first portion of the body of the first user (e.g., 900b, 900c, 900g, and/or 900 h), 950g and/or 950 h) includes a first visual fidelity metric (and/or precision) (e.g., a low fidelity metric such that the first visual indication of the first portion of the body includes a blurred representation of the first portion of the body (e.g., a blurred, transparent, and/or other visual indication of the first portion of the body that is not an anatomically accurate depiction and/or representation of the first portion of the body)). In some embodiments, the predetermined area includes a three-dimensional area relative to (e.g., extending around and/or partially around) a second portion of the body of the first user that indicates that the first portion of the body of the first user is inactive and/or not moving (e.g., hands in pockets, hands on buttocks, hands on both sides of the first user, double-arm intersection).

Displaying a representation (e.g., 950) of the first user (e.g., 900) includes (in some embodiments, displaying a virtual avatar in place of the first user in an augmented reality environment): in accordance with a determination that a first portion (e.g., 900b, 900c, 900g, and/or 900 h) of a body of a first user (e.g., 900) is in a second region of a physical environment (e.g., 901) (e.g., a region of the physical environment 901 that is outside of region 902 and/or region 904), wherein the second region is separate from the first region (e.g., 902 and/or 904) (e.g., the first portion of the body of the first user is positioned outside of a predetermined area relative to the second portion (e.g., hip, waist, torso, and/or abdomen region) of the body of the first user), a computer system (e.g., 101, 700, 704, and/or 944) generates, via a display generating component (e.g., 120, 700a, 704a, and/or 944 a) displays (1008) a second visual indication (e.g., 950b, 950c, 950g, and/or 950 h) of a first portion (e.g., 900b, 900c, 900g, and/or 900 h) of a body of a first user (e.g., 950b, 950c, 950g, and/or 950 h) (e.g., a representation of the first portion of the body of the first user that is an anatomically accurate representation and/or depiction of the first portion of the body of the first user), wherein the second visual indication (e.g., 950b, 950c, 950g, and/or 950 h) of the first portion (e.g., 900b, 900c, 900g, and/or 900 h) of the body of the first user (e.g., the high fidelity metric is such that the first visual indication of the first portion of the body comprises an accurate depiction of the first portion of the body).

When the first user is likely to communicate during the real-time communication session using the first portion of the body, displaying the first visual indication and/or the second visual indication based on determining that the first portion of the first user's body is in the first zone and/or the second zone provides improved visual feedback by displaying the portion of the representation with increased clarity, which provides improved visual feedback.

In some embodiments, displaying the representation (e.g., 950) of the first user (e.g., 900) includes defining (e.g., a second portion of the first user's body (e.g., a hand, forearm, and/or elbow) in a third region (e.g., 902 and/or 904) of a physical environment (e.g., 901) in which the first user (e.g., 900) is located (e.g., a third region of the physical environment in which the first user is located that is different from a first region of the physical environment in which the first user is located) in accordance with a determination that the second portion of the first user's body (e.g., 900b, 900c, 900g, and/or 904) is defined (e.g., a second portion of the first user's body is positioned within a predetermined region of the third portion (e.g., a waist, abdomen, and/or abdomen region) relative to the first user's body) with respect to the first user's body (e.g., 900), and generating (e.g., a visual indication (e.g., 101, 944 and/or 950) of the third region (e.g., 900 a, 900 a) and/or 900b, 900, or (e.g., 900) via the display (e.g., 950, 900 and/or 900 a) and/or 900 h) of the third region (e.g., 950 and/or 900b, 950), the third visual indication of the second portion of the body of the first user is different from the first visual indication and the second visual indication of the first portion of the body of the first user, wherein the third visual indication (e.g., 950b, 950c, 950g, and/or 950 h) of the second portion of the body (e.g., 900b, 900c, 900g, and/or 900 h) of the first user (e.g., comprises a first visual fidelity metric (and/or precision) (e.g., the low fidelity metric is such that the third visual indication of the second portion of the body comprises a blurred representation of the second portion of the body (e.g., blurred, transparent, and/or other visual indications of the second portion of the body) that is not an anatomically accurate depiction and/or representation of the second portion of the body).

Displaying the third visual indication in the third zone based on determining that the second portion of the first user's body is in the third zone provides improved visual feedback by displaying the represented portion with increased clarity when the first user is likely to communicate during the real-time communication session using the particular portion of the body, which provides improved visual feedback.

In some embodiments, the first region (e.g., 902 and/or 904) surrounds a predetermined portion (e.g., 900a, 900f, and/or 900 i) of the body of the first user (e.g., the first region includes a predetermined area relative to a second portion (e.g., hip, waist, torso, and/or abdomen area) of the body of the first user that extends at least partially around the second portion of the body of the first user.

The first region surrounding a predetermined portion of the body of the first user enables to make a determination as to whether the first portion of the body of the first user is within an area where the first user is unlikely to use the first portion of the body for communication during the real-time communication session, which provides improved visual feedback.

In some embodiments, the first region (e.g., 902 and/or 904) extends in three dimensions relative to a physical environment (e.g., 901) in which the first user (e.g., 900) is located (e.g., the first region extends along three different axes into a spatial region in the physical environment in which the first user is located).

The first region extending in three dimensions relative to the physical environment enables making a determination as to whether the first portion of the first user's body is within an area where the first user is unlikely to use the first portion of the body for communication during the real-time communication session, which provides improved visual feedback.

In some embodiments, the first region (e.g., 902) of the physical environment (e.g., 901) in which the first user (e.g., 900) is located corresponds to a first portion (e.g., 900b and/or 900 c) of the body of the user (e.g., 900) in which the first user is located, the first region of the physical environment corresponding to only the first portion of the body of the first user such that the visual fidelity and/or amount of visual indication of the second portion of the body of the first user (e.g., 900 b) is unchanged and/or maintained when the second portion of the body of the first user is determined to be positioned in the first region, and the third region (e.g., 904) of the physical environment (e.g., 901) in which the first user (e.g., 900) is located corresponds to a second portion (e.g., 900g and/or 900 h) of the body of the first user (e.g., 900) different from the first portion of the body of the first user (e.g., 900) in which the visual fidelity and/or amount of visual indication of the second portion of the body of the first user is not changed and/or maintained when the second portion of the body of the first user (e.g., 900) is positioned in the first region of the first user and/or other physical environment corresponding to the first portion of the first user is not changed and/or is determined to be maintained.

The first region corresponding to the first portion of the body of the first user and the third region corresponding to the second portion of the body of the first user provide improved visual feedback by displaying different portions of the representation that may be used by the first user for communication during the real-time communication session, which provides improved visual feedback.

In some embodiments, when a computer system (e.g., 101, 700, 704, and/or 944) displays a representation (e.g., 950) of a first user (e.g., 900) in an augmented reality environment, the computer system (e.g., 101, 700, 704, and/or 944) receives an indication of a state (e.g., position, orientation, and/or pose) of a first portion (e.g., 900b, 900c, 900g, and/or 900 h) of a body of the first user (e.g., 900) within a physical environment (e.g., 901) in which the first user (e.g., 900) is located (e.g., receives information and/or data indicating a position and/or movement of the first portion of the body of the first user in the physical environment in which the first user is located). In response to receiving an indication of a state of a first portion (e.g., 900b, 900c, 900g, and/or 900 h) of a body of a first user (e.g., 900) within a physical environment (e.g., 901) in which the first user (e.g., 900) is located and in accordance with determining an indication of a state of a first portion (e.g., 900b, 900c, 900g, and/or 900 h) of a body of the first user (e.g., 900) within a first region (e.g., 902 and/or 904) of a physical environment (e.g., 901) in which the first user (e.g., 900) is located, the computer system (e.g., 101, 700, 704, and/or 944) maintains a first visual indication (e.g., 950b, 950c, 950g, and/or 950 h) of a first visual indication (e.g., of 950b, 950c, 950g, and/or 950 h) of a first visual indication of a first portion of a body of a first user (e.g., 900) within the physical environment in which the first user is located, while the first portion of the first user is located continues to be visually indicated in the first region of the first body despite the first visual indication of the first portion of the first user's body being located. In some implementations, in response to receiving an indication of a state of a first portion of a body of a first user within a physical environment in which the first user is located and in accordance with a determination that the first portion of the body of the first user is in a second region of the physical environment in which the first user is located, a computer system (e.g., 101, 700, 704, and/or 944) maintains a display of a second visual indication of the first portion of the body of the first user that includes a second visual fidelity metric.

Maintaining the display of the first visual indication of the first portion of the first user's body in response to receiving the indication of the state of the first portion of the first user's body provides improved visual feedback by displaying the first portion of the first user's body in a less clear manner when the first user is less likely to use the first portion of the body for communication despite detecting the location of the first portion of the first user's body.

In some embodiments, the first portion (e.g., 900b and/or 900 c) of the body of the first user (e.g., 900) comprises a portion of the body of the first user (e.g., 900b and/or 900 c) of the physical environment (e.g., 901) in which the first user (e.g., 900) is located, the first region (e.g., 902) of the physical environment (e.g., 901) comprises an area (e.g., the first region) of the physical environment (e.g., 901) corresponding to a pocket of an article of clothing worn by the first user (e.g., 900) at least partially surrounding a portion of the body of the first user comprising pants and/or jackets worn by the first user in the physical environment), and the first visual indication (e.g., 900b and/or 900 c) of the body of the first user (e.g., 900) comprises a more than the predetermined amount of applied visual blur metric of the hand (e.g., 900b and/or 900 c) of the first user (e.g., 900) of the second amount of applied blur.

Displaying a blurred representation of the hand when the hand is in an area corresponding to a pocket of an article of clothing worn by the user enables improved visual feedback to be provided by displaying the hand with lower clarity when the first user is less likely to use the hand to communicate.

In some embodiments, the first portion (e.g., 900b and/or 900 c) of the body of the first user (e.g., 900) comprises a portion of the body (e.g., 900b and/or 900 c) of the first user (e.g., 900) and the first region (e.g., 902) of the physical environment (e.g., 901) in which the first user (e.g., 900) is located comprises a region (e.g., a region below the waist and proximate to the body of the user (e.g., 901) of the physical environment (e.g., a waist and generally occupied by a pocket of pants) (e.g., the first region at least partially surrounds a portion of the body of the first user comprising a pocket of pants and/or jackets worn by the first user in the physical environment) and the first visual indication (e.g., 900b and/or 900 c) of the first portion of the body (e.g., 950b and/or 950 c) of the first user (e.g., 900) comprises a visual fidelity metric comprising a visual fidelity of the first user (e.g., 900) and/or a visual fidelity metric that is applied to a greater than the predetermined amount of visual fidelity.

Displaying the blurred representation of the hand when the hand is in an area near the user's waist provides improved visual feedback by displaying the hand at a lower resolution when the first user is less likely to use the hand to communicate, which improves visual feedback.

In some embodiments, the first portion (e.g., 900g and/or 900 h) of the body of the first user (e.g., 900) comprises an elbow (e.g., 900g and/or 900 h) of the first user (e.g., 900), the first region (e.g., 904) of the physical environment (e.g., 901) in which the first user (e.g., 900) is located comprises an area (e.g., a first region at least partially surrounding a torso (e.g., 900a, 900f, and/or 900 i) of the body of the first user (e.g., 900) comprising the torso of the first user and/or a portion above the waist of the user), and the first visual indication (e.g., 950g and/or 950 h) of the first portion (e.g., 900 g., 900 h) of the body of the first user (e.g., 900) comprises a representation (e.g., 950 g., 900g and/or 900 h) of the first visual indication that the first visual fidelity metric is applied more than the second visual fidelity metric is applied.

Displaying the blurred representation of the elbow when the elbow is in an area proximate to the torso of the first user's body provides improved visual feedback by displaying the elbow at a lower resolution when the first user is less likely to use the elbow for communication, which improves visual feedback.

In some implementations, while displaying a representation (e.g., 950) of a first user (e.g., 900), a computer system (e.g., 101, 700, 704, and/or 944) detects movement of the first user (e.g., 900) within a physical environment (e.g., 901) in which the first user (e.g., 900) is located (e.g., movement of at least a portion of the first user's body via one or more sensors in communication with the computer system). After detecting movement of the first user (e.g., 900) within the physical environment (e.g., 901) in which the first user (e.g., 900) is located, the computer system (e.g., 101, 700, 704, and/or 944) displays at least a portion of the representation (e.g., 950) of the first user (e.g., 900) with an appearance (e.g., a visual fidelity measure, such as a first visual fidelity measure and/or a second visual fidelity measure) determined based on a location of the first region (e.g., 902 and/or 904) in the physical environment (e.g., 901) in which the first user is located, wherein the first region (e.g., 902 and/or 904) is moved so as to maintain a position of the first region (e.g., 902 and/or 904) relative to the body (e.g., 900) of the first user in which the first region of the physical environment is located moves with the first user such that the first region of the physical environment in which the first user is located remains substantially stationary relative to the at least one body part of the first user as the first user moves around the physical environment.

Maintaining the position of the first region relative to the body of the first user improves visual feedback by displaying the portion of the first user's representation with increased clarity when the user may be communicating using the first portion of the body regardless of the movement of the first user in the physical environment.

In some implementations, while displaying a representation (e.g., 950) of a first user (e.g., 900), a computer system (e.g., 101, 700, 704, and/or 944) detects movement of the first user (e.g., 900) within a physical environment (e.g., 901) in which the first user (e.g., 900) is located (e.g., movement of at least a portion of the first user's body via one or more sensors in communication with the computer system). After detecting movement of the first user (e.g., 900) within the physical environment (e.g., 901) in which the first user (e.g., 900) is located, the computer system (e.g., 101, 700, 704, and/or 944) displays at least a portion of the representation (e.g., 950) of the first user with an appearance (e.g., a visual fidelity measure, such as a first visual fidelity measure and/or a second visual fidelity measure) determined based on a location of a second region (e.g., a region of the physical environment that is external to regions 902 and/or 904) in the physical environment (e.g., 901) in which the first user is located, wherein the second region is moved so as to maintain a location of the second region relative to the body (e.g., 900) of the first user (e.g., the second region of the physical environment in which the first user is located) moving with the first user such that the second region of the physical environment in which the first user is located remains substantially stationary relative to the at least one part of the body of the first user as the first user moves around the physical environment.

Maintaining the position of the second region relative to the body of the first user improves visual feedback by displaying the portion of the representation of the first user with increased clarity when the user may be communicating using the first portion of the body regardless of movement of the first user in the physical environment.

In some embodiments, aspects/operations of methods 800 and 1000 may be interchanged, substituted, and/or added between the methods. For example, the appearance of portions of the representation displayed by the computer system performing method 800 may be adjusted based on the location and/or positioning of one or more body parts of the first user relative to the region of the physical environment in which the first user is located. For the sake of brevity, these details are not repeated here.

The foregoing description, for purposes of explanation, has been described with reference to specific embodiments. However, the illustrative discussions above are not intended to be exhaustive or to limit the invention to the precise forms disclosed. Many modifications and variations are possible in light of the above teaching. The embodiments were chosen and described in order to best explain the principles of the invention and its practical application to thereby enable others skilled in the art to best utilize the invention and various described embodiments with various modifications as are suited to the particular use contemplated.

As described above, one aspect of the present technology is to collect and use data from various sources to improve the XR experience of the user. The present disclosure contemplates that in some examples, such collected data may include personal information data that uniquely identifies or may be used to contact or locate a particular person. Such personal information data may include demographic data, location-based data, telephone numbers, email addresses, tweet IDs, home addresses, data or records related to the user's health or fitness level (e.g., vital sign measurements, medication information, exercise information), date of birth, or any other identifying or personal information.

The present disclosure recognizes that the use of such personal information data in the present technology may be used to benefit users. For example, personal information data may be used to improve the XR experience of the user. In addition, the present disclosure contemplates other uses for personal information data that are beneficial to the user. For example, health and fitness data may be used to provide insight into the overall health of a user, or may be used as positive feedback to individuals using technology to pursue health goals.

The present disclosure contemplates that entities responsible for collecting, analyzing, disclosing, transmitting, storing, or otherwise using such personal information data will adhere to established privacy policies and/or privacy practices. In particular, such entities should exercise and adhere to privacy policies and practices that are recognized as meeting or exceeding industry or government requirements for maintaining the privacy and security of personal information data. Such policies should be readily accessible to the user and should be updated as the collection and/or use of the data changes. Personal information from users should be collected for legal and reasonable use by entities and not shared or sold outside of these legal uses. In addition, such collection/sharing should be performed after informed consent is received from the user. In addition, such entities should consider taking any necessary steps to defend and secure access to such personal information data and to ensure that others who have access to personal information data adhere to their privacy policies and procedures. In addition, such entities may subject themselves to third party evaluations to prove compliance with widely accepted privacy policies and practices. In addition, policies and practices should be adjusted to collect and/or access specific types of personal information data and to suit applicable laws and standards including specific considerations of jurisdiction. For example, in the united states, the collection or acquisition of certain health data may be governed by federal and/or state law, such as the health insurance flow and liability act (HIPAA); while health data in other countries may be subject to other regulations and policies and should be processed accordingly. Thus, different privacy practices should be maintained for different personal data types in each country.

In spite of the foregoing, the present disclosure also contemplates embodiments in which a user selectively prevents use or access to personal information data. That is, the present disclosure contemplates that hardware elements and/or software elements may be provided to prevent or block access to such personal information data. For example, with respect to an XR experience, the present technology may be configured to allow a user to choose to "opt-in" or "opt-out" to participate in the collection of personal information data during or at any time after registration with a service. In another example, the user may choose not to provide data for service customization and/or generating a user representation. In yet another example, the user may choose to limit the length of time that data is maintained or to prohibit development of the customized service and/or generation of the user representation altogether. In addition to providing the "opt-in" and "opt-out" options, the present disclosure also contemplates providing notifications related to accessing or using personal information. For example, the user may be notified that his personal information data will be accessed when the application is downloaded, and then be reminded again just before the personal information data is accessed by the application.

Further, it is an object of the present disclosure that personal information data should be managed and processed to minimize the risk of inadvertent or unauthorized access or use. Once the data is no longer needed, risk can be minimized by limiting the data collection and deleting the data. In addition, and when applicable, included in certain health-related applications, the data de-identification may be used to protect the privacy of the user. De-identification may be facilitated by removing specific identifiers (e.g., date of birth, etc.), controlling the amount or specificity of stored data (e.g., collecting location data at a city level instead of at an address level), controlling how data is stored (e.g., aggregating data among users), and/or other methods, as appropriate.

Thus, while the present disclosure broadly covers the use of personal information data to implement one or more of the various disclosed embodiments, the present disclosure also contemplates that the various embodiments may be implemented without accessing such personal information data. That is, various embodiments of the present technology do not fail to function properly due to the lack of all or a portion of such personal information data. For example, a representation of the user may be generated based on non-personal information data or absolute minimum amount of personal information (such as content requested by a device associated with the user, other non-personal information available to the service, or publicly available information).

Claims

1. A method, comprising:

At a computer system in communication with a display generation component and with an external computer system associated with a first user:

In response to receiving a request to display a representation of the first user in an augmented reality environment:

Displaying the representation of the first user in the augmented reality environment via the display generating component, wherein the representation of the first user comprises:

a visual indication of a portion of the first user's body in the augmented reality environment, wherein:

the visual indication of the portion of the body of the first user has an appearance determined based at least in part on one or more objects in the augmented reality environment; and

The visual indication of the portion of the body of the first user represents an estimated state of the portion of the body estimated based on indirect information about the state of the portion of the body when the computer system is unable to obtain direct information about the state of the portion of the body.

2. The method of claim 1, wherein the representation of the first user further comprises a second visual indication of a second portion of the body of the first user, wherein the second visual indication of the second portion of the body of the first user comprises a blurred extension of the second portion of the body of the first user.

3. The method of any of claims 1-2, wherein the visual indication of the portion of the body of the first user comprises a shadow.

4. The method of claim 3, wherein the shadow represents a portion of the representation of the first user that is displayed with visual fidelity that is below a visual fidelity threshold amount.

5. The method of claim 3, wherein the shadow represents a portion of the representation of the first user that is not currently displayed via the display generating component of the computer system.

6. The method of any of claims 3 to 5, further comprising:

displaying, via the display generating component, the shadow in the augmented reality environment at a first location relative to the one or more objects in the augmented reality environment; and

In response to detecting movement of the first user, the shadow is displayed in the augmented reality environment at a second location different from the first location relative to the one or more objects in the augmented reality environment via the display generating component.

7. The method of any of claims 3 to 6, further comprising:

displaying, in the augmented reality environment, the shadow having a first shape in the augmented reality environment via the display generating component; and

In response to detecting a change in the position of the first user, the shadow having a second shape different from the first shape in the augmented reality environment is displayed in the augmented reality environment via the display generating component.

8. The method of any of claims 3-7, wherein the augmented reality environment comprises a representation of a ground of a physical environment of the first user, and wherein the shadow is displayed on the ground of the physical environment.

9. The method of any of claims 3-7, wherein the augmented reality environment comprises a representation of a ground surface of a virtual environment, and wherein the shadows are displayed on the ground surface of the virtual environment.

10. The method of any of claims 3-9, wherein the augmented reality environment comprises a representation of illumination from a light source that illuminates at least a portion of a physical environment corresponding to the augmented reality environment, and wherein the shadow has an appearance determined based at least in part on the illumination from the light source that illuminates the portion of the physical environment.

11. The method of any of claims 3-9, wherein the augmented reality environment includes a representation of virtual light illuminating at least a portion of the augmented reality environment, and wherein the shadow has an appearance determined based at least in part on the representation of the virtual light illuminating the portion of the augmented reality environment.

12. The method of any of claims 1-11, wherein the representation of the first user includes a third visual indication of a foot of the body of the first user, and wherein the method further comprises:

Displaying, via the display generating component, the third visual indication of the foot of the body of the first user at a fourth location in the augmented reality environment when the representation of the first user is displayed at a third location in the augmented reality environment; and

In response to detecting movement of the first user, displaying, via the display generating component, in the augmented reality environment:

The representation of the first user at a fifth location in the augmented reality environment, wherein the fifth location is determined based at least in part on the movement of the first user, and wherein the fifth location is above or below the third location; and

The third visual indication of the foot of the body of the first user at a sixth location in the augmented reality environment.

13. The method of any of claims 1-12, wherein displaying the representation of the first user in the augmented reality environment comprises:

in accordance with a determination that a set of one or more criteria is met, the visual indication of the portion of the body of the first user including furniture in the augmented reality environment; and

In accordance with a determination that the set of one or more criteria is not met, the visual indication of the portion of the body of the first user without the furniture in the augmented reality environment.

14. The method of claim 13, wherein the set of one or more criteria includes a criterion that is met when the computer system receives an indication that the first user is in a sitting position, and wherein the piece of furniture in the augmented reality environment is a representation of a chair.

15. The method of claim 14, wherein the representation of the chair comprises a parameterized chair having an appearance with one or more properties independent of supporting a physical object of the first user in the seated position in a physical environment in which the first user is located.

16. The method of any one of claims 1 to 15, further comprising:

detecting that the first user is touching a surface of an object in a physical environment in which the first user is located when the representation of the first user is displayed in the augmented reality environment; and

In response to detecting that the first user is touching the surface of the object in the physical environment, a representation of the surface is displayed in the augmented reality environment.

17. The method of any one of claims 1 to 16, further comprising:

Detecting that the first user is touching an object in a physical environment in which the first user is located when the representation of the first user is displayed in the augmented reality environment; and

In response to detecting that the first user is touching the surface of the object in the physical environment, a representation of the object is displayed in the augmented reality environment.

18. The method of any one of claims 1 to 17, further comprising:

Detecting movement of a second portion of the body of the first user while displaying the visual indication of the first user at a seventh location relative to the one or more objects in the augmented reality environment; and

In response to detecting movement of the second portion of the body of the first user, the visual indication of the first user at an eighth location different from the seventh location relative to the one or more objects in the augmented reality environment is displayed.

19. A non-transitory computer readable storage medium storing one or more programs configured for execution by one or more processors of a computer system in communication with a display generation component and with an external computer system associated with a first user, the one or more programs comprising instructions for performing the method of any of claims 1-18.

20. A computer system in communication with a display generation component and with an external computer system associated with a first user, the computer system comprising:

One or more processors; and

A memory storing one or more programs configured to be executed by the one or more processors, the one or more programs comprising instructions for performing the method of any of claims 1-18.

21. A computer system in communication with a display generation component and with an external computer system associated with a first user, the computer system comprising:

means for performing the method according to any one of claims 1 to 18.

22. A non-transitory computer readable storage medium storing one or more programs configured for execution by one or more processors of a computer system in communication with a display generation component and in communication with an external computer system associated with a first user, the one or more programs comprising instructions for:

a visual indication of a portion of the body of the first user in the augmented reality environment, wherein:

23. A computer system in communication with a display generation component and with an external computer system associated with a first user, the computer system comprising:

One or more processors; and

A memory storing one or more programs configured to be executed by the one or more processors, the one or more programs comprising instructions for:

24. A computer system in communication with a display generation component and with an external computer system associated with a first user, the computer system comprising:

in response to receiving a request to display a representation of a first user in an augmented reality environment:

means for displaying the representation of the first user in the augmented reality environment via the display generating component, wherein the representation of the first user comprises:

25. A method, comprising:

Displaying the representation of the first user in the augmented reality environment via the display generating component, wherein displaying the representation of the first user comprises:

In accordance with a determination that a first portion of the first user's body is in a first region of a physical environment in which the first user is located, wherein the first region is defined relative to the first user's body, displaying, via the display generating component, a first visual indication of the first portion of the first user's body, wherein the first visual indication of the first portion of the first user's body comprises a first visual fidelity measure; and

In accordance with a determination that the first portion of the body of the first user is in a second region of the physical environment, wherein the second region is separate from the first region, a second visual indication of the first portion of the body of the first user is displayed via the display generation component, wherein the second visual indication of the first portion of the body of the first user comprises a second visual fidelity metric different from the first visual fidelity metric.

26. The method of claim 25, wherein displaying the representation of the first user comprises:

in accordance with a determination that a second portion of the body of the first user is in a third region of the physical environment in which the first user is located, wherein the third region is defined relative to the body of the first user, a third visual indication of the second portion of the body of the first user is displayed via the display generating component, wherein the third visual indication of the second portion of the body of the first user comprises the first visual fidelity metric.

27. The method of any of claims 25 to 26, wherein the first zone surrounds a predetermined portion of the body of the first user.

28. The method of any of claims 25 to 27, wherein the first zone extends in three dimensions relative to the physical environment in which the first user is located.

29. The method of any one of claims 25 to 28, wherein:

The first region of the physical environment in which the first user is located corresponds to the first portion of the body of the first user; and

A third region of the physical environment in which the first user is located corresponds to a second portion of the body of the first user that is different from the first portion of the body of the first user.

30. The method of any of claims 25 to 29, further comprising:

Receiving an indication of a state of the first portion of the body of the first user within the physical environment in which the first user is located when the representation of the first user is displayed in the augmented reality environment; and

In response to receiving the indication of the state of the first portion of the body of the first user within the physical environment in which the first user is located:

In accordance with a determination that the first portion of the body of the first user is in the first region of the physical environment in which the first user is located, a display of the first visual indication of the first portion of the body of the first user including the first visual fidelity metric is maintained.

31. The method of any one of claims 25 to 30, wherein:

the first portion of the body of the first user comprises a hand of the first user;

The first region of the physical environment in which the first user is located includes an area of the physical environment corresponding to a pocket of an article of clothing worn by the first user; and

The first visual indication of the first portion of the body of the first user comprising the first visual fidelity metric comprises a blurred representation of the hand of the first user.

32. The method of any one of claims 25 to 30, wherein:

The first region of the physical environment in which the first user is located includes an area of the physical environment proximate to a waist of the user; and

33. The method of any one of claims 25 to 30, wherein:

the first portion of the body of the first user includes an elbow of the first user;

the first region of the physical environment in which the first user is located includes a region surrounding a torso of the body of the first user; and

The first visual indication of the first portion of the body of the first user including the first visual fidelity metric includes a blurred representation of the elbow of the first user.

34. The method of any of claims 25 to 33, further comprising:

Detecting movement of the first user within the physical environment in which the first user is located while the representation of the first user is displayed; and

After detecting the movement of the first user within the physical environment in which the first user is located, at least a portion of the representation of the first user is displayed in an appearance determined based on a location of the first region in the physical environment, wherein the first region is moved so as to maintain a position of the first region relative to the body of the first user.

35. The method of any of claims 25 to 34, further comprising:

After detecting the movement of the first user within the physical environment in which the first user is located, at least a portion of the representation of the first user is displayed in an appearance determined based on a position of the second region in the physical environment, wherein the second region is moved so as to maintain a position of the second region relative to the body of the first user.

36. A non-transitory computer readable storage medium storing one or more programs configured for execution by one or more processors of a computer system in communication with a display generation component and with an external computer system associated with a first user, the one or more programs comprising instructions for performing the method of any of claims 25-35.

37. A computer system in communication with a display generation component and with an external computer system associated with a first user, the computer system comprising:

One or more processors; and

A memory storing one or more programs configured to be executed by the one or more processors, the one or more programs comprising instructions for performing the method of any of claims 25-35.

38. A computer system in communication with a display generation component and with an external computer system associated with a first user, the computer system comprising:

means for performing the method according to any one of claims 25 to 35.

39. A non-transitory computer readable storage medium storing one or more programs configured for execution by one or more processors of a computer system in communication with a display generation component and in communication with an external computer system associated with a first user, the one or more programs comprising instructions for:

40. A computer system in communication with a display generation component and with an external computer system associated with a first user, the computer system comprising:

One or more processors; and

41. A computer system in communication with a display generation component and with an external computer system associated with a first user, the computer system comprising:

means for displaying the representation of the first user in the augmented reality environment via the display generating component, wherein displaying the representation of the first user comprises: