WO2023130435A1 - 交互方法、头戴式显示设备、***及存储介质 - Google Patents
交互方法、头戴式显示设备、***及存储介质 Download PDFInfo
- Publication number
- WO2023130435A1 WO2023130435A1 PCT/CN2022/070966 CN2022070966W WO2023130435A1 WO 2023130435 A1 WO2023130435 A1 WO 2023130435A1 CN 2022070966 W CN2022070966 W CN 2022070966W WO 2023130435 A1 WO2023130435 A1 WO 2023130435A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- head
- mounted display
- display device
- image
- external device
- Prior art date
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
Definitions
- the present application relates to the field of head-mounted display devices, and in particular, relates to an interaction method, a head-mounted display device, a system, and a storage medium.
- head-mounted display devices With the development of technology and the improvement of living standards, head-mounted display devices are gradually becoming popular.
- the appearance of the head-mounted display device is usually in the form of a goggle or a helmet, which puts the display screen close to the user's eyes and projects a picture to the user's eyes at a close distance.
- Head-mounted display devices are divided into two types: translucent and non-transparent. For non-transparent head-mounted display devices, users lose their eyes' perception of the outside world after wearing the head-mounted display device. It creates a lot of inconvenience when trying to operate the equipment.
- one of the objectives of the present application is to provide an interaction method, a head-mounted display device, a system, and a storage medium.
- an embodiment of the present application provides an interaction method for a head-mounted display device, the method including:
- the image area corresponding to the external device can be superimposed and displayed in the display image of the head-mounted display device, so that the wearer of the head-mounted display device can operate the external device while "seeing" the external device, improving The operation convenience of the external device; further, the image area corresponding to the external device is superimposed and displayed only when the wearer is detected to have the intention to operate the external device, and there is no need to display the image area corresponding to the external device at all times, so as to avoid blocking the head-mounted display image of the display device.
- an embodiment of the present application provides an interaction method for a head-mounted display device, the method including:
- the virtual device In response to detecting that the wearer has an intention to operate the virtual device, superimposing and displaying the virtual device in the display image of the head-mounted display device, the virtual device is obtained based on the camera of the head-mounted display device The virtual device ID is generated.
- the virtual device is superimposed and displayed on the display image of the head-mounted display device by means of the virtual device identifier set in the physical environment.
- the virtual device is superimposed and displayed only when it is detected that the wearer has the intention to operate the virtual device, and there is no need to display the virtual device at all times. Avoid blocking the display image of the head-mounted display device.
- the embodiment of the present application provides a head-mounted display device, including:
- processors one or more processors
- the one or more processors individually or jointly execute the executable instructions to perform the method as described in the first aspect or the second aspect.
- an embodiment of the present application provides an interactive system, including the head-mounted display device and an external device as described in the third aspect.
- the embodiment of the present application provides a computer-readable storage medium, the computer-readable storage medium stores executable instructions, and when the executable instructions are executed by a processor, the first aspect or the second aspect is implemented. the method described.
- FIG. 1 is a usage scene diagram of a head-mounted display device and external devices provided by an embodiment of the present application
- Fig. 2 is a schematic structural diagram of an interactive system provided by an embodiment of the present application.
- FIG. 3 is a schematic diagram of an image area corresponding to a keyboard superimposed on a display image of a head-mounted display device according to an embodiment of the present application;
- FIG. 4 is a schematic structural diagram of another interactive system provided by an embodiment of the present application.
- FIG. 5 is a schematic diagram of a usage scenario of the interactive system shown in FIG. 4 provided by an embodiment of the present application;
- FIG. 6 is a schematic diagram of superimposing an image area corresponding to a remote control device in a display image of a head-mounted display device provided by an embodiment of the present application;
- FIG. 7 is a schematic flowchart of an interaction method for a head-mounted display device provided in an embodiment of the present application.
- Fig. 8 is a schematic flowchart of another interaction method for a head-mounted display device provided by an embodiment of the present application.
- FIG. 9 is a schematic structural diagram of a head-mounted display device provided by an embodiment of the present application.
- a head-mounted display device 100 and an external device 200 are shown.
- the user wears the head-mounted display device 100
- there is a need for the user to operate some external devices 200 but in this case the user cannot see the external devices in the physical environment, and the user may not be familiar with the operation mode of the external devices, so that The convenience for the user to operate the external device is poor.
- the physical environment refers to the physical world that people can sense and/or interact with without the help of electronic devices. People are able to directly sense and/or interact with the physical environment, such as through sight, touch, hearing, taste and smell.
- the embodiment of the present application provides an interaction method for a head-mounted display device, which can shoot external devices through the camera of the head-mounted display device, and the captured image includes the image area corresponding to the external device , and then can superimpose and display the image area corresponding to the external device in the display image of the head-mounted display device, then the wearer of the head-mounted display device can operate the external device while "seeing" the external device, and improve the safety of the external device. Ease of operation. Furthermore, the inventor found that if the image area corresponding to the external device is superimposed and displayed on the display image of the head-mounted display device, it will cause continuous occlusion of the display image of the head-mounted display device, affecting the viewing experience of the user.
- the embodiment of the present application further realizes that the image area corresponding to the external device is superimposed on the display image of the head-mounted display device only when it is detected that the wearer has the intention to operate the external device, and there is no need to display the corresponding image area of the external device at all times.
- the image area of the head-mounted display device is avoided from blocking the display image of the head-mounted display device, and the operation convenience of the external device is improved while reducing the impact on the normal display of the display image of the head-mounted display device.
- the head-mounted display device mentioned in the embodiment of the present application includes but is not limited to smart glasses, a helmet provided with a display, goggles or a head-up display and the like.
- the displayed image of the head-mounted display device may be a virtual image, an image taken from a physical environment, or an image obtained by superimposing a virtual image and an image taken from a physical environment.
- the head-mounted display device includes a glasses body, one end of the glasses body is provided with a wearing part for cooperating with a user, and the other end of the glasses body is provided with at least a display.
- the display can display images within the wearer's field of vision, and the images displayed on the display include but are not limited to virtual images and/or images captured from physical environments.
- the head-mounted display device may also be provided with an inward-facing or outward-facing sensor/device, so as to provide a good interaction effect for the wearer.
- the head-mounted display device further includes a camera installed on the main body of the glasses, and the camera can shoot external devices in the physical environment to obtain images including the external devices; wherein, the camera includes not limited to a color camera, Monochrome cameras, monochrome cameras, 3D depth-sensing cameras, stereo cameras or time-of-flight (ToF) cameras, etc.
- the head-mounted display device further includes an attitude sensor provided on the glasses body, and the attitude sensor is used to detect the head posture of the wearer, for example, the content displayed on the display can be determined according to the head posture of the wearer.
- the head-mounted display device further includes an eye tracking sensor, which is used to detect the wearer's eye movement, for example, the wearer's eye gaze range can be determined according to the wearer's eye movement, and then the display corresponding to the displayed content.
- an eye tracking sensor which is used to detect the wearer's eye movement, for example, the wearer's eye gaze range can be determined according to the wearer's eye movement, and then the display corresponding to the displayed content.
- the external device may be an input device of a head-mounted display device.
- the head-mounted display device in the related art
- the input device of the display device may be some input device with a relatively simple operation mode.
- the input device may only include two or three operation controls with specified functions, and the input function of the first input device is limited.
- the external device when the user can "see" the external device, the external device may not only include an input device with a relatively simple operation mode, but also include an input device with a variety of different input functions, so as to satisfy Diversified input requirements of users.
- the input device of the head-mounted display device includes, but is not limited to, at least one of a keyboard, a mouse, a touch screen, a stylus, a rocker (such as a game joystick), and a handwriting pad.
- Various input devices can be Meet the user's personalized input needs.
- FIG. 2 shows an interactive system
- the interactive system includes a head-mounted display device 100 and a keyboard 201 .
- FIG. 3 shows a schematic diagram of superimposing the image area corresponding to the keyboard 201 in the display image 20 of the head-mounted display device 100 .
- the head-mounted display device 100 responds to the detection that the wearer has The intention of operating the external device is to obtain the image of the keyboard 201 captured by the camera, the image includes the image area 10 corresponding to the keyboard 201, and then the image area 10 corresponding to the keyboard 201 can be superimposed on the display image 20 of the head-mounted display device 100, so that The wearer can "see" the keyboard 201 in the physical environment when he intends to operate the keyboard 201, so as to realize the convenient operation of the keyboard 201 and improve the use experience of the wearer.
- the external device may be a device related to a head-mounted display device, for example, the external device may be a remote control device of a movable platform.
- the movable platform is provided with a photographing device, and the remote control device can control the movement process of the movable platform and/or the photographing process of the photographing device, for example, the remote control device can directly or indirectly send a message to the movable platform under the user's manipulation.
- the control command is transmitted to control the movement direction or movement speed of the movable platform.
- the movable platform is also communicatively connected with the head-mounted display device, and the movable platform can transmit the image taken by the shooting device to the head-mounted display device, and then the image taken by the movable platform based on the shooting device can be displayed on the In the head-mounted display device.
- the movable platform includes but not limited to vehicles, unmanned aerial vehicle (UAV), unmanned ship or mobile robot and so on.
- FIG. 4 shows an interactive system, which includes a head-mounted display device 100, a remote control device 202 and a drone 301 .
- FIG. 5 shows a usage scenario of the interactive system described in FIG. 4 .
- the drone 301 is provided with a photographing device
- the head-mounted display device 100 is provided with a camera.
- the remote control device 202 can directly or indirectly transmit control instructions to the UAV 301 based on the operation of the wearer of the head-mounted display device 100, so as to control the flight of the UAV 301 and/or the shooting device in the UAV 301 Take an image.
- the UAV 301 can take pictures during the flight and transmit them to the head-mounted display device 100 .
- the head-mounted display device 100 can display images captured by the drone 301 .
- the head-mounted display device 100 can respond to the detected The wearer intends to operate the remote control device 202 and acquires an image captured by the camera, and the image includes an image area corresponding to the remote control device 202 . Please refer to FIG.
- the head-mounted display device 100 can superimpose the image area 10 corresponding to the remote control device 202 on the display image 20 of the head-mounted display device, so that the wearer can " "See” the remote control device 202 in the physical environment, so as to realize the convenient operation of the remote control device 202 and improve the wearer's use experience.
- FIG. 7 is a schematic flowchart of a method for interacting with a head-mounted display device according to an embodiment of the present application. The method can be executed by the head-mounted display device, and the method includes :
- step S101 it is detected whether the wearer of the head-mounted display device intends to operate an external device.
- step S102 in response to detecting that the wearer has an intention to operate an external device, superimposing and displaying an image captured by a camera of the head-mounted display device on the display image of the head-mounted display device, the The image captured by the camera includes the image area corresponding to the external device.
- the head-mounted display device when the user is wearing the head-mounted display device, can display images within the user's field of view.
- the displayed image of the head-mounted display device may be a virtual image (for example, as shown in FIG. 3 ), or an image obtained from a physical environment, or a superimposed image of a virtual image and an image obtained from a physical environment.
- Image exemplary, the virtual image may be generated based on a preset modeling rule and using the principle of imaging geometry; for example, the virtual image includes but is not limited to a virtual game screen, a virtual building screen, or a virtual character screen.
- the head-mounted display device communicates with a mobile platform equipped with a shooting device, and the mobile platform includes but is not limited to drones, vehicles, ships, or mobile robots;
- the images include real-time images collected by the photographing device of the movable platform or images or video frames collected in a historical time period.
- the image captured from the physical environment may be an image obtained from a third-party platform, such as an image or a video frame shared by other users on a social platform.
- the head-mounted display device when the user wears the head-mounted display device, can detect whether the wearer of the head-mounted display device intends to operate the external device, so as to determine whether to wear the head-mounted display device The external device is superimposed on the display image of the display device.
- the head-mounted display device may detect whether the wearer intends to operate the external device in real time, or intermittently detect whether the wearer has the intention to operate the external device.
- the head-mounted display device may detect whether the wearer intends to operate the external device according to the wearer's head posture and/or eye gaze range.
- the head-mounted display device is equipped with an attitude sensor, and the wearer's head attitude can be determined through the data collected by the attitude sensor; the attitude sensor can include motion sensors such as a three-axis gyroscope, a three-axis accelerometer, and a three-axis electronic compass .
- the head-mounted display device may be installed with an eye tracking sensor, and the gaze range of the wearer's eyes may be determined through data collected by the eye tracking sensor.
- the height of the external device is generally lower than that of the head-mounted display device.
- the user wants to operate the external device he may lower his head subconsciously to determine the location of the external device. Therefore, if the head-mounted display device detects that the head posture indicates the wearer's head-down posture, it can be determined that the wearer has an intention to operate the external device.
- the head-mounted display device when the head-mounted display device displays visual information, it determines the gaze range of the wearer's eyes according to the data collected by the eye tracking sensor and/or the data collected by the posture sensor, and then obtains the display corresponding to the gaze range of the eyes.
- the content is presented to the user.
- the user can change the gaze range of the eyes by turning the head or eyeballs, so as to watch different display contents.
- a virtual component corresponding to the external device can be set, and the virtual component can be displayed on the head-mounted display device; when the wearer wants to operate the external device, he can change the gaze range of the eye by turning the head or eyeballs to find If the display content of the virtual component is displayed, if the head-mounted display device detects that the display content within the gaze range of the eyes includes the virtual component corresponding to the external device, it may determine that the wearer has the intention to operate the external device.
- the external device is an input device
- the virtual component corresponding to the external device may be an input virtual component.
- the external device is a remote control device, and the virtual component corresponding to the external device may be a virtual remote control.
- the head posture indicates the wearer's head-down posture and that the display content within the gaze range of the eyes includes a virtual component corresponding to the external device. In this case, it is determined that the wearer has an intention to operate the external device.
- the head-mounted display device is provided with designated controls, and the wearer can communicate his needs to the head-mounted display device by operating the external device when the wearer needs to operate the external device.
- the head-mounted display device may determine that the wearer intends to operate the external device in response to the triggering of the specified control, and then control the camera installed in the head-mounted display device to capture images.
- the head-mounted display device may not display the image captured by the camera of the head-mounted display device in response to the wearer not intending to operate the external device.
- the image area corresponding to the external device is not displayed, so as to avoid blocking the display image of the head-mounted display device.
- the head-mounted display device may determine that the wearer has no intention to operate the external device when detecting any one or more of the following situations: the wearer's head posture indicates that the wearer is looking up The posture, the displayed content within the gaze range of the wearer's eyes does not include the virtual component corresponding to the external device, or the specified part of the wearer does not exist in the image captured by the camera.
- the head-mounted display device when the head-mounted display device superimposes and displays the image area corresponding to the external device in the displayed image, it may periodically detect whether the wearer of the head-mounted display device operates the external device. Intent, if it is detected that the wearer has no intention to operate the external device, stop displaying the image area corresponding to the external device, so that the user can see the complete content of the displayed image.
- the camera of the head-mounted display device may be fixedly arranged in the head-mounted display device.
- the head-mounted display device is provided with a wearing part that cooperates with the user, and the camera is arranged on the outside of the head-mounted display device and is opposite to the wearing part of the head-mounted display device, so that The physical environment can be photographed to obtain images including external equipment.
- the height of the external device is generally lower than the height of the head-mounted display device based on the user's habit of operating the external device, then the camera can be set on the head-mounted display device.
- the outside of the display device, and the direction of the field of view of the camera can be tilted downward (for example, the direction of the field of view of the camera and the direction of gravity are at a preset angle, and the angle is less than 90 degrees), so as to capture images including external devices; of course , when the field of view of the camera is large, for example, if the camera is a fisheye camera, any setting of the camera on the outside of the head-mounted display device can capture images including external devices. There is also no restriction on the setting method on the outside of the display device.
- the camera may be arranged on the head-mounted display device and be rotatably connected to the head-mounted display device.
- the camera can be rotated artificially (for example, the camera is rotatably connected to the head-mounted display device through a knob), or the camera can also be rotated automatically under the action of a driving component.
- the head-mounted display device can control the rotation of the camera so that the lens assembly of the camera faces the external device, so that the camera can capture images including the external device, so that In response to the wearer's intention to operate the external device, the head-mounted display device superimposes and displays the image captured by the camera on the display image of the head-mounted display device, where the image includes an image area corresponding to the external device.
- the head-mounted display device may pre-store the reference information of the external device, and during the process of turning the camera, the head-mounted display device may check whether there is an external device in the image captured by the camera according to the reference information of the external device , when it is determined that the image captured by the camera includes the external device, the camera is controlled to stop rotating.
- the head-mounted display device in order to obtain images including external devices with better display effects, includes one or more supplementary lights.
- the head-mounted display device can detect the brightness of the environment where the external device is located, and when the brightness of the environment where the external device is located is less than a preset brightness threshold, Turn on the fill light.
- the preset brightness threshold can be specifically set according to the actual application scene, and this embodiment does not impose any limitation on this.
- the preset brightness threshold can be determined according to the brightness requirement of the camera when capturing images.
- the brightness of the environment where the external device is located can meet the brightness requirement of the camera, thereby obtaining an image with better display effect and improving the recognizability of the external device in the image.
- the ambient light brightness may be determined according to the number of photons received by the photosensitive element in the camera and/or the pixel value of the pixel in the image collected by the camera.
- the number of photons is positively correlated with the brightness of the environment where the external device is located, that is, the more the number of photons is, the greater the brightness of the environment where the external device is located.
- the pixel value of the pixel is also positively correlated with the brightness of the environment where the external device is located.
- the pixel value of the pixel is represented by a grayscale value, and a higher grayscale value indicates a greater brightness of the environment where the external device is located.
- the brightness of the environment where the external device is located can be determined according to any one of the number of photons and the pixel values of the pixels in the image, or can be jointly determined based on the two.
- the embodiment does not impose any limitation on this, and specific settings can be made according to actual application scenarios.
- the brightness of the supplementary light can be adjusted according to the brightness of the environment where the external device is located, for example, the brightness of the supplementary light is negatively correlated with the brightness of the environment where the external device is located Relationship, that is, the darker the brightness of the environment where the external device is located, the brighter the brightness of the supplementary light, on the contrary, the brighter the brightness of the environment where the external device is located, the darker the brightness of the supplementary light, so that the supplementary light can be avoided or reduced
- the brightness of the lamp affects others.
- the head-mounted display device can determine the supplementary light that needs to be turned on according to the difference between the brightness of the environment where the external device is located and the preset brightness threshold. number of lights. For example, when the brightness of the environment where the external device is located is less than the preset brightness threshold, the number of fill lights that need to be turned on is positively correlated with the difference, that is, the greater the difference, the more the number of fill lights that need to be turned on. Realize that the auxiliary camera captures images that meet the brightness requirements. The smaller the difference, the fewer the number of fill lights that need to be turned on, which can avoid or reduce the impact of the brightness of the fill lights on others.
- the images captured by the camera may contain other content that the user does not pay attention to
- the images captured by the camera may affect the head-mounted display device.
- the display image of the display device causes a large area of occlusion, which is not conducive to the user's viewing experience. Therefore, after the head-mounted display device acquires the original image captured by the camera, it can identify the original image to determine the image position corresponding to the external device, and then according to the image position corresponding to the external device in the original image, The original image is cropped to obtain a cropped image, the cropped image includes the image area corresponding to the external device, and then the cropped image can be superimposed and displayed on the display image.
- the display position of the image area corresponding to the external device in the display image is determined at least according to the relative positional relationship between the wearer and the external device, so that the external
- the image area corresponding to the device is adapted to the relative positional relationship between the wearer and the relative positional relationship between the wearer and the external device in the physical environment, so that the image area corresponding to the external device is compatible with the physical environment
- the scales of the external devices are the same, and the wearer can find the external devices in the physical environment based on the display position of the "seeing" external devices, so as to realize the convenient operation of the external devices in the physical environment.
- the display position of the image area corresponding to the external device in the display image can be based on the relative positional relationship between the wearer's eyes and the display of the head-mounted display device, the camera The relative positional relationship between the display and the display, and the relative positional relationship between the camera and the external device are determined.
- the relative positional relationship between the image area corresponding to the external device and the wearer's eyes in the display space is the same as the relative positional relationship between the wearer's eyes and the external device in the physical environment, and the wearer can
- the display position of the "seen" external device corresponds to find the external device in the physical environment, so as to realize the convenient operation of the external device in the physical environment.
- the image captured by the camera in order to allow the user to see the complete content of the display image of the head-mounted display device during the operation of the external device, the image captured by the camera can be superimposed in a semi-transparent or semi-opaque form displayed in the display image, so that the user can also see the blocked part of the display image of the head-mounted display device through the image captured by the camera, so as to improve the viewing experience of the user.
- the fixed translucency may prevent the wearer from seeing the image captured by the camera (that is, the external device), and the operation of the external device may be affected;
- the overall tone is light
- the fixed translucency may make it impossible for the wearer to see the occluded content in the displayed image, affecting the user's viewing experience.
- this embodiment can set the adaptive change of the transparency of the image captured by the camera.
- the transparency of the image captured by the camera can be determined according to the color system of the displayed image in the superimposed area; wherein, the superimposed area refers to that when the displayed image and the image captured by the camera are superimposed and displayed, the displayed image is captured by the camera The resulting image occluded regions.
- the transparency corresponding to the color system of the displayed image in the superimposed area belonging to the dark color system is smaller than the transparency corresponding to the color system of the displayed image in the superimposed area belonging to the light color system.
- the darker the color system of the superimposed area the smaller the transparency of the image captured by the camera is set, making the image captured by the camera clearer, so that the wearer can clearly see the image area corresponding to the external device, realizing convenient
- the operation process is permanent; the lighter the color system of the superimposed area, the greater the transparency of the image captured by the camera is set, so that the superimposed area (that is, the blocked area in the display area) is clearer, so that the wearer can see the display clearly.
- the full content of the image is permanent
- this embodiment does not impose any restrictions on the operation mode of the external device.
- the external device can be operated by the user's hand, or the external device can be operated by the user's foot. Therefore , when the wearer uses a designated part (such as a hand or a foot) to operate the external device, the image captured by the camera may further include an image area corresponding to the wearer's designated part.
- the image captured by the camera may further include an image area corresponding to the wearer's designated part.
- the image captured by the camera may further include an image area corresponding to the wearer's designated part.
- FIG. 3 when the wearer operates the keyboard with hands, the image area corresponding to the wearer's hands is superimposed on the displayed image.
- the wearer operates the remote control device by hand the image area corresponding to the wearer's hand is superimposed on the displayed image.
- the head-mounted display device may recognize the operation position of the designated part of the wearer on the external device from the image captured by the camera of the head-mounted display device, and then The operation data generated by the wearer operating the external device is acquired according to the content indicated by the operation position.
- the external device is communicatively connected to the head-mounted display device; then the head-mounted display device can directly obtain the operation generated by the wearer operating the external device through the communication connection. data.
- the external device is communicatively connected to the target device corresponding to the head-mounted display device; operation data.
- the target device includes a terminal device or a server.
- the terminal device includes, but is not limited to, a mobile phone, a computer, or a tablet, and the like.
- the operation data generated by the wearer operating the external device can be sent to the server, so that the server can send the operation data to the participant Other equipment for the meeting.
- the external device is an input device, and the operation data may be input text or image.
- the external device is a remote control device of the movable platform, and the operation data may be control content corresponding to the movable platform, such as controlling the acceleration of the movable platform.
- the display image of the head-mounted display device can be superimposed and displayed with the operation prompt information corresponding to the external device, so as to assist the wearer to accurately operate the external device .
- the external device is a keyboard
- the operation prompt information corresponding to the external device may be prompt information related to an input method.
- the external device is a remote control device including a joystick and/or buttons
- the operation prompt information corresponding to the external device may be the control logic of the movable platform corresponding to different operation modes of the joystick and/or buttons.
- the wearer's operation can also be superimposed and displayed in the display image of the head-mounted display device.
- the operation data generated by the external device so that the user can confirm the accuracy of the operation data.
- the head-mounted display device communicates with other devices, and the display image of the head-mounted display device and the image area corresponding to the external device can also be sent to the remote device, so that other devices can The display image superimposed with the image area corresponding to the external device is displayed in .
- the head-mounted display device or the target device corresponding to the head-mounted display device can also send the operation data generated by the wearer's operation of the external device to other devices, so that other devices display on the head-mounted display device The operation data is superimposed on the image.
- a user may wear a head-mounted display device to participate in an online conference, and the head-mounted display device may directly or indirectly communicate with other devices belonging to the same conference channel, such as
- the head-mounted display device can communicate with other devices belonging to the same conference channel through the server, and all or part of the content displayed on the head-mounted display device can also be correspondingly displayed on other devices.
- the anchor user can wear a head-mounted display device for live broadcast
- the head-mounted display device can communicate with the live broadcast server
- the head-mounted display device can display images, superimpose The image area or operation data corresponding to the external device of the host is sent to the live broadcast server
- the live broadcast server can send the content displayed on the head-mounted display device to the audience client who enters the live broadcast channel where the host is located, and the head-mounted display device All or part of the displayed content can also be correspondingly displayed in the viewer client.
- the external device is an input device (such as a keyboard) as an example for illustration:
- the head-mounted display device may include one or more cameras, which are placed on the outside of the head-mounted display device and Opposite to the wearing part of the head-mounted display device, it is used to photograph the physical environment in front of or below the wearer.
- the head-mounted display device includes a posture sensor and an eye-tracking sensor. The head-mounted display device can determine the wearer's head posture based on the data collected by the posture sensor, and determine the wearer's eye gaze range based on the data collected by the eye-tracking sensor. .
- the head-mounted display device determines that the wearer is lowering his head based on the wearer's head posture and eye gaze range
- the display content within the eye gaze range is a text input component
- the current display content has a text input component
- the head-mounted display device can recognize the original image captured by the camera to obtain the image position corresponding to the external device, and then extract the keyboard image area from the original image according to the image position corresponding to the external device, and display it on the head-mounted display
- the keyboard image area is superimposed on the display image of the device; the keyboard image area seen by the wearer is consistent with the scale and shape of the keyboard in the physical environment, so that the user can operate the keyboard image in the physical environment based on the seen keyboard image. keyboard.
- the head-mounted display device can recognize information such as the wearer's finger movements and/or operated keys based on the original image captured by the camera, and obtain the text content entered by the wearer's keyboard in the physical environment.
- the embodiment of the present application also provides another interaction method for a head-mounted display device, which includes:
- step S201 it is detected whether the wearer of the head-mounted display device intends to operate the virtual device.
- step S202 in response to detecting that the wearer intends to operate a virtual device, superimpose and display a virtual device in the display image of the head-mounted display device, the virtual device is based on the head-mounted display device It is generated by the virtual device identification obtained by the camera shooting.
- the virtual device is superimposed and displayed on the display image of the head-mounted display device by means of the virtual device identifier set in the physical environment.
- the virtual device is superimposed and displayed only when it is detected that the wearer has the intention to operate the virtual device, and there is no need to display the virtual device at all times. Avoid blocking the display image of the head-mounted display device.
- the virtual device logo can be a pattern, which can be set on different types of carriers, such as printed matter or the screen of an electronic device.
- the camera captures the pattern, it can be expanded on the reference surface corresponding to the pattern.
- a virtual device so that the user can "see” the virtual device and operate it.
- the virtual device is a virtual input device, and the virtual input device includes at least one of a virtual keyboard, a virtual mouse, a virtual touch screen, and a virtual tablet; or, the virtual device is a virtual remote control device , the head-mounted display device communicates with the movable platform, the head-mounted display device can generate control instructions based on user operations on the virtual device, and directly or indirectly transmit the control instructions to the movable platform, the Images captured by the movable platform can be displayed on the head-mounted display device.
- the detecting whether the wearer of the head-mounted display device intends to operate the virtual device includes: whether the operator has the intention to operate the virtual device.
- the head posture indicates the wearer's head-down posture and/or the display content within the gaze range of the eyes includes a virtual component corresponding to the virtual device, it is determined that the wearer has an operation The intent of the virtual device.
- the head posture is determined through the data collected by the posture sensor arranged in the head-mounted display device; and/or, the gaze range of the eyes is determined through the The data collected by the eye tracking sensor is determined.
- the display position of the virtual device in the display image is determined at least according to the relative positional relationship between the wearer and the virtual device identifier, so that the virtual device and the wearer in the display space
- the relative positional relationship between the physical environment and the relative positional relationship between the wearer and the virtual device logo in the physical environment is adapted to provide a reference surface for the virtual device in the physical environment, ensuring that the user can accurately locate the device by means of the reference surface Virtual equipment, and realize the precise operation of virtual equipment, and improve the accuracy of operation.
- the display position of the virtual device in the display image is based on the relative positional relationship between the wearer's eyes and the display of the head-mounted display device, and the relationship between the camera and the display.
- the relative positional relationship between the cameras and the relative positional relationship between the camera and the virtual device is determined.
- the camera is arranged on the outside of the head-mounted display device, facing away from the wearing part of the head-mounted display device.
- the camera is arranged on the head-mounted display device and is rotatably connected to the head-mounted display device.
- the method further includes: controlling the rotation of the camera so that a lens assembly of the camera faces the virtual device identification.
- the virtual device is superimposed and displayed in the display image in a semi-transparent or semi-opaque form.
- the transparency of the virtual device is determined according to the color system of the display image in the overlay area.
- the transparency corresponding to the color system of the displayed image in the superimposed area belonging to the dark color system is smaller than the transparency corresponding to the color system of the displayed image in the superimposed area belonging to the light color system.
- the head-mounted display device includes a supplementary light; the method further includes: if the brightness of the environment where the virtual device is identified is lower than a preset brightness threshold, turning on the supplementary light.
- the brightness of the supplementary light is adjusted according to the brightness of the environment where the virtual device identifier is located, and the brightness of the supplementary light is negatively correlated with the brightness of the environment where the virtual device identifier is located.
- the image captured by the camera also includes the image area corresponding to the designated part of the wearer; the method further includes: identifying from the image captured by the camera of the head-mounted display device The operating position of the designated part of the wearer on the virtual device; obtaining the operation data generated by the wearer operating the virtual device according to the content indicated by the operating position.
- the method further includes: superimposing and displaying the operation data generated by the wearer operating the virtual device in the display image of the head-mounted display device.
- the method further includes: superimposing and displaying operation prompt information corresponding to the virtual device in the display image of the head-mounted display device.
- the wearer's head posture indicates the wearer's head-up posture
- the wearer's eyes The displayed content within the gazing range does not include the virtual component corresponding to the virtual device, or the specified part of the wearer does not exist in the image captured by the camera.
- the displayed image includes a virtual image and/or an image captured from a physical environment.
- the head-mounted display device is connected in communication with a movable platform equipped with a photographing device; the images captured from the physical environment include images collected by the movable platform in real time.
- the embodiment of the present application also provides a head-mounted display device 100, including:
- processors 101 one or more processors 101;
- a memory 102 for storing instructions executable by the processor 101;
- the one or more processors 101 individually or jointly execute the executable instructions, so as to perform any one of the methods described above.
- the processor 101 executes the executable instructions included in the memory 102, and the processor 101 can be a central processing unit (Central Processing Unit, CPU), and can also be other general-purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuit (Application Specific Integrated Circuit, ASIC), off-the-shelf programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.
- a general-purpose processor may be a microprocessor, or the processor may be any conventional processor, or the like.
- the memory 102 stores executable instructions of the interaction method of the head-mounted display device, and the memory 102 may include at least one type of storage medium, and the storage medium includes a flash memory, a hard disk, a multimedia card, a card-type memory (for example, SD or DX memory, etc.), Random Access Memory (RAM), Static Random Access Memory (SRAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), Programmable Read Only Memory (PROM), Magnetic storage, magnetic disks, optical disks, etc. Also, the device may cooperate with a network storage device that performs a storage function of the memory through a network connection.
- RAM Random Access Memory
- SRAM Static Random Access Memory
- ROM Read Only Memory
- EEPROM Electrically Erasable Programmable Read Only Memory
- PROM Programmable Read Only Memory
- Magnetic storage magnetic disks, optical disks, etc.
- the device may cooperate with a network storage device that performs a storage function of the memory through a network connection.
- the memory 102 may be an internal storage unit of the head-mounted display device 100 , such as a hard disk or memory of the head-mounted display device 100 .
- the memory 102 can also be an external storage device of the head-mounted display device 100, such as a plug-in hard disk equipped on the head-mounted display device 100, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, flash memory card (Flash Card), etc.
- the memory 102 may also include both an internal storage unit of the head-mounted display device 100 and an external storage device.
- the memory 102 is used to store computer programs and data.
- the memory 102 can also be used to temporarily store data that has been output or will be output.
- Displays include but are not limited to LCD (Liquid Crystal Display) displays, LED (Light Emitting Diode) displays, or OLED (Organic Light-Emitting Diode, Organic Light-Emitting Diode) displays.
- LCD Liquid Crystal Display
- LED Light Emitting Diode
- OLED Organic Light-Emitting Diode, Organic Light-Emitting Diode
- the processor is configured to: detect whether the wearer of the head-mounted display device intends to operate an external device; in response to detecting that the wearer intends to operate an external device, The image captured by the camera of the head-mounted display device is superimposed on the displayed image of the display, and the image captured by the camera includes the image area corresponding to the external device.
- the external device is an input device, and the input device includes at least one of a keyboard, a mouse, a touch screen, and a tablet; or, the external device is a remote control device, and the remote control device can directly or indirectly
- the control instruction is transmitted to the movable platform, and the image captured by the movable platform can be displayed on the head-mounted display device.
- the processor is further configured to detect whether the wearer intends to operate the external device according to the wearer's head posture and/or eye gaze range. Exemplarily, if the head posture indicates the wearer's head-down posture and/or the display content within the gaze range of the eyes includes a virtual component corresponding to the external device, it is determined that the wearer has an operation The intent of the external device.
- the head-mounted display device includes an attitude sensor and/or an eye tracking sensor.
- the head posture is determined by data collected by a posture sensor arranged in the head-mounted display device; and/or, the gaze range of the eyes is collected by an eye tracking sensor arranged in the head-mounted display device The data is confirmed.
- the processor is further configured to obtain an original image captured by the camera of the head-mounted display device; and according to the image position corresponding to the external device in the original image, crop the original image to obtain A cropped image, where the cropped image includes the image area corresponding to the external device; and the cropped image is superimposed and displayed on the displayed image in the display.
- the display position of the image area corresponding to the external device in the display image is determined at least according to a relative positional relationship between the wearer and the external device.
- the display position of the image area corresponding to the external device in the display image is based on the relative positional relationship between the wearer's eyes and the display of the head-mounted display device, the camera and the The relative positional relationship between the displays and the relative positional relationship between the camera and the external device are determined.
- the camera is arranged outside the head-mounted display device, facing away from the wearing part of the head-mounted display device.
- the camera is arranged on the head-mounted display device and is rotatably connected to the head-mounted display device.
- the processor is further configured to: control the rotation of the camera so that the lens assembly of the camera faces the external device.
- the image captured by the camera is superimposed and displayed in the display image in a semi-transparent or semi-opaque form.
- the transparency of the image captured by the camera is determined according to the color system of the displayed image in the superimposed area.
- the color system of the display image in the overlay area belongs to the transparency corresponding to the dark color system, which is less than the transparency corresponding to the color system of the display image in the overlay area belongs to the light color system.
- the head-mounted display device includes a supplementary light; the processor is further configured to turn on the supplementary light if the brightness of the environment where the external device is located is less than a preset brightness threshold.
- the brightness of the supplementary light is adjusted according to the brightness of the environment where the external device is located, and the brightness of the supplementary light is negatively correlated with the brightness of the environment where the external device is located.
- the image captured by the camera further includes an image area corresponding to the designated part of the wearer; the processor is further configured to identify the The operating position of the designated part of the wearer on the external device; and the operation data generated by the wearer operating the external device is acquired according to the content indicated by the operating position.
- the external device is communicatively connected to the head-mounted display device; the processor is further configured to acquire operation data generated by the wearer operating the external device through the communication connection.
- the external device is communicatively connected to a target device corresponding to the head-mounted display device; the target device is configured to acquire operation data generated by the wearer operating the external device through the communication connection.
- the target device includes a terminal device or a server.
- the processor is further configured to superimpose and display the operation data generated by the wearer operating the external device in the display image of the head-mounted display device.
- the processor is further configured to superimpose and display operation prompt information corresponding to the external device in the display image of the head-mounted display device.
- the processor is further configured to not display the image captured by the camera of the head-mounted display device in response to the wearer not intending to operate the external device.
- the processor is further configured to determine that the wearer has no intention to operate the external device if any one or more of the following conditions are met: the wearer's head posture indicates the wearer's head-up posture , the displayed content within the gaze range of the wearer's eyes does not include the virtual component corresponding to the external device, or there is no designated part of the wearer in the image captured by the camera.
- the displayed image includes a virtual image and/or an image captured from a physical environment.
- the head-mounted display device is communicatively connected to a movable platform equipped with a photographing device; the images captured from the physical environment include images collected by the movable platform in real time.
- Various implementations described herein can be implemented using a computer readable medium such as computer software, hardware, or any combination thereof.
- the embodiments described herein can be implemented by using Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays ( FPGA), processors, controllers, microcontrollers, microprocessors, electronic units designed to perform the functions described herein.
- ASICs Application Specific Integrated Circuits
- DSPs Digital Signal Processors
- DSPDs Digital Signal Processing Devices
- PLDs Programmable Logic Devices
- FPGA Field Programmable Gate Arrays
- processors controllers, microcontrollers, microprocessors, electronic units designed to perform the functions described herein.
- an embodiment such as a procedure or a function may be implemented with a separate software module that allows at least one function or operation to be performed.
- the software codes can be implemented by a software application (or program
- the embodiment of the present application also provides an interactive system, including a head-mounted display device 100 and an external device 200 .
- the external device is an input device, and the input device includes at least one of a keyboard, mouse, touch screen, and tablet; or, please refer to Figure 4, the external device is a remote control device, the remote control device can directly or indirectly transmit control commands to the movable platform, and the images captured by the movable platform can be displayed on the head-mounted display device.
- the system further includes a movable platform, and the movable platform is communicatively connected with the head-mounted display device.
- non-transitory computer-readable storage medium including instructions, such as a memory including instructions, which are executable by a processor of an apparatus to perform the above method.
- the non-transitory computer readable storage medium may be ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like.
- a non-transitory computer-readable storage medium when the instructions in the storage medium are executed by the processor of the terminal, the terminal can execute the above method.
- relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is a relationship between these entities or operations. There is no such actual relationship or order between them.
- the term "comprises”, “comprises” or any other variation thereof is intended to cover a non-exclusive inclusion such that a process, method, article or apparatus comprising a set of elements includes not only those elements but also other elements not expressly listed elements, or also elements inherent in such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase “comprising a " does not exclude the presence of additional identical elements in the process, method, article or apparatus comprising said element.
Landscapes
- Engineering & Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computer Graphics (AREA)
- Computer Hardware Design (AREA)
- Software Systems (AREA)
- Human Computer Interaction (AREA)
- Controls And Circuits For Display Device (AREA)
- User Interface Of Digital Computer (AREA)
Abstract
一种交互方法、头戴式显示设备、***及存储介质。交互方法包括:检测所述头戴式显示设备的佩戴者是否有操作外部设备的意图;响应于检测到所述佩戴者有操作外部设备的意图,在所述头戴式显示设备的显示图像中叠加显示所述头戴式显示设备的摄像头拍摄得到的图像,所述摄像头拍摄得到的图像包括所述外部设备对应的图像区域。实现头戴式显示设备的佩戴者能够在"看到"外部设备的情况下操作外部设备,提高外部设备的操作便捷性,并且在检测到佩戴者有操作外部设备的意图的情况下才叠加显示外部设备对应的图像区域,避免遮挡所述头戴式显示设备的显示图像。
Description
本申请涉及头戴式显示设备领域,具体而言,涉及一种交互方法、头戴式显示设备、***及存储介质。
随着技术的发展以及生活水平的提高,头戴式显示设备逐渐普及。头戴式显示设备的外型通常是眼罩或头盔的形式,把显示屏贴近用户的眼睛,近距离对用户的眼睛投射画面。头戴式显示设备又分为可透光和不可透光两种,对于不可透光的头戴式显示设备,用户穿戴了头戴式显示设备后就失去了眼睛对于外界世界的感知,在用户想要操作设备时会产生诸多不便。
发明内容
有鉴于此,本申请的目的之一是提供一种交互方法、头戴式显示设备、***及存储介质。
第一方面,在一些场景中,在用户穿戴头戴式显示设备时,存在操作一些外部设备的需求,而该情况下用户无法看到物理环境中的外部设备,加之用户可能对外部设备的操作方式不熟悉,使得用户操作外部设备的便捷性较差。
因此,本申请实施例提供了一种头戴式显示设备的交互方法,所述方法包括:
检测所述头戴式显示设备的佩戴者是否有操作外部设备的意图;
响应于检测到所述佩戴者有操作外部设备的意图,在所述头戴式显示设备的显示图像中叠加显示所述头戴式显示设备的摄像头拍摄得到的图像,所述摄像头拍摄得到的图像包括所述外部设备对应的图像区域。
本实施例中,能够在头戴式显示设备的显示图像中叠加显示外部设备对应的图像区域,则头戴式显示设备的佩戴者能够在“看到”外部设备的情况下操作外部设备,提高外部设备的操作便捷性;进一步地,在检测到佩戴者有操作外部设备的意图的情 况下才叠加显示外部设备对应的图像区域,无需时刻显示外部设备对应的图像区域,避免遮挡所述头戴式显示设备的显示图像。
第二方面,在一些场景中,在用户穿戴头戴式显示设备时,存在操作一些虚拟设备的需求,比如通过手指定位点按虚拟设备实现数据输入,由于虚拟设备的位置不确定,用户对于虚拟设备的操作位置把握不准,使得用户操作虚拟设备的便捷性较差。
因此,本申请实施例提供了一种头戴式显示设备的交互方法,所述方法包括:
检测所述头戴式显示设备的佩戴者是否有操作虚拟设备的意图;
响应于检测到所述佩戴者有操作虚拟设备的意图,在所述头戴式显示设备的显示图像中叠加显示虚拟设备,所述虚拟设备是基于所述头戴式显示设备的摄像头拍摄得到的虚拟设备标识生成的。
本实施例中,借助于在物理环境中设置的虚拟设备标识在头戴式显示设备的显示图像中叠加显示虚拟设备,虚拟设备标识可以提供一参考面从而让用户对应了解到虚拟设备在物理环境中所在位置,从而实现对虚拟设备进行精准操作,提高虚拟设备的操作便捷性;进一步地,在检测到佩戴者有操作虚拟设备的意图的情况下才叠加显示虚拟设备,无需时刻显示虚拟设备,避免遮挡所述头戴式显示设备的显示图像。
第三方面,本申请实施例提供了一种头戴式显示设备,包括:
一个或多个处理器;
用于存储所述处理器可执行指令的存储器;
用于显示图像的显示器;
所述一个或多个处理器单独或者共同执行所述可执行指令,以执行如第一方面或第二方面所述的方法。
第四方面,本申请实施例提供了一种交互***,包括如第三方面所述的头戴式显示设备和外部设备。
第五方面,本申请实施例提供了一种计算机可读存储介质,所述计算机可读存储介质存储有可执行指令,所述可执行指令被处理器执行时实现如第一方面或第二方面所述的方法。
为了更清楚地说明本申请实施例中的技术方案,下面将对实施例描述中所需要使用的附图作简单地介绍,显而易见地,下面描述中的附图仅仅是本申请的一些实施例,对于本领域普通技术人员来讲,在不付出创造性劳动性的前提下,还可以根据这些附 图获得其他的附图。
图1是本申请实施例提供的一种头戴式显示设备和外部设备的使用场景图;
图2是本申请实施例提供的一种交互***的结构示意图;
图3是本申请实施例提供在头戴式显示设备的显示图像中叠加键盘对应的图像区域的示意图;
图4是本申请实施例提供的另一种交互***的结构示意图;
图5是本申请实施例提供的一种图4所示的交互***的使用场景示意图;
图6是本申请实施例提供的在头戴式显示设备的显示图像中叠加遥控设备对应的图像区域的示意图;
图7是本申请实施例提供的一种头戴式显示设备的交互方法的流程示意图;
图8是本申请实施例提供的另一种头戴式显示设备的交互方法的流程示意图;
图9是本申请实施例提供的头戴式显示设备的结构示意图。
下面将结合本申请实施例中的附图,对本申请实施例中的技术方案进行清楚、完整地描述,显然,所描述的实施例仅仅是本申请一部分实施例,而不是全部的实施例。基于本申请中的实施例,本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例,都属于本申请保护的范围。
相关技术的一些场景中,比如如图1所示,示出了头戴式显示设备100和外部设备200。在用户穿戴头戴式显示设备100时,存在用户需要操作一些外部设备200的需求,而该情况下用户无法看到物理环境中的外部设备,加之用户可能对外部设备的操作方式不熟悉,使得用户操作外部设备的便捷性较差。
其中,物理环境指的是人们在没有电子设备帮助的情况下能够感测和/或交互的物理世界。人们能够诸如通过视觉、触觉、听觉、味觉和嗅觉来直接感测物理环境和/或与物理环境交互。
基于相关技术中的问题,本申请实施例提供了一种头戴式显示设备的交互方法,可以通过头戴式显示设备的摄像头拍摄外部设备,拍摄得到的图像中包括有外部设备对应的图像区域,进而能够在头戴式显示设备的显示图像中叠加显示外部设备对应的图像区域,则头戴式显示设备的佩戴者能够在“看到”外部设备的情况下操作外部设备,提高外部设备的操作便捷性。进一步地,发明人发现,如果所述外部设备对应的 图像区域一直在头戴式显示设备的显示图像上叠加显示,会对头戴式显示设备的显示图像造成持续遮挡,影响用户的观看体验,因此,本申请实施例进一步实现在检测到佩戴者有操作外部设备的意图的情况下,才在所述头戴式显示设备的显示图像中叠加外部设备对应的图像区域,无需时刻显示外部设备对应的图像区域,避免遮挡所述头戴式显示设备的显示图像,保证在减少影响所述头戴式显示设备的显示图像的正常显示的情况下提高外部设备的操作便捷性。
在一些实施例中,本申请实施例提及的头戴式显示设备包括但不限于智能眼镜,设置有显示器的头盔、护目镜或者平视显示器等等。所述头戴式显示设备的显示图像可以是虚拟图像、也可以是从物理环境拍摄得到的图像,还可以是虚拟图像和从物理环境拍摄得到的图像叠加后的图像。
在一示例性的实施例中,头戴式显示设备包括眼镜主体,所述眼镜主体的一端设有用于与使用者配合的佩戴部,所述眼镜本体的另一端至少设有显示器。在用户佩戴所述头戴式显示设备之后,所述显示器能够在佩戴者的视野范围内显示图像,所述显示器显示的图像包括但不限于虚拟图像和/或从物理环境拍摄得到的图像。
头戴式显示设备还可以设置有面向内或者面向外的传感器/装置,实现为佩戴者提供良好的交互效果。示例性的,头戴式显示设备还包括设于眼镜主体的摄像头,所述摄像头能够对物理环境中的外部设备进行拍摄,得到包括外部设备的图像;其中,所述摄像头包括不限于彩色摄像头、单色摄像头、黑白摄像头、3D深度感测摄像头、立体摄像头或者飞行时间(ToF)摄像头等。示例性的,头戴式显示设备还包括设于眼镜主体的姿态传感器,姿态传感器用于检测佩戴者的头部姿态,例如可以根据佩戴者的头部姿态确定显示器显示的内容。示例性的,头戴式显示设备还包括眼球跟踪传感器,眼球跟踪传感器用于检测佩戴者的眼球运动情况,例如可以根据佩戴者的眼球运动情况确定出佩戴者的眼部注视范围,进而确定显示器对应显示的内容。
在一示例性的应用场景中,所述外部设备可以是头戴式显示设备的输入设备,相关技术中由于用户无法看到物理环境中的外部设备,为了方便用户操作,相关技术中头戴式显示设备的输入设备可能是一些操作方式较为简单的输入设备,比如该输入设备可能仅包括两个或者三个具有指定功能的操作控件,第一输入设备的输入功能有限。而本申请实施例在实现用户可以“看到”外部设备的情况下,则所述外部设备除了可以包括操作方式较为简单的输入设备,还可以包括具有多种不同输入功能的输入设备,从而满足用户多样化的输入需求。示例性的,头戴式显示设备的输入设备包括但不限于键盘、鼠标、触控屏、触控笔、摇杆(比如游戏操纵杆)和手写板中的至少一个, 多样化的输入设备可以满足用户的个性化输入需求。
在一个例子中,请参阅图2以及图3,图2示出了一种交互***,所述交互***包括头戴式显示设备100和键盘201。图3示出了在头戴式显示设备100的显示图像20中叠加键盘201对应的图像区域的示意图。用户在穿戴头戴式显示设备100的过程中,不仅可以看到头戴式显示设备的显示图像20,并在存在操作外部设备的需求时,头戴式显示设备100响应于检测到佩戴者有操作外部设备的意图,获取摄像头拍摄的键盘201图像,该图像包括键盘201对应的图像区域10,进而可以将键盘201对应的图像区域10叠加在头戴式显示设备100的显示图像20中,使得佩戴者在有操作键盘201的意图的情况下可以“看到”物理环境中的键盘201,实现对键盘201的便捷性操作,提高了佩戴者的使用体验。
在另一示例性的实施例中,所述外部设备可以是与头戴式显示设备相关的设备,比如所述外部设备可以是可移动平台的遥控设备。可移动平台设置有拍摄装置,所述遥控设备能够对可移动平台的运动过程和/或拍摄装置的拍摄过程进行控制,例如所述遥控设备能够在用户的操纵下直接或间接的向可移动平台传输控制指令,以控制可移动平台的运动方向或者运动速度等。可移动平台还与头戴式显示设备通信连接,所述可移动平台能够将拍摄装置拍摄的图像传输给头戴式显示设备,进而所述可移动平台基于所述拍摄装置拍摄的图像能够显示于所述头戴式显示设备中。其中,所述可移动平台包括但不限于车辆、无人机(UAV)、无人船或者移动机器人等等。
这里以无人机为例进行示例性说明:请参阅图4以及图5,图4示出了一种交互***,所述交互***包括头戴式显示设备100、遥控设备202和无人机301。图5示出了图4所述的交互***的使用场景。其中,无人机301设置有拍摄装置,头戴式显示设备100设置有摄像头。所述遥控设备202可以基于头戴式显示设备100的佩戴者的操作,直接或间接的向无人机301传输控制指令,以控制无人机301飞行和/或无人机301中的拍摄装置拍摄图像。无人机301可以在飞行过程中拍摄飞行途中的画面并传输给头戴式显示设备100。头戴式显示设备100能够显示无人机301拍摄的图像。其中,为了让用户能够便捷性操作遥控设备202以实现对无人机301的精准控制,进而在头戴式显示设备100中看到期望的显示图像,头戴式显示设备100可以响应于检测到佩戴者有操作遥控设备202的意图,获取摄像头拍摄的图像,该图像中包括有遥控设备202对应的图像区域。请参阅图6,头戴式显示设备100可以将遥控设备202对应的图像区域10叠加在头戴式显示设备的显示图像20中,使得佩戴者在有操作遥控设备202的意图的情况下可以“看到”物理环境中的遥控设备202,以实现对遥控设 备202的便捷性操作,提高了佩戴者的使用体验。
在一些实施例中,请参阅图7,图7为本申请实施例提供的一种头戴式显示设备的交互方法的流程示意图,所述方法可由头戴式显示设备来执行,所述方法包括:
在步骤S101中,检测所述头戴式显示设备的佩戴者是否有操作外部设备的意图。
在步骤S102中,响应于检测到所述佩戴者有操作外部设备的意图,在所述头戴式显示设备的显示图像中叠加显示所述头戴式显示设备的摄像头拍摄得到的图像,所述摄像头拍摄得到的图像包括所述外部设备对应的图像区域。
在一些实施例中,在用户穿戴头戴式显示设备的过程中,头戴式显示设备可以在用户的视野范围显示图像。其中,头戴式显示设备的显示图像可以是虚拟图像(比如如图3所示)、也可以是从物理环境拍摄得到的图像、或者还可以是虚拟图像和从物理环境拍摄得到的图像叠加后的图像。示例性,所述虚拟图像可以是基于预设的建模规则,利用成像几何原理生成;比如虚拟图像包括但不限于虚拟游戏画面、虚拟建筑画面或者虚拟人物画面等等。示例性的,头戴式显示设备与搭载有拍摄装置的可移动平台通信连接,可移动平台包括但不限于无人机、车辆、船只或者移动机器人等等;其中,所述从物理环境拍摄得到的图像包括所述可移动平台的拍摄装置实时采集的图像或者在历史时间段内采集的图像或者视频帧。示例性的,所述从物理环境拍摄得到的图像可以是从第三方平台中获取的图像,比如可以是其他用户分享到社交平台上的图像或者视频帧。
在一些实施例中,在用户穿戴头戴式显示设备的过程中,头戴式显示设备可以检测所述头戴式显示设备的佩戴者是否有操作外部设备的意图,从而确定是否要在头戴式显示设备的显示图像中叠加显示外部设备。示例性的,在检测到用户穿戴头戴式显示设备后,头戴式显示设备可以实时检测佩戴者是否有操作外部设备的意图,也可以间歇性检测佩戴者是否有操作外部设备的意图。
在一些可能的实现方式中,头戴式显示设备可以根据佩戴者的头部姿态和/或眼部注视范围,检测所述佩戴者是否有操作外部设备的意图。示例性的,头戴式显示设备安装有姿态传感器,可以通过姿态传感器采集的数据确定佩戴者的头部姿势;姿态传感器可以包含三轴陀螺仪、三轴加速度计以及三轴电子罗盘等运动传感器。示例性的,头戴式显示设备可以安装有眼球跟踪传感器,可以通过眼球跟踪传感器采集的数据确定佩戴者的眼部注视范围。
示例性的,为了方便用户操作外部设备,外部设备所在高度一般低于头戴式显示设备的高度,用户在想要操作外部设备时,可能会下意识低头以确定外部设备所在位 置。因此,如果头戴式显示设备检测到所述头部姿态指示佩戴者的低头姿势,可以确定佩戴者具有操作外部设备的意图。
示例性的,头戴式显示设备在进行视觉信息显示时,根据眼球跟踪传感器采集的数据和/或姿态传感器采集的数据确定佩戴者的眼部注视范围,进而获取与眼部注视范围对应的显示内容呈现给用户。用户可以通过转动头部或者眼球以改变眼部注视范围,从而观看到不同的显示内容。因此,可以设置与外部设备对应的虚拟组件,该虚拟组件可以在头戴式显示设备中进行显示;佩戴者想要操作外部设备时,可以通过转动头部或者眼球以改变眼部注视范围,找到显示有虚拟组件的显示内容,则头戴式显示设备如果检测到在所述眼部注视范围内的显示内容包括与所述外部设备对应的虚拟组件,可以确定佩戴者具有操作外部设备的意图。示例性的,所述外部设备为输入设备,则与外部设备对应的虚拟组件可以是输入虚拟组件。示例性的,所述外部设备为遥控设备,则与外部设备对应的虚拟组件可以是虚拟遥控控件。
当然,为了提高检测准确性,可以在检测到所述头部姿态指示佩戴者的低头姿势以及在所述眼部注视范围内的显示内容包括与所述外部设备对应的虚拟组件两者均成立的情况下,确定佩戴者具有操作外部设备的意图。
在其他的可能的实现方式中,头戴式显示设备设置有指定控件,佩戴者在需要操作外部设备的情况下,可以通过操作外部设备以向头戴式显示设备传达自己的需求。头戴式显示设备可以响应于该指定控件的触发,确定佩戴者具有操作外部设备的意图,进而控制设置于头戴式显示设备中的摄像头拍摄图像。
在一些实施例中,头戴式显示设备可以响应于所述佩戴者无操作外部设备的意图,不显示所述头戴式显示设备的摄像头拍摄得到的图像。本实施例中,在佩戴者无操作外部设备的意图的情况下,不显示外部设备对应的图像区域,避免遮挡所述头戴式显示设备的显示图像。示例性的,头戴式显示设备可以在检测到以下任一种或多种情况时,确定所述佩戴者无操作外部设备的意图:所述佩戴者的头部姿态指示所述佩戴者的抬头姿势、所述佩戴者的眼部注视范围内的显示内容不包括与所述外部设备对应的虚拟组件、或者所述摄像头拍摄得到的图像中不存在所述佩戴者的指定部位。
在一可能的实施方式中,比如头戴式显示设备在显示图像中叠加显示所述外部设备对应的图像区域期间,可以周期性检测所述头戴式显示设备的佩戴者是否有操作外部设备的意图,如果检测到佩戴者无操作外部设备的意图,则停止显示所述外部设备对应的图像区域,从而用户可以看到显示图像的完整内容。
在一些实施例中,头戴式显示设备的摄像头可以固定设置在所述头戴式显示设备 中。示例性的,头戴式显示设备设置有与使用者配合的佩戴部,所述摄像头设置在所述头戴式显示设备的外侧,并与所述头戴式显示设备的佩戴部相背,从而可以对物理环境进行拍摄,获得包括外部设备的图像。示例性的,考虑到在用户佩戴头戴式显示设备时,综合用户操作外部设备的习惯,外部设备所在高度一般低于头戴式显示设备的高度,则所述摄像头可以设置在所述头戴式显示设备的外侧,且摄像头的视场方向可以倾斜朝下(比如摄像头的视场方向与重力方向呈预设夹角,且该夹角小于90度),实现拍摄包括外部设备的图像;当然,在摄像头的视场范围较大的情况下,比如摄像头为鱼眼摄像头,则摄像头在头戴式显示设备外侧任意设置均能拍摄到包括外部设备的图像,则本实施例对摄像头在头戴式显示设备外侧的设置方式也可以不做任何限制。
在另一些实施例中,考虑到外部设备所处位置的不确定性,所述摄像头可以设置于所述头戴式显示设备,且与所述头戴式显示设备可转动连接。示例性的,所述摄像头可以在人为作用下进行转动(比如所述摄像头通过旋钮实现与头戴式显示设备可转动连接),或者所述摄像头也可以在驱动部件的作用下实现自动转动。
以所述摄像头可实现自动转动为例,头戴式显示设备可以控制所述摄像头转动以使得所述摄像头的镜头组件朝向所述外部设备,使得所述摄像头可以拍摄到包括外部设备的图像,以便头戴式显示设备响应于所述佩戴者有操作外部设备的意图,在所述头戴式显示设备的显示图像中叠加显示所述摄像头拍摄到的图像,该图像包括外部设备对应的图像区域。
示例性的,头戴式显示设备可以预存外部设备的参考信息,在转动所述摄像头的过程中,头戴式显示设备可以根据所述外部设备的参考信息检查摄像头拍摄的图像中是否有外部设备,在确定摄像头拍摄的图像中包括有外部设备时,控制摄像头停止转动。
在一些实施例中,为了获取显示效果更佳的包括外部设备的图像,所述头戴式显示设备包括一个或多个补光灯。头戴式显示设备的摄像头在拍摄图像的过程中,头戴式显示设备可以检测所述外部设备所处环境的亮度,在所述外部设备所处环境的亮度小于预设亮度阈值的情况下,开启所述补光灯。可以理解的是,所述预设亮度阈值可依据实际应用场景进行具体设置,本实施例对此不做任何限制,比如预设亮度阈值可依据摄像头在进行图像采集时的亮度要求所确定。本实施例通过在适当时机开启补光灯,使得外部设备所处环境的亮度能够符合摄像头的亮度要求,从而获得显示效果更佳的图像,也提高了图像中的外部设备的可识别性。
示例性的,对于所述外部设备所处环境的亮度的检测,可以根据摄像头中的感光 元件接收到的光子数量和/或摄像头采集的图像中像素的像素值来确定所述环境光亮度。其中,所述光子数量与外部设备所处环境的亮度呈正相关关系,即所述光子数量越多,外部设备所处环境的亮度越大。所述像素的像素值与外部设备所处环境的亮度也呈正相关关系,比如所述像素的像素值以灰度值表示,灰度值越高则表示外部设备所处环境的亮度越大。可以理解的是,可以根据光子数量和所述图像中像素的像素值中的任一者来确定外部设备所处环境的亮度,也可以根据两者来共同确定外部设备所处环境的亮度,本实施例对此不做任何限制,可依据实际应用场景进行具体设置。
在一些可能的实施方式中,所述补光灯的亮度可以根据所述外部设备所处环境的亮度进行调整,比如所述补光灯的亮度与所述外部设备所处环境的亮度成负相关关系,即外部设备所处环境的亮度越暗,所述补光灯的亮度越亮,反之,外部设备所处环境的亮度越亮,补光灯的亮度越暗,从而可以避免或者减少补光灯的亮度对旁人造成影响。
在一些可能的实施方式中,在补光灯的数量有多个的情况下,头戴式显示设备可以根据外部设备所处环境的亮度和预设亮度阈值之间的差异,确定需要开启的补光灯的数量。比如在所述外部设备所处环境的亮度小于预设亮度阈值时,需要开启的补光灯的数量与所述差异成正相关关系,即差异越大,需要开启的补光灯的数量越多,实现辅助摄像头拍摄到符合亮度要求的图像,差异越小,需要开启的补光灯的数量越少,可以避免或者减少补光灯的亮度对旁人造成影响。
在一些实施例中,考虑到摄像头拍摄得到的图像中可能包含了用户不关注的其他内容,如果将摄像头拍摄得到的图像都叠加在头戴式显示设备的显示图像中,可能会对头戴式显示设备的显示图像造成大面积遮挡,不利于用户的观看体验。因此,头戴式显示设备在获取摄像头拍摄的原始图像之后,可以对原始图像进行识别以确定所述外部设备对应的图像位置,然后根据所述原始图像中所述外部设备对应的图像位置,对所述原始图像进行裁剪以得到裁剪后的图像,所述裁剪后的图像包括所述外部设备对应的图像区域,进而可以在所述显示图像中叠加显示所述裁剪后的图像。本实施例中,仅在头戴式显示设备的显示图像中叠加显示所述外部设备对应的图像区域,避免原始图像中的无效内容对头戴式显示设备的显示图像造成遮挡,提高用户的观看体验。
在一些实施例中,所述外部设备对应的图像区域在所述显示图像中的显示位置至少根据所述佩戴者与所述外部设备之间的相对位置关系确定,从而可以实现在显示空间中外部设备对应的图像区域与佩戴者的相对位置关系、与物理环境中佩戴者与所述外部设备之间的相对位置关系两者相适应,这样,所述外部设备对应的图像区域与物 理环境中的外部设备尺度一致,佩戴者可以基于“看到”的外部设备所在的显示位置对应地找到物理环境中的外部设备,以实现对物理环境中的外部设备的便捷性操作。
示例性的,为了提高显示位置的准确性,为了实现所述外部设备对应的图像区域与物理环境中的外部设备尺度一致,综合考虑用户的眼部、显示器、摄像头和外部设备各个因素之间的相关关系,则所述外部设备对应的图像区域在所述显示图像中的显示位置可以根据所述佩戴者的眼部与所述头戴式显示设备的显示器之间的相对位置关系、所述摄像头与所述显示器之间的相对位置关系、以及所述摄像头与所述外部设备之间的相对位置关系确定。这样,在显示空间中外部设备对应的图像区域与佩戴者眼部的相对位置关系、与物理环境中佩戴者的眼部与所述外部设备之间的相对位置关系两者相同,佩戴者可以基于“看到”的外部设备所在的显示位置对应地找到物理环境中的外部设备,以实现对物理环境中的外部设备的便捷性操作。
在一些实施例中,为了让用户在操作外部设备的过程中也可以看到头戴式显示设备的显示图像的完整内容,则所述摄像头拍摄得到的图像可以以半透明或者半不透明的形式叠加显示在所述显示图像中,这样用户也可以透过所述摄像头拍摄得到的图像看到头戴式显示设备的显示图像中被遮挡的部分,提高用户的观看体验。
在另一些实施例中,考虑到在显示图像的色系深浅程度不同的情况下,摄像头拍摄得到的图像如果均以固定的透明度叠加显示,可能无法适用于显示图像的不同显示情况;比如在显示图像的整体色调为深色系的情况下,固定的半透明度可能会使佩戴者无法看清所述摄像头拍摄得到的图像(即外部设备),操作外部设备可能受到影响;又比如在显示图像的整体色调为浅色系的情况下,固定的半透明度可能会使佩戴者无法看清显示图像中被遮挡的内容,影响用户的观看体验。因此,为了实现在观看显示图像的完整内容和观看摄像头拍摄得到的图像(即外部设备)两者之间达到平衡,本实施例实现可以设置摄像头拍摄得到的图像的透明度的自适应变化,所述摄像头拍摄得到的图像的透明度可以根据所述显示图像在叠加区域的色系确定;其中,所述叠加区域指的是显示图像和摄像头拍摄得到的图像在叠加显示时,显示图像被所述摄像头拍摄得到的图像遮挡的区域。
示例性的,显示图像在叠加区域的色系属于深色系对应的透明度,小于,显示图像在叠加区域的色系属于浅色系对应的透明度。换句话说,叠加区域的色系越深,所述摄像头拍摄得到的图像的透明度设置得越小,使得摄像头拍摄得到的图像越清晰,从而佩戴者可以看清外部设备对应的图像区域,实现便捷性的操作过程;叠加区域的色系越浅,所述摄像头拍摄得到的图像的透明度设置得越大,使得叠加区域(即显示 区域中被遮挡的区域)越清晰,从而佩戴者可以看清显示图像的完整内容。
在一些实施例中,本实施例对于外部设备的操作方式不做任何限制,比如所述外部设备可以由用户的手部操作,或者如所述外部设备是由用户的脚部来操作的,因此,在佩戴者使用指定部位(比如用手或者用脚)操作外部设备时,所述摄像头拍摄得到的图像还可以包括所述佩戴者的指定部位对应的图像区域。在一个例子中,请参阅图3,在佩戴者用手操作键盘时,叠加在显示图像中的还包括佩戴者的手部对应的图像区域。在另一个例子中,请参阅图6,在佩戴者用手操作遥控设备时,叠加在显示图像中的还包括所述佩戴者的手部对应的图像区域。则在一些可能的实施方式中,头戴式显示设备可以从所述头戴式显示设备的摄像头拍摄得到的图像中,识别所述佩戴者的指定部位在所述外部设备上的操作位置,然后根据所述操作位置指示的内容获取所述佩戴者操作所述外部设备产生的操作数据。
在另一些可能的实施方式中,所述外部设备与所述头戴式显示设备通信连接;则头戴式显示设备可以通过所述通信连接直接获取所述佩戴者操作所述外部设备产生的操作数据。在又一些可能的实施方式中,所述外部设备与所述头戴式显示设备对应的目标设备通信连接;所述目标设备用于通过所述通信连接获取所述佩戴者操作所述外部设备产生的操作数据。示例性的,所述目标设备包括终端设备或者服务器。在一个例子中,所述终端设备包括但不限于手机、电脑或者平板等等。在另一个例子中,比如在使用头戴式显示设备参与线上会议的场景中,可以将所述佩戴者操作所述外部设备产生的操作数据发送给服务器,以便由服务器将操作数据发送给参与会议的其他设备。
示例性的,所述外部设备为输入设备,则所述操作数据可以是输入的文字或者图像等。示例性的,所述外部设备为可移动平台的遥控设备,则所述操作数据可以是可移动平台对应的控制内容,比如控制可移动平台加速等。
在一些实施例中,为了可以让用户明确外部设备的操作方式,可以在所述头戴式显示设备的显示图像中叠加显示所述外部设备对应的操作提示信息,以辅助佩戴者准确操作外部设备。示例性的,外部设备为键盘,则外部设备对应的操作提示信息可以是有关于输入法的提示信息。示例性的,外部设备为包括有摇杆和/或按键的遥控设备,则外部设备对应的操作提示信息可以是摇杆和/或按键的不同操作方式所对应的可移动平台控制逻辑。
在一些实施例中,为了提高操作准确性,在获得所述佩戴者操作所述外部设备产生的操作数据之后,还可以在所述头戴式显示设备的显示图像中叠加显示所述佩戴者 操作所述外部设备产生的操作数据,以便让用户确认操作数据的准确性。
在一些实施例中,所述头戴式显示设备与其他设备通信连接,则还可以将头戴式显示设备的显示图像和所述外部设备对应的图像区域发送给远端设备,以在其他设备中显示叠加了外部设备对应的图像区域的显示图像。以及头戴式显示设备或者与所述头戴式显示设备对应的目标设备还可以将佩戴者操作所述外部设备产生的操作数据发送给其他设备,以使其他设备在头戴式显示设备的显示图像中叠加显示所述操作数据。
在一示例性的场景中,比如在线上会议场景,用户可以佩戴头戴式显示设备参与线上会议,头戴式显示设备可以与属于同一会议频道中的其他设备直接或者间接地通信连接,比如头戴式显示设备可以通过服务器与属于同一会议频道中的其他设备通信连接,则在头戴式显示设备中显示的所有或者部分内容也可以对应地在其他设备中进行显示。在另一示例性的场景中,比如在直播场景中,主播用户可以佩戴头戴式显示设备进行直播,头戴式显示设备可以与直播服务器通信连接,头戴式显示设备可以将显示图像,叠加的外部设备对应的图像区域或者操作数据等发送给直播服务器,直播服务器可以将在头戴式显示设备中显示的内容发送给进入主播所在直播频道的观众客户端,则在头戴式显示设备中显示的所有或者部分内容也可以对应地在观众客户端中进行显示。
在一示例性的实施例中,以所述外部设备为输入设备(比如键盘)为例进行说明:头戴式显示设备可以包括一个或多个摄像头,其置于头戴式显示设备的外侧且与头戴式显示设备的佩戴部相背,用于拍摄佩戴者前方或者下方的物理环境。头戴式显示设备包括有姿态传感器和眼球跟踪传感器,头戴式显示设备可以根据姿态传感器采集的数据确定佩戴者的头部姿态,以及根据眼球跟踪传感器采集的数据确定佩戴者的眼部注视范围。当头戴式显示设备基于佩戴者的头部姿态和眼部注视范围确定佩戴者正在向下低头,眼部注视范围内的显示内容为文本输入组件,并且当前的显示内容中具有文本输入组件,确定佩戴者具有文字输入的意图(换句话说,具有操作物理环境中的键盘的意图)。则进一步的,头戴式显示设备可以对摄像头拍摄的原始图像进行识别以获取外部设备对应的图像位置,进而根据外部设备对应的图像位置从原始图像中提取键盘图像区域,并在头戴式显示设备的显示图像中叠加显示所述键盘图像区域;其中,佩戴者看到的键盘图像区域与物理环境中的键盘的尺度、形态一致,实现用户可以基于看到的键盘图像对应操作物理环境中的键盘。头戴式显示设备可以根据摄像头拍摄的原始图像识别佩戴者的手指动作和/或***作的按键等信息,获得佩戴者在物理环境中的键盘输入的文字内容。
以上实施方式中的各种技术特征可以任意进行组合,只要特征之间的组合不存在冲突或矛盾,因此上述实施方式中的各种技术特征的任意进行组合也属于本说明书公开的范围。
在一些场景中,在用户穿戴头戴式显示设备时,存在操作一些虚拟设备的需求,比如通过手指定位点按虚拟设备实现数据输入,由于虚拟设备在物理环境中的位置不确定,导致用户对于虚拟设备的操作位置把握不准,使得用户操作虚拟设备的便捷性较差。
基于此,则请参阅图8,本申请实施例还提供了另一种头戴式显示设备的交互方法,所述方法包括:
在步骤S201中,检测所述头戴式显示设备的佩戴者是否有操作虚拟设备的意图。
在步骤S202中,响应于检测到所述佩戴者有操作虚拟设备的意图,在所述头戴式显示设备的显示图像中叠加显示虚拟设备,所述虚拟设备是基于所述头戴式显示设备的摄像头拍摄得到的虚拟设备标识生成的。
本实施例中,借助于在物理环境中设置的虚拟设备标识在头戴式显示设备的显示图像中叠加显示虚拟设备,虚拟设备标识可以提供一参考面从而让用户对应了解到虚拟设备在物理环境中所在位置,从而实现对虚拟设备进行精准操作,提高虚拟设备的操作便捷性;进一步地,在检测到佩戴者有操作虚拟设备的意图的情况下才叠加显示虚拟设备,无需时刻显示虚拟设备,避免遮挡所述头戴式显示设备的显示图像。
在一些实施例中,虚拟设备标识可以是一种图案,该图案可以设置于不同类型的载体,例如印刷物或者电子设备的屏幕,摄像头拍摄到该图案时,可以在该图案对应的参考面上展开虚拟设备,以使得用户可以“看到”该虚拟设备,并对其进行操作。
在一些实施例中,所述虚拟设备为虚拟输入设备,所述虚拟输入设备包括虚拟键盘、虚拟鼠标、虚拟触控屏和虚拟手写板中的至少一个;或者,所述虚拟设备为虚拟遥控设备,所述头戴式显示设备与可移动平台通信连接,所述头戴式显示设备能够基于用户在虚拟设备上的操作生成控制指令,并直接或间接的向可移动平台传输控制指令,所述可移动平台拍摄的图像能够显示于所述头戴式显示设备。
在一些实施例中,所述检测所述头戴式显示设备的佩戴者是否有操作虚拟设备的意图,包括:根据所述佩戴者的头部姿态和/或眼部注视范围,检测所述佩戴者是否有操作虚拟设备的意图。示例性的,若所述头部姿态指示所述佩戴者的低头姿势和/或在所述眼部注视范围内的显示内容包括与所述虚拟设备对应的虚拟组件,确定所述佩戴 者具有操作虚拟设备的意图。示例性的,所述头部姿态通过设置在所述头戴式显示设备中的姿态传感器采集的数据确定;和/或,所述眼部注视范围通过设置在所述头戴式显示设备中的眼球跟踪传感器采集的数据确定。
在一些实施例中,所述虚拟设备在所述显示图像中的显示位置至少根据所述佩戴者与所述虚拟设备标识之间的相对位置关系确定,从而实现在显示空间中虚拟设备与佩戴者的相对位置关系、与物理环境中佩戴者与虚拟设备标识之间的相对位置关系两者相适应,实现为虚拟设备在物理环境中提供了一个参考面,保证用户可以借助于参考面上精准定位虚拟设备,并实现对虚拟设备的精确操作,提高操作准确性。
示例性的,所述虚拟设备在所述显示图像中的显示位置根据所述佩戴者的眼部与所述头戴式显示设备的显示器之间的相对位置关系、所述摄像头与所述显示器之间的相对位置关系、以及所述摄像头与所述虚拟设备之间的相对位置关系确定。
在一些实施例中,所述摄像头设置在所述头戴式显示设备的外侧,并与所述头戴式显示设备的佩戴部相背。
在一些实施例中,所述摄像头设置于所述头戴式显示设备,且与所述头戴式显示设备可转动连接。示例性的,所述方法还包括:控制所述摄像头转动以使得所述摄像头的镜头组件朝向所述虚拟设备标识。
在一些实施例中,所述虚拟设备以半透明或者半不透明的形式叠加显示在所述显示图像中。
在一些实施例中,所述虚拟设备的透明度根据所述显示图像在叠加区域的色系确定。示例性的,显示图像在叠加区域的色系属于深色系对应的透明度,小于,显示图像在叠加区域的色系属于浅色系对应的透明度。
在一些实施例中,所述头戴式显示设备包括补光灯;所述方法还包括:若所述虚拟设备标识所处环境的亮度小于预设亮度阈值,开启所述补光灯。
在一些实施例中,所述补光灯的亮度根据所述虚拟设备标识所处环境的亮度进行调整,所述补光灯的亮度与所述虚拟设备标识所处环境的亮度成负相关关系。
在一些实施例中,所述摄像头拍摄得到的图像还包括所述佩戴者的指定部位对应的图像区域;所述方法还包括:从所述头戴式显示设备的摄像头拍摄得到的图像中,识别所述佩戴者的指定部位在所述虚拟设备上的操作位置;根据所述操作位置指示的内容获取所述佩戴者操作所述虚拟设备产生的操作数据。
在一些实施例中,所述方法还包括:在所述头戴式显示设备的显示图像中叠加显示所述佩戴者操作所述虚拟设备产生的操作数据。
在一些实施例中,所述方法还包括:在所述头戴式显示设备的显示图像中叠加显示所述虚拟设备对应的操作提示信息。
在一些实施例中,还包括:响应于所述佩戴者无操作虚拟设备的意图,不显示所述虚拟设备。示例性的,若符合以下任一种或多种条件,确定所述佩戴者无操作虚拟设备的意图:所述佩戴者的头部姿态指示所述佩戴者的抬头姿势、所述佩戴者的眼部注视范围内的显示内容不包括与所述虚拟设备对应的虚拟组件、或者所述摄像头拍摄得到的图像中不存在所述佩戴者的指定部位。
在一些实施例中,所述显示图像包括虚拟图像和/或从物理环境拍摄得到的图像。
在一些实施例中,所述头戴式显示设备与搭载有拍摄装置的可移动平台通信连接;所述从物理环境拍摄得到的图像包括所述可移动平台实时采集的图像。
以上实施方式中的各种技术特征可以任意进行组合,只要特征之间的组合不存在冲突或矛盾,因此上述实施方式中的各种技术特征的任意进行组合也属于本说明书公开的范围。
相应地,请参阅图9,本申请实施例还提供了一种头戴式显示设备100,包括:
一个或多个处理器101;
用于存储所述处理器101可执行指令的存储器102;
用于显示图像的显示器103;
所述一个或多个处理器101单独或者共同执行所述可执行指令,以执行上述任意一项所述的方法。
所述处理器101执行所述存储器102中包括的可执行指令,所述处理器101可以是中央处理单元(Central Processing Unit,CPU),还可以是其他通用处理器、数字信号处理器(Digital Signal Processor,DSP)、专用集成电路(Application Specific Integrated Circuit,ASIC)、现成可编程门阵列(Field-Programmable Gate Array,FPGA)或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件等。通用处理器可以是微处理器或者该处理器也可以是任何常规的处理器等。
所述存储器102存储头戴式显示设备的交互方法的可执行指令,所述存储器102可以包括至少一种类型的存储介质,存储介质包括闪存、硬盘、多媒体卡、卡型存储器(例如,SD或DX存储器等等)、随机访问存储器(RAM)、静态随机访问存储器(SRAM)、只读存储器(ROM)、电可擦除可编程只读存储器(EEPROM)、可编程只读存储器(PROM)、磁性存储器、磁盘、光盘等等。而且,设备可以与通过网络连接执行存储器 的存储功能的网络存储装置协作。存储器102可以是头戴式显示设备100的内部存储单元,例如头戴式显示设备100的硬盘或内存。存储器102也可以是头戴式显示设备100的外部存储设备,例如头戴式显示设备100上配备的插接式硬盘,智能存储卡(Smart Media Card,SMC),安全数字(Secure Digital,SD)卡,闪存卡(Flash Card)等。进一步地,存储器102还可以既包括头戴式显示设备100的内部存储单元也包括外部存储设备。存储器102用于存储计算机程序和数据。存储器102还可以用于暂时地存储已经输出或者将要输出的数据。
显示器包括但不限于LCD(Liquid Crystal Display)显示器、LED(Light Emitting Diode)显示器、或者OLED(有机发光二极管,Organic Light-Emitting Diode)显示器等。
在一示例性的实施例中,所述处理器用于:检测所述头戴式显示设备的佩戴者是否有操作外部设备的意图;响应于检测到所述佩戴者有操作外部设备的意图,在显示器的显示图像中叠加显示所述头戴式显示设备的摄像头拍摄得到的图像,所述摄像头拍摄得到的图像包括所述外部设备对应的图像区域。
可选地,所述外部设备为输入设备,所述输入设备包括键盘、鼠标、触控屏和手写板中的至少一个;或者,所述外部设备为遥控设备,所述遥控设备能够直接或间接的向可移动平台传输控制指令,所述可移动平台拍摄的图像能够显示于所述头戴式显示设备。
可选地,所述处理器还用于根据所述佩戴者的头部姿态和/或眼部注视范围,检测所述佩戴者是否有操作外部设备的意图。示例性的,若所述头部姿态指示所述佩戴者的低头姿势和/或在所述眼部注视范围内的显示内容包括与所述外部设备对应的虚拟组件,确定所述佩戴者具有操作外部设备的意图。
可选地,所述头戴式显示设备包括有姿态传感器和/或眼球跟踪传感器。所述头部姿态通过设置在所述头戴式显示设备中的姿态传感器采集的数据确定;和/或,所述眼部注视范围通过设置在所述头戴式显示设备中的眼球跟踪传感器采集的数据确定。
可选地,所述处理器还用于获取所述头戴式显示设备的摄像头拍摄的原始图像;根据所述原始图像中所述外部设备对应的图像位置,对所述原始图像进行裁剪以得到裁剪后的图像,所述裁剪后的图像包括所述外部设备对应的图像区域;在显示器中的显示图像中叠加显示所述裁剪后的图像。
可选地,所述外部设备对应的图像区域在所述显示图像中的显示位置至少根据所述佩戴者与所述外部设备之间的相对位置关系确定。示例性的,所述外部设备对应的 图像区域在所述显示图像中的显示位置根据所述佩戴者的眼部与所述头戴式显示设备的显示器之间的相对位置关系、所述摄像头与所述显示器之间的相对位置关系、以及所述摄像头与所述外部设备之间的相对位置关系确定。
可选地,所述摄像头设置在所述头戴式显示设备的外侧,并与所述头戴式显示设备的佩戴部相背。
可选地,所述摄像头设置于所述头戴式显示设备,且与所述头戴式显示设备可转动连接。可选地,所述处理器还用于:控制所述摄像头转动以使得所述摄像头的镜头组件朝向所述外部设备。
可选地,所述摄像头拍摄得到的图像以半透明或者半不透明的形式叠加显示在所述显示图像中。
可选地,所述摄像头拍摄得到的图像的透明度根据所述显示图像在叠加区域的色系确定。比如显示图像在叠加区域的色系属于深色系对应的透明度,小于,显示图像在叠加区域的色系属于浅色系对应的透明度。
可选地,所述头戴式显示设备包括补光灯;所述处理器还用于若所述外部设备所处环境的亮度小于预设亮度阈值,开启所述补光灯。比如所述补光灯的亮度根据所述外部设备所处环境的亮度进行调整,所述补光灯的亮度与所述外部设备所处环境的亮度成负相关关系。
可选地,所述摄像头拍摄得到的图像还包括所述佩戴者的指定部位对应的图像区域;所述处理器还用于从所述头戴式显示设备的摄像头拍摄得到的图像中,识别所述佩戴者的指定部位在所述外部设备上的操作位置;根据所述操作位置指示的内容获取所述佩戴者操作所述外部设备产生的操作数据。
可选地,所述外部设备与所述头戴式显示设备通信连接;所述处理器还用于通过所述通信连接获取所述佩戴者操作所述外部设备产生的操作数据。
可选地,所述外部设备与所述头戴式显示设备对应的目标设备通信连接;所述目标设备用于通过所述通信连接获取所述佩戴者操作所述外部设备产生的操作数据。
可选地,所述目标设备包括终端设备或者服务器。
可选地,所述处理器还用于在所述头戴式显示设备的显示图像中叠加显示所述佩戴者操作所述外部设备产生的操作数据。
可选地,所述处理器还用于在所述头戴式显示设备的显示图像中叠加显示所述外部设备对应的操作提示信息。
可选地,所述处理器还用于响应于所述佩戴者无操作外部设备的意图,不显示所 述头戴式显示设备的摄像头拍摄得到的图像。
可选地,所述处理器还用于若符合以下任一种或多种条件,确定所述佩戴者无操作外部设备的意图:所述佩戴者的头部姿态指示所述佩戴者的抬头姿势、所述佩戴者的眼部注视范围内的显示内容不包括与所述外部设备对应的虚拟组件、或者所述摄像头拍摄得到的图像中不存在所述佩戴者的指定部位。
可选地,所述显示图像包括虚拟图像和/或从物理环境拍摄得到的图像。
可选地,所述头戴式显示设备与搭载有拍摄装置的可移动平台通信连接;所述从物理环境拍摄得到的图像包括所述可移动平台实时采集的图像。
这里描述的各种实施方式可以使用例如计算机软件、硬件或其任何组合的计算机可读介质来实施。对于硬件实施,这里描述的实施方式可以通过使用特定用途集成电路(ASIC)、数字信号处理器(DSP)、数字信号处理装置(DSPD)、可编程逻辑装置(PLD)、现场可编程门阵列(FPGA)、处理器、控制器、微控制器、微处理器、被设计为执行这里描述的功能的电子单元中的至少一种来实施。对于软件实施,诸如过程或功能的实施方式可以与允许执行至少一种功能或操作的单独的软件模块来实施。软件代码可以由以任何适当的编程语言编写的软件应用程序(或程序)来实施,软件代码可以存储在存储器中并且由控制器执行。
上述设备中各个单元的功能和作用的实现过程具体详见上述方法中对应步骤的实现过程,在此不再赘述。
相应地,请参阅图1,本申请实施例还提供了一种交互***,包括头戴式显示设备100和外部设备200。
示例性的,请参阅图2,所述外部设备为输入设备,所述输入设备包括键盘、鼠标、触控屏、手写板中的至少一个;或者,请参与图4,所述外部设备为遥控设备,所述遥控设备能够直接或间接的向可移动平台传输控制指令,所述可移动平台拍摄的图像能够显示于所述头戴式显示设备。
示例性的,请参与图4,所述***还包括可移动平台,所述可移动平台与头戴式显示设备通信连接。
在示例性实施例中,还提供了一种包括指令的非临时性计算机可读存储介质,例如包括指令的存储器,上述指令可由装置的处理器执行以完成上述方法。例如,非临时性计算机可读存储介质可以是ROM、随机存取存储器(RAM)、CD-ROM、磁带、软盘和光数据存储设备等。
一种非临时性计算机可读存储介质,当存储介质中的指令由终端的处理器执行时, 使得终端能够执行上述方法。需要说明的是,在本文中,诸如第一和第二等之类的关系术语仅仅用来将一个实体或者操作与另一个实体或操作区分开来,而不一定要求或者暗示这些实体或操作之间存在任何这种实际的关系或者顺序。术语“包括”、“包含”或者其任何其他变体意在涵盖非排他性的包含,从而使得包括一系列要素的过程、方法、物品或者设备不仅包括那些要素,而且还包括没有明确列出的其他要素,或者是还包括为这种过程、方法、物品或者设备所固有的要素。在没有更多限制的情况下,由语句“包括一个……”限定的要素,并不排除在包括所述要素的过程、方法、物品或者设备中还存在另外的相同要素。
以上对本申请实施例所提供的方法和装置进行了详细介绍,本文中应用了具体个例对本申请的原理及实施方式进行了阐述,以上实施例的说明只是用于帮助理解本申请的方法及其核心思想;同时,对于本领域的一般技术人员,依据本申请的思想,在具体实施方式及应用范围上均会有改变之处,综上所述,本说明书内容不应理解为对本申请的限制。
Claims (32)
- 一种头戴式显示设备的交互方法,其特征在于,所述方法包括:检测所述头戴式显示设备的佩戴者是否有操作外部设备的意图;响应于检测到所述佩戴者有操作外部设备的意图,在所述头戴式显示设备的显示图像中叠加显示所述头戴式显示设备的摄像头拍摄得到的图像,所述摄像头拍摄得到的图像包括所述外部设备对应的图像区域。
- 根据权利要求1所述的方法,其特征在于,所述外部设备为输入设备,所述输入设备包括键盘、鼠标、触控屏和手写板中的至少一个;或者,所述外部设备为遥控设备,所述遥控设备能够直接或间接的向可移动平台传输控制指令,所述可移动平台拍摄的图像能够显示于所述头戴式显示设备。
- 根据权利要求1所述的方法,其特征在于,所述检测所述头戴式显示设备的佩戴者是否有操作外部设备的意图,包括:根据所述佩戴者的头部姿态和/或眼部注视范围,检测所述佩戴者是否有操作外部设备的意图。
- 根据权利要求3所述的方法,其特征在于,若所述头部姿态指示所述佩戴者的低头姿势和/或在所述眼部注视范围内的显示内容包括与所述外部设备对应的虚拟组件,确定所述佩戴者具有操作外部设备的意图。
- 根据权利要求3所述的方法,其特征在于,所述头部姿态通过设置在所述头戴式显示设备中的姿态传感器采集的数据确定;和/或,所述眼部注视范围通过设置在所述头戴式显示设备中的眼球跟踪传感器采集的数据确定。
- 根据权利要求1所述的方法,其特征在于,所述在所述头戴式显示设备的显示图像中叠加所述头戴式显示设备的摄像头拍摄得到的图像,包括:获取所述头戴式显示设备的摄像头拍摄的原始图像;根据所述原始图像中所述外部设备对应的图像位置,对所述原始图像进行裁剪以得到裁剪后的图像,所述裁剪后的图像包括所述外部设备对应的图像区域;在所述显示图像中叠加显示所述裁剪后的图像。
- 根据权利要求1或6所述的方法,其特征在于,所述外部设备对应的图像区域在所述显示图像中的显示位置至少根据所述佩戴者与所述外部设备之间的相对位置关系确定。
- 根据权利要求7所述的方法,其特征在于,所述外部设备对应的图像区域在所 述显示图像中的显示位置根据所述佩戴者的眼部与所述头戴式显示设备的显示器之间的相对位置关系、所述摄像头与所述显示器之间的相对位置关系、以及所述摄像头与所述外部设备之间的相对位置关系确定。
- 根据权利要求1所述的方法,其特征在于,所述摄像头设置在所述头戴式显示设备的外侧,并与所述头戴式显示设备的佩戴部相背。
- 根据权利要求1所述的方法,其特征在于,所述摄像头设置于所述头戴式显示设备,且与所述头戴式显示设备可转动连接。
- 根据权利要求10所述的方法,其特征在于,所述方法还包括:控制所述摄像头转动以使得所述摄像头的镜头组件朝向所述外部设备。
- 根据权利要求1所述的方法,其特征在于,所述摄像头拍摄得到的图像以半透明或者半不透明的形式叠加显示在所述显示图像中。
- 根据权利要求1所述的方法,其特征在于,所述摄像头拍摄得到的图像的透明度根据所述显示图像在叠加区域的色系确定。
- 根据权利要求13所述的方法,其特征在于,显示图像在叠加区域的色系属于深色系对应的透明度,小于,显示图像在叠加区域的色系属于浅色系对应的透明度。
- 根据权利要求1所述的方法,其特征在于,所述头戴式显示设备包括补光灯;所述方法还包括:若所述外部设备所处环境的亮度小于预设亮度阈值,开启所述补光灯。
- 根据权利要求15所述的方法,其特征在于,所述补光灯的亮度根据所述外部设备所处环境的亮度进行调整,所述补光灯的亮度与所述外部设备所处环境的亮度成负相关关系。
- 根据权利要求1至16任意一项所述的方法,其特征在于,所述摄像头拍摄得到的图像还包括所述佩戴者的指定部位对应的图像区域;所述方法还包括:从所述头戴式显示设备的摄像头拍摄得到的图像中,识别所述佩戴者的指定部位在所述外部设备上的操作位置;根据所述操作位置指示的内容获取所述佩戴者操作所述外部设备产生的操作数据。
- 根据权利要求1至16任意一项所述的方法,其特征在于,所述外部设备与所述头戴式显示设备通信连接;所述方法还包括:通过所述通信连接获取所述佩戴者操作所述外部设备产生的操作数据。
- 根据权利要求1至16任意一项所述的方法,其特征在于,所述外部设备与所述头戴式显示设备对应的目标设备通信连接;所述目标设备用于通过所述通信连接获取所述佩戴者操作所述外部设备产生的操作数据。
- 根据权利要求19所述的方法,其特征在于,所述目标设备包括终端设备或者服务器。
- 根据权利要求1、17至19任意一项所述的方法,其特征在于,所述方法还包括:在所述头戴式显示设备的显示图像中叠加显示所述佩戴者操作所述外部设备产生的操作数据。
- 根据权利要求1所述的方法,其特征在于,所述方法还包括:在所述头戴式显示设备的显示图像中叠加显示所述外部设备对应的操作提示信息。
- 根据权利要求1所述的方法,其特征在于,还包括:响应于所述佩戴者无操作外部设备的意图,不显示所述头戴式显示设备的摄像头拍摄得到的图像。
- 根据权利要求23所述的方法,其特征在于,若符合以下任一种或多种条件,确定所述佩戴者无操作外部设备的意图:所述佩戴者的头部姿态指示所述佩戴者的抬头姿势、所述佩戴者的眼部注视范围内的显示内容不包括与所述外部设备对应的虚拟组件、或者所述摄像头拍摄得到的图像中不存在所述佩戴者的指定部位。
- 根据权利要求1至24任意一项所述的方法,其特征在于,所述显示图像包括虚拟图像和/或从物理环境拍摄得到的图像。
- 根据权利要求25所述的方法,其特征在于,所述头戴式显示设备与搭载有拍摄装置的可移动平台通信连接;所述从物理环境拍摄得到的图像包括所述可移动平台实时采集的图像。
- 一种头戴式显示设备的交互方法,其特征在于,所述方法包括:检测所述头戴式显示设备的佩戴者是否有操作虚拟设备的意图;响应于检测到所述佩戴者有操作虚拟设备的意图,在所述头戴式显示设备的显示图像中叠加显示虚拟设备,所述虚拟设备是基于所述头戴式显示设备的摄像头拍摄得到的虚拟设备标识生成的。
- 一种头戴式显示设备,其特征在于,包括:一个或多个处理器;用于存储所述处理器可执行指令的存储器;用于显示图像的显示器;所述一个或多个处理器单独或者共同执行所述可执行指令,以执行如权利要求1至27任意一项所述的方法。
- 一种交互***,其特征在于,包括如权利要求28所述的头戴式显示设备和外部设备。
- 根据权利要求29所述的交互***,其特征在于,所述外部设备为输入设备,所述输入设备包括键盘、鼠标、触控屏、手写板中的至少一个;或者,所述外部设备为遥控设备,所述遥控设备能够直接或间接的向可移动平台传输控制指令,所述可移动平台拍摄的图像能够显示于所述头戴式显示设备。
- 根据权利要求29所述的交互***,其特征在于,所述***还包括可移动平台。
- 一种计算机可读存储介质,其特征在于,所述计算机可读存储介质存储有可执行指令,所述可执行指令被处理器执行时实现如权利要求1至27任一项所述的方法。
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
PCT/CN2022/070966 WO2023130435A1 (zh) | 2022-01-10 | 2022-01-10 | 交互方法、头戴式显示设备、***及存储介质 |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
PCT/CN2022/070966 WO2023130435A1 (zh) | 2022-01-10 | 2022-01-10 | 交互方法、头戴式显示设备、***及存储介质 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2023130435A1 true WO2023130435A1 (zh) | 2023-07-13 |
Family
ID=87072874
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CN2022/070966 WO2023130435A1 (zh) | 2022-01-10 | 2022-01-10 | 交互方法、头戴式显示设备、***及存储介质 |
Country Status (1)
Country | Link |
---|---|
WO (1) | WO2023130435A1 (zh) |
Citations (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN103946734A (zh) * | 2011-09-21 | 2014-07-23 | 谷歌公司 | 叠加有对于外部设备的控制和指令的可穿戴计算机 |
CN104793731A (zh) * | 2015-01-04 | 2015-07-22 | 北京君正集成电路股份有限公司 | 可穿戴设备的信息输入方法及可穿戴设备 |
CN106484085A (zh) * | 2015-08-31 | 2017-03-08 | 北京三星通信技术研究有限公司 | 在头戴式显示器中显示真实物体的方法及其头戴式显示器 |
CN106716302A (zh) * | 2014-09-11 | 2017-05-24 | 诺基亚技术有限公司 | 用于显示图像的方法、设备和计算机程序 |
CN107250891A (zh) * | 2015-02-13 | 2017-10-13 | Otoy公司 | 头戴式显示器与真实世界对象之间的相互通信 |
US20180088682A1 (en) * | 2016-09-29 | 2018-03-29 | Tower Spring Global Limited | Head mounted display |
US20180173404A1 (en) * | 2016-12-20 | 2018-06-21 | Adobe Systems Incorporated | Providing a user experience with virtual reality content and user-selected, real world objects |
CN110249368A (zh) * | 2016-12-05 | 2019-09-17 | 奇跃公司 | 混合现实环境中的虚拟用户输入控制 |
-
2022
- 2022-01-10 WO PCT/CN2022/070966 patent/WO2023130435A1/zh unknown
Patent Citations (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN103946734A (zh) * | 2011-09-21 | 2014-07-23 | 谷歌公司 | 叠加有对于外部设备的控制和指令的可穿戴计算机 |
CN106716302A (zh) * | 2014-09-11 | 2017-05-24 | 诺基亚技术有限公司 | 用于显示图像的方法、设备和计算机程序 |
CN104793731A (zh) * | 2015-01-04 | 2015-07-22 | 北京君正集成电路股份有限公司 | 可穿戴设备的信息输入方法及可穿戴设备 |
CN107250891A (zh) * | 2015-02-13 | 2017-10-13 | Otoy公司 | 头戴式显示器与真实世界对象之间的相互通信 |
CN106484085A (zh) * | 2015-08-31 | 2017-03-08 | 北京三星通信技术研究有限公司 | 在头戴式显示器中显示真实物体的方法及其头戴式显示器 |
US20180088682A1 (en) * | 2016-09-29 | 2018-03-29 | Tower Spring Global Limited | Head mounted display |
CN110249368A (zh) * | 2016-12-05 | 2019-09-17 | 奇跃公司 | 混合现实环境中的虚拟用户输入控制 |
US20180173404A1 (en) * | 2016-12-20 | 2018-06-21 | Adobe Systems Incorporated | Providing a user experience with virtual reality content and user-selected, real world objects |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US20220153412A1 (en) | Control method, control system, and smart glasses for first person view unmanned aerial vehicle flight | |
JP6058184B1 (ja) | ヘッドマウントディスプレイシステムを制御するための方法、および、プログラム | |
CN108292489B (zh) | 信息处理装置和图像生成方法 | |
KR102098277B1 (ko) | 시선 추적을 이용한 시인성 개선 방법, 저장 매체 및 전자 장치 | |
US10356382B2 (en) | Information processing device, information processing method, and program | |
US11487354B2 (en) | Information processing apparatus, information processing method, and program | |
US10296080B2 (en) | Systems and methods to simulate user presence in a real-world three-dimensional space | |
US11500510B2 (en) | Information processing apparatus and non-transitory computer readable medium | |
US20200062181A1 (en) | Immersive display | |
US20230336865A1 (en) | Device, methods, and graphical user interfaces for capturing and displaying media | |
CN112639685A (zh) | 模拟现实(sr)中的显示设备共享和交互 | |
KR20200040716A (ko) | 시선 추적을 이용한 시인성 개선 방법, 저장 매체 및 전자 장치 | |
US20240221291A1 (en) | Methods for time of day adjustments for environments and environment presentation during communication sessions | |
US11699412B2 (en) | Application programming interface for setting the prominence of user interface elements | |
WO2023130435A1 (zh) | 交互方法、头戴式显示设备、***及存储介质 | |
KR20200041133A (ko) | 촬영 가이드 제공 장치 및 프로그램 | |
KR20190129982A (ko) | 전자기기 및 그 제어 방법 | |
JP6999538B2 (ja) | 情報処理方法、情報処理プログラム、情報処理システム、および情報処理装置 | |
TW202213994A (zh) | 擴增實境系統與其顯示亮度調整方法 | |
CN111142660A (zh) | 一种显示装置、画面显示方法和存储介质 | |
WO2023137758A1 (zh) | 可移动平台的控制方法、头戴式设备、***和存储介质 | |
JP7427739B2 (ja) | 表示装置 | |
US20240103684A1 (en) | Methods for displaying objects relative to virtual surfaces | |
US20240233288A1 (en) | Methods for controlling and interacting with a three-dimensional environment | |
US11558599B2 (en) | Electronic apparatus, control method for electronic apparatus, and non-transitory computer-readable storage medium |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 22917907 Country of ref document: EP Kind code of ref document: A1 |