WO2023001113A1 - Display method and electronic device - Google Patents

Display method and electronic device Download PDF

Info

Publication number
WO2023001113A1
WO2023001113A1 PCT/CN2022/106280 CN2022106280W WO2023001113A1 WO 2023001113 A1 WO2023001113 A1 WO 2023001113A1 CN 2022106280 W CN2022106280 W CN 2022106280W WO 2023001113 A1 WO2023001113 A1 WO 2023001113A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
depth
frame
field
user
Prior art date
Application number
PCT/CN2022/106280
Other languages
French (fr)
Chinese (zh)
Inventor
王松
沈钢
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Publication of WO2023001113A1 publication Critical patent/WO2023001113A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer

Definitions

  • the present application relates to the field of electronic technology, in particular to a display method and electronic equipment.
  • VR technology is a means of human-computer interaction created with the help of computer and sensor technology.
  • VR technology integrates computer graphics technology, computer simulation technology, sensor technology, display technology and other scientific technologies to create a virtual environment, and users can immerse themselves in the virtual environment by wearing VR glasses.
  • the virtual environment is presented through continuous refreshing of many three-dimensional images, and the three-dimensional images include objects in different depths of field, giving users a sense of three-dimensionality.
  • VAC Vergence accommodation conflict
  • the purpose of the present application is to provide a display method and an electronic device for improving VR experience.
  • a display method includes: displaying N frames of images to the user through a display device; N is a positive integer; on the i-th frame of the N frames of images, the definition of the first object in the first depth of field is the first definition; In the j-th frame of the N frames of images, the definition of the first object in the first depth of field is the second definition; in the k-th frame of the N frames of images, in the first The sharpness of the first object in the depth of field is the third sharpness; wherein, the first sharpness is smaller than the second sharpness, and the second sharpness is greater than the third sharpness, i, j , k are all positive integers less than N, i ⁇ j ⁇ k; wherein, the first depth of field is greater than the second depth of field, or, the distance between the first depth of field and the depth of field where the user gaze point is greater than the first distance.
  • the definition of the first object in the first depth of field in the image stream (that is, N frames of images) displayed by the VR display device (for example, VR glasses) to the user varies from high to low.
  • the definition of the first object with a larger depth of field or the first object farther away from the user's gaze point varies from high to low.
  • the VR display device displays the i-th frame or the k-th frame of images
  • the definition of the first object is relatively low, it can relieve human brain fatigue.
  • the VR display device displays the jth frame of image
  • the brain can capture the details of the first object, and the details of the first object will not be lost. Therefore, in this way, the fatigue of the human brain can be relieved, and it can also ensure that the human brain can absorb enough details of the object, and the user experience is better.
  • the gaze point of the user remains unchanged. That is to say, when the user wears VR glasses to watch the virtual environment, if the user's gaze point does not change, the definition of the first object far away from the user's gaze point may be high or low (or rise or fall). For example, when the VR glasses display the i-th frame or the k-th frame of images, since the definition of the first object is relatively low, it can relieve human brain fatigue.
  • the details of the first object can be avoided from being lost, and sufficient details of the first object can be captured in the brain. In this way, it can not only alleviate the fatigue of the human brain, but also ensure that the human brain can absorb enough details of the object, and the user experience is better.
  • the first depth of field changes accordingly.
  • the distance between the first depth of field and the depth of field where object A is located is greater than the first distance (for example, 0.5m), so the first depth of field is greater than or equal to 1m ;
  • the distance between the first depth of field and the depth of field where object B is located is greater than the first distance (for example, 0.5m), so the first depth of field becomes greater than or equals 1.3m; therefore, as the user's focus changes, the first depth of field changes.
  • the second depth of field includes: the depth of field specified by the user, the depth of field of the user's gaze point, the default depth of field of the system, the depth of field of the virtual image plane, the depth of field corresponding to the virtual scene, and the subject object on the i-th frame image At least one of the depths of field. That is to say, in the image stream displayed by the display device, the definition of the first object whose depth of field is greater than the second depth of field varies from high to low.
  • the second depth of field may be determined in multiple ways, for example, method 1, the second depth of field may be determined according to a VR scene, and the preset depth of field varies with different VR scenes.
  • the VR scene includes but is not limited to at least one of VR games, VR viewing, VR teaching and the like.
  • the second depth of field may be set by the user. It should be understood that different VR applications may set different second depths of field. Mode 3, the second depth of field may also be the default depth of field. Mode 4, the second depth of field may also be the depth of field where the virtual image plane is located. Mode 5, the second depth of field may also be the depth of field where the main object is located in the picture currently being displayed by the VR display device. Method 6. The second depth of field is the depth of field where the user's gaze point is located. It should be noted that several methods for determining the second depth of field are listed above, but the embodiments of the present application are not limited to the above methods, and other methods for determining the second depth of field are also feasible.
  • the method before displaying the N frames of images, the method further includes: detecting that the user triggers an operation for starting the eye protection mode, the user's viewing time is greater than the first duration, and the user's eyesight within the second duration At least one of the number of blinks/squints is greater than the first number.
  • the display device initially displays with the same resolution, that is, all objects in the displayed image stream have the same resolution.
  • the user’s viewing time is greater than the first duration
  • the number of blinks/squints of the user’s eyes within the second duration is greater than the first number, the first in the displayed image stream
  • the sharpness of the first object in the depth of field goes up and down.
  • the technical solution of the present application is started (the definition of the first object at the first depth of field in the image stream increases or decreases), saving images. Power consumption caused by processing (such as blurring the first object on the i-th frame and the k-th frame image).
  • the clarity of the second object in the third depth of field in the N frames of images is the same. That is to say, in the image stream, the first object at the first depth of field has higher or lower sharpness, while the second object at the third depth of field has the same sharpness.
  • the third depth of field is smaller than the first depth of field. That is, the definition of the first object with a larger depth of field varies from high to low, while the definition of the second object with a smaller depth of field remains unchanged.
  • the human eye looks at nearby objects, the more details it sees, the clearer it is, and when looking at distant objects, the less details it sees, the more blurred. Therefore, when the distant objects are blurred and the nearby objects are clear on the image displayed by the display device, the virtual environment felt by people is more in line with the real situation.
  • the clarity of distant objects in the image stream displayed by the display device varies from high to low, and is not always in a blurred state, so it can ensure that the human brain can obtain sufficient details of distant objects.
  • the definition of objects far away from the user's gaze point varies from high to low, and is not always in a blurred state, so it can ensure that the human brain can obtain sufficient details of objects far away from the user's gaze point.
  • a possible implementation manner is that the time interval between the display time of the j-th frame of image and the display time of the i-th frame of image is less than or equal to the user's visual dwell time; and/or, the k-th The time interval between the display time of the frame image and the display time of the jth frame image is less than or equal to the time dwell time.
  • the definition of the first object on the i-th frame of image is low, and the definition of the first object on the j-th frame of image is high. Display the i-th frame image and the j-th frame image, in this case, the human brain will fuse the i-th frame image and the j-th frame image to ensure that the human brain can capture enough details of the first object.
  • n and m may be determined according to the user's visual dwell time and image refresh frame rate. Assume that the user's visual dwell time is T, and the image refresh frame rate is P. Then, within T time, T/P frame images can be displayed, then n is less than or equal to T/P, and m is less than or equal to T/P. Therefore, the i-th frame image and the j-th frame image are displayed within the user's visual dwell time. In this way, the human brain will fuse the i-th frame image and the j-th frame image to ensure that the human brain can capture enough details of the first object.
  • the display device includes a first display screen and a second display screen, the first display screen is used to present images to the user's left eye, and the second display screen is used to display images to the user's right eye presenting an image; the images displayed on the first display screen and the second display screen are synchronized.
  • the synchronization of the images displayed on the first display screen and the second display screen can be understood as that the first display screen displays the i-th frame image, then the second display screen also displays the i-th frame image, ensuring the order of the images displayed on the two display screens unanimous.
  • the first display screen and the second display screen respectively display the N frames of images; it can be understood that the first display screen and the second display screen display the same group of image streams, and the first object in the image streams Clarity is high and low. Since the image streams displayed on the two display screens are the same, and the displayed images are synchronized, for example, the i-th frame and the j-th frame are displayed at the same time, and so on. Therefore, when the first object is blurred on the first display screen, the first object is also blurred on the second display screen. That is to say, the change trend of the clarity of the first object in the image stream displayed on the first display screen and the image stream displayed on the second display screen is the same, for example, both have a change trend of "blurry-clear-blurry-clear".
  • the first display screen displays the N frames of images; the second display screen displays another N frames of images; the other N frames of images have the same image content as the N frames of images; the other On the i-th image in the N frames of images, the definition of the first object in the first depth of field is the fourth definition; in the j-th image in the other N frames of images, in the first depth of field The definition of the first object is the fifth definition; the definition of the first object in the first depth of field on the k-th frame image in the other N frames of images is the sixth definition; Wherein, the fourth definition is greater than the fifth definition, and the fourth definition is smaller than the sixth definition.
  • the first object is blurred on the first display screen
  • the first object is clear on the second display screen. That is to say, the change trend of the definition of the first object in the image stream displayed on the first display screen and the image stream displayed on the second display screen may be opposite.
  • the sharpness of the first object in the image stream displayed on the first display screen changes alternately from "blur-clear-fuzzy-clear"
  • the distant objects in the image stream displayed on the second display screen appear "clear-fuzzy-clear-clear-clear”. Fuzzy" alternately.
  • the fourth definition is greater than the first definition; and/or, the fifth definition is smaller than the second definition; and/or, the sixth definition is greater than the first Three clarity.
  • the display screens corresponding to the left and right eyes synchronously display images (for example, the i-th frame image)
  • the first object on the left-eye image is blurred
  • the first object on the right-eye image is clear.
  • the display screens corresponding to the left and right eyes synchronously display images (for example, the j-th frame image)
  • the first object on the left-eye image is clear
  • the first object on the right-eye image is blurred.
  • the display method provided by the embodiment of the present application may be applicable to various application scenarios.
  • game applications eg, VR game applications
  • simulated driving eg, VR driving
  • simulated teaching eg, VR teaching
  • the like Take VR games and VR driving as examples below.
  • the N frames of images are images related to games; the games may be VR games.
  • the N frames of images are images generated by a VR game application.
  • the second depth of field includes: in the game scene, the depth of field where the game character corresponding to the user is located, or the depth of field where the body part (such as an arm) of the game character corresponding to the user is located, or the current depth of field of the game character corresponding to the user.
  • the depth of field where the game equipment held (such as guns) is located; and/or, the depth of field where the user’s gaze point is located includes: in the game scene, the depth of field where the game character corresponding to the game opponent is located, or the depth of field where the building is located, or, the depth of field where the user’s corresponding
  • the N frames of images are images related to vehicle driving; for example, the N frames of images are images generated by a VR driving application.
  • the second depth of field includes: in the vehicle driving scene, the depth of field of the vehicle currently driven by the user, or the depth of field of the steering wheel of the vehicle currently driven by the user, or the depth of field of the windshield of the vehicle currently driven by the user; and /or, the depth of field where the user's gaze point is located includes: in the vehicle driving scene, other users driving vehicles on the driving road (such as a vehicle driving in front of the user's current driving vehicle), or objects set on the roadside of the driving road (such as, trees along the road, signs, etc.).
  • the definition of the first object whose depth of field is greater than the second depth of field or the first object far away from the user's gaze point can be high or low, which can not only relieve fatigue, but also ensure that the first object is captured in the human brain enough details to ensure the vehicle driving (eg, VR driving) experience.
  • the image in the i-th frame is an image after blurring the first object on the original image in the i-th frame;
  • the image in the j-th frame is the original image in the j-th frame or The image of the first object on the j-frame original image after clearing processing;
  • the k-th frame image is an image after blurring the first object on the k-th frame of the original image; wherein, the first The definition of all objects on the i-frame original image, the j-th frame original image, and the k-th frame original image is the same;
  • the j-th frame image is after the definition processing of the first object on the j-th frame original image image, including: the image of the jth frame is an image obtained by fusing the image of the ith frame and the original image of the jth frame; or, the image of the jth frame is blurred on the image of the ith frame An image obtained by fusing the image information lost during processing with the original image of the jth frame.
  • the j-th frame image is an image obtained by fusing the i-th frame image and the j-th frame original image, including: the area where the first object is located on the j-th frame image
  • the image block in is obtained by fusion of the first image block and the second image block; wherein, the first image block is an image block in the area where the first object is located on the ith frame image, and the second image block is The image block is an image block in the area where the first object is located on the original image of the jth frame.
  • an electronic device including:
  • processor memory, and, one or more programs
  • the one or more programs are stored in the memory, the one or more programs include instructions, and when the instructions are executed by the processor, the electronic device performs the above-mentioned first aspect The method steps provided.
  • a computer-readable storage medium is provided, the computer-readable storage medium is used to store a computer program, and when the computer program is run on a computer, the computer executes the method as provided in the above-mentioned first aspect .
  • a computer program product including a computer program, and when the computer program is run on a computer, the computer is made to execute the method provided in the first aspect above.
  • a graphical user interface on an electronic device has a display screen, a memory, and a processor, the processor is used to execute one or more computer programs stored in the memory, the The graphical user interface includes a graphical user interface displayed when the electronic device executes the method provided in the first aspect above.
  • the embodiment of the present application further provides a chip system, the chip system is coupled with the memory in the electronic device, and is used to call the computer program stored in the memory and execute the technical solution of the first aspect of the embodiment of the present application.
  • “Coupling” in the embodiments of the application means that two components are directly or indirectly combined with each other.
  • FIG. 1 is a schematic diagram of vergence-accommodation conflict provided by an embodiment of the present application
  • FIG. 2 is a schematic diagram of a VR system provided by an embodiment of the present application.
  • FIG. 3A is a schematic diagram of the principle of human eye convergence provided by an embodiment of the present application.
  • Fig. 3B is a schematic diagram of the human eye structure provided by an embodiment of the present application.
  • Fig. 4 is a schematic diagram of the adjustment of the human eye ciliary muscle to the lens provided by an embodiment of the present application;
  • Fig. 5 is a schematic diagram of VR glasses provided by an embodiment of the present application.
  • FIGS. 6A to 6C are schematic diagrams of virtual image planes corresponding to images displayed by VR glasses provided by an embodiment of the present application.
  • FIGS. 7A to 7B are schematic diagrams of a first application scenario provided by an embodiment of the present application.
  • FIGS. 8A to 8B are schematic diagrams of a second application scenario provided by an embodiment of the present application.
  • FIG. 9 is a schematic structural diagram of a VR wearable device provided by an embodiment of the present application.
  • FIG. 10 is a schematic diagram of clear gaze points of users and blurred gaze points of non-users in a VR virtual environment provided by an embodiment of the present application;
  • FIG. 11 is a schematic flow chart of an image generation principle provided by an embodiment of the present application.
  • Fig. 14 is a schematic diagram of an image acquired by a user's human brain provided by an embodiment of the present application.
  • FIG. 15 is a schematic diagram of an image stream generation process provided by an embodiment of the present application.
  • FIG. 16 is a schematic diagram of fuzzy objects in the foreground and clear objects in the foreground in the VR virtual environment provided by an embodiment of the present application;
  • Fig. 17 is another schematic flowchart of the principle of image generation provided by an embodiment of the present application.
  • 20 to 23 are schematic diagrams of image streams displayed on the left-eye display screen and the right-eye display screen on the display device provided by an embodiment of the present application;
  • FIG. 24 is a schematic structural diagram of an electronic device provided by an embodiment of the present application.
  • At least one of the embodiments of the present application involves one or more; wherein, a plurality means greater than or equal to two.
  • words such as “first” and “second” are only used for the purpose of distinguishing descriptions, and cannot be understood as express or implied relative importance, nor can they be understood as express or imply order.
  • the first object and the second object do not represent the importance of the two, or represent the order of the two, but to distinguish the objects.
  • VR technology is a means of human-computer interaction created with the help of computer and sensor technology.
  • VR technology integrates computer graphics technology, computer simulation technology, sensor technology, display technology and other science and technology to create a virtual environment.
  • the virtual environment includes three-dimensional realistic images generated by computers and dynamically played in real time to bring visual perception to users; moreover, in addition to the visual perception generated by computer graphics technology, there are also perceptions such as hearing, touch, force, and movement.
  • the user can see the VR game interface by wearing the VR wearable device, and can interact with the VR game interface through gestures, handles, and other operations, as if in a game.
  • Augmented Reality (AR) technology refers to superimposing computer-generated virtual objects on real-world scenes to enhance the real world.
  • AR technology needs to collect real-world scenes, and then add a virtual environment to the real world.
  • VR technology creates a complete virtual environment, and all users see are virtual objects; while AR technology superimposes virtual objects on the real world, that is, it includes objects in the real world. Also includes dummy objects.
  • the user wears transparent glasses, through which the real environment around can be seen, and virtual objects can also be displayed on the glasses, so that the user can see both real objects and virtual objects.
  • Mixed reality technology is to build a bridge of interactive feedback information between the virtual environment, the real world and users by introducing real scene information (or called real scene information) into the virtual environment. , thereby enhancing the realism of the user experience.
  • the real object is virtualized (for example, using a camera to scan the real object for 3D reconstruction to generate a virtual object), and the virtualized real object is introduced into the virtual environment, so that the user can see in the virtual environment real object.
  • the technical solution provided by the embodiment of the present application may be applicable to a VR scene, an AR scene or an MR scene.
  • VR VR
  • AR and MR it can also be applied to other scenarios.
  • glasses-free 3D scenes glasses-free 3D display, glasses-free 3D projection, etc.
  • theaters such as 3D movies
  • VR software in electronic equipment etc., in short, can be applied to any scene that needs to display three-dimensional images.
  • the following mainly introduces the VR scene as an example.
  • FIG. 2 is a schematic diagram of a VR system according to an embodiment of the present application.
  • the VR system includes a VR wearable device 100 and an image generating device 200 .
  • the image generating device 200 includes a host (such as a VR host) or a server (such as a VR server).
  • the VR wearable device 100 is connected (wired connection or wireless connection) with a VR host or a VR server.
  • the VR host or VR server may be a device with relatively large computing power.
  • the VR host can be a device such as a mobile phone, a tablet computer, or a notebook computer, and the VR server can be a cloud server, etc.
  • the VR host or VR server is responsible for generating images, etc., and then sends the images to the VR wearable device 100 for display, and the user wearing the VR wearable device 100 can see the images.
  • the VR wearable device 100 may be a head mounted device (Head Mounted Display, HMD), such as glasses, a helmet, and the like.
  • the VR system in FIG. 2 may not include the image generating device 200 .
  • the VR wearable device 100 has image generation capabilities locally, and does not need to acquire images from the image generation device 200 (VR host or VR server) for display.
  • the VR wearable device 100 can display a three-dimensional image. Since different objects have different depths of field (refer to the introduction below) on the three-dimensional image, the virtual environment can be shown to the user through the three-dimensional image.
  • a three-dimensional image includes objects at different image depths.
  • a VR wearable device displays a three-dimensional image, and the user wearing the VR wearable device sees a three-dimensional scene (that is, a virtual environment).
  • a three-dimensional scene that is, a virtual environment.
  • Different objects in the three-dimensional scene have different distances from the user's eyes, presenting a three-dimensional effect. Therefore, the image depth can be understood as the distance between the object on the 3D image and the user's eyes.
  • the larger the image depth the farther away from the user's eyes visually, which looks like a distant view.
  • the smaller the image depth the closer to the user's eyes visually. , which looks like a close-up.
  • Image depth may also be referred to as "depth of field”.
  • the human eye can obtain a light signal in the actual scene and process the light signal in the brain to realize visual experience.
  • the optical signal in the actual scene may include reflected light from different objects, and/or an optical signal directly emitted by a light source. Since the light signal of the actual scene can carry the relevant information (such as size, position, color, etc.) of each object in the actual scene, the brain can obtain the information of the object in the actual scene by processing the light signal , that is, to obtain visual experience.
  • the left eye and the right eye watch the same object, the viewing angles are slightly different. Therefore, the scene seen by the left eye and the right eye is actually different.
  • the left eye can obtain the light signal of the two-dimensional image (hereinafter referred to as the left eye image) on the plane where the focus of the human eye is perpendicular to the line of sight of the left eye.
  • the right eye can obtain the light signal of the two-dimensional image (hereinafter referred to as the right eye image) on the plane where the focus of the human eye is perpendicular to the line of sight of the right eye.
  • the image for the left eye is slightly different from the image for the right eye.
  • the brain can obtain information about different objects in the current scene by processing the light signals of the left-eye image and the right-eye image.
  • Stereo vision experience can also be called binocular stereo vision.
  • converging when viewing an object in an actual scene, the user will experience two processes of convergence (vergence) and zoom (accommodation). Wherein, converging may also be referred to as converging, which is not limited in this application.
  • Convergence can be understood as adjusting the line of sight of the human eye so that it points to the object.
  • the object is a triangle in FIG. 3A as an example
  • the sight lines of the left eye and the right eye can be respectively directed to the The object turns (points to the object).
  • Vergence angle (Vergence angle) and Vergence depth (Vergence distance)
  • the convergence angle ⁇ the angle formed by the line of sight of the left eye and the line of sight of the right eye when the two eyes observe an object.
  • the brain can judge the depth of the object by obtaining the convergence angle ⁇ of the eyes, that is, the convergence depth. It can be understood that the closer the observed object is to the human eye, the larger the convergence angle ⁇ and the smaller the convergence depth. Correspondingly, the farther the observed object is from the human eye, the smaller the convergence angle ⁇ and the greater the convergence depth.
  • Zooming can be understood as adjusting the human eye to the correct focal length when observing an object.
  • the brain controls the lens to adjust to the correct focus through the ciliary muscle.
  • Figure 3B is a schematic diagram of the composition of the human eye.
  • the human eye may include a lens and ciliary muscle, as well as a retina located in the fundus.
  • the crystalline lens can function as a zoom lens, converging the light rays entering the human eye. In order to converge the incident light onto the retina of the human eye fundus, so that the scene in the actual scene can form a clear image on the retina.
  • the ciliary muscle can be used to adjust the shape of the lens.
  • the ciliary muscle can adjust the diopter of the lens by contracting or relaxing, so as to achieve the effect of adjusting the focal length of the lens. Therefore, objects at different distances in the actual scene can be clearly imaged on the retina through the lens.
  • FIG. 4 is a diagram illustrating the adjustment of the ciliary muscle to the lens when the human eye observes objects at different distances.
  • FIG. 4 is a diagram illustrating the adjustment of the ciliary muscle to the lens when the human eye observes objects at different distances.
  • FIG. 4 which is a diagram illustrating the adjustment of the ciliary muscle to the lens when the human eye observes objects at different distances.
  • FIG. 4 which is a diagram illustrating the adjustment of the ciliary muscle to the lens when the human eye observes objects at different distances.
  • (a) of FIG. 4 when the human eye observes a distant object, take the object as a non-light source as an example.
  • the ciliary muscle can control the state of the lens to the state shown in Figure 4 (a), such as the ciliary muscle relaxes, and controls the lens to be flat and the diopter is small, so that parallel incident light can pass through the lens Converge on the retina of the fundus.
  • the human eye observes a relatively close object, in combination with (b) in FIG. 4 , take the object as a non-light source as an example.
  • the reflected light from the surface of the object may enter the human eye according to the optical path shown in (b) in FIG. 4 .
  • the ciliary muscle can keep the state of the lens in the state shown in (b) in Figure 4, as the ciliary muscle contracts, the lens protrudes, and the diopter becomes larger, so that the lens shown in (b) in Figure 4 Incident light can pass through the lens and then converge on the retina in the fundus of the eye. That is to say, when the human eye observes objects at different distances, the contraction or relaxation state of the ciliary muscle is different.
  • depth This depth may be referred to as zoom depth.
  • the human brain will determine the convergence depth according to the convergence angle ⁇ .
  • the human brain will determine the zoom depth according to the contraction or relaxation state of the ciliary muscle. Both the convergence depth and the zoom depth represent the object distance The distance between the user's glasses.
  • the depth of convergence and the depth of zoom are coordinated or consistent.
  • the brain cannot accurately judge the depth of the object, and a sense of fatigue will occur, which will affect the user experience.
  • the depth inconsistency of the object indicated by the vergence depth and the zoom depth may also be called vergence accommodation conflict (Vergence accommodation conflict, VAC).
  • FIG. 5 is a schematic diagram of VR glasses.
  • two display screens (such as display screen 501 and display screen 502) can be set in VR glasses, and display screen 501 and display screen 502 can be independent display screens, or display screen 501 and display screen 502 can be different display areas on the same display screen), and each display screen has a display function.
  • Each display screen can be used to display corresponding content to one eye (such as left eye or right eye) of the user through a corresponding eyepiece.
  • the display screen 501 corresponds to the eyepiece 503
  • the display screen 502 corresponds to the eyepiece 504
  • a left-eye image corresponding to the virtual environment may be displayed.
  • the light of the left-eye image can pass through the eyepiece 503 and converge at the left eye, so that the left eye can see the left-eye image.
  • the right-eye image corresponding to the virtual environment may be displayed.
  • the light of the right-eye image can pass through the eyepiece 504 and converge at the right eye, so that the right eye sees the right-eye image.
  • the brain can fuse the left-eye image and the right-eye image, so that the user can see objects in the virtual environment corresponding to the left-eye image and the right-eye image.
  • the image seen by human eyes is actually an image corresponding to the image displayed on the display screen on the virtual image plane 600 as shown in FIG. 6A .
  • the left-eye image seen by the left eye may be a virtual image corresponding to the left-eye image on the virtual image plane 600 .
  • the right-eye image seen by the right eye may be a virtual image corresponding to the right-eye image on the virtual image plane 600 .
  • the zoom distance may be the distance from the virtual image plane 600 to the human eye (depth 1 as shown in FIG. 6B ).
  • the objects in the virtual environment displayed by the VR glasses to the user are often not on the virtual image plane 600 .
  • the observed object triangle in the virtual environment in FIG. 6B (because it is a virtual environment, the triangle is represented by a dotted line) is not on the virtual image plane 600 .
  • the depth of convergence should be the depth of the observed object (ie, triangle) in the virtual environment.
  • the depth of convergence may be depth 2 as shown in FIG. 6B .
  • depth 1 and depth 2 are not consistent at this time. In this way, the brain cannot accurately judge the depth of the observed object, thereby causing brain fatigue and affecting user experience.
  • a virtual environment includes multiple observed objects. As shown in FIG. 6C , there are two observed objects, wherein the observed object 1 is a triangle (dotted line) as an example, and the observed object 2 is a sphere (dashed line) as an example. For each observed object, there will be cases where the depth of convergence and the depth of zoom are different.
  • the clarity of all virtual objects (that is, observed objects) on an image is the same.
  • the observed objects are described by taking triangles and spheres as examples.
  • the triangle and the sphere are displayed on the same image with the same definition, and the virtual image plane 600 corresponding to the triangle and the sphere are at the same depth (that is, depth 1), so the human brain will consider the zoom of the two observed objects based on the same virtual image plane The depth should be the same. But in fact, the convergence depth of the two observed objects is different.
  • the convergence depth of the triangle is depth 2
  • the convergence depth of the sphere is depth 3. Based on the different convergence depths, the human brain will think that the zoom depth of the two observed objects should be It is different.
  • this technology can adjust the virtual image plane 600 to the depth of field where the observed object (such as a triangle) is located in the virtual environment, so that the zoom depth and the convergence depth are consistent.
  • this technology can adjust the virtual image surface corresponding to the triangle to the depth of field where the triangle is located, and adjust the virtual image surface corresponding to the sphere to the depth of field where the circle is located.
  • the zoom depth corresponding to the triangle is consistent with the depth of convergence
  • the zoom depth and convergence depth corresponding to the ball are also consistent, thus overcoming VAC.
  • this technology of adjusting the position of the virtual image plane requires the support of certain optical hardware, such as stepping motors.
  • additional optical hardware will increase the cost; on the other hand, adding optical hardware will increase the volume of VR glasses. Difficult to apply to light and small VR wearable devices.
  • an embodiment of the present application provides a display method.
  • a VR display device such as VR glasses
  • the clarity of the sphere and the triangle can be set to be different.
  • a sphere is blurred and a triangle is clear.
  • the human brain will think that the zoom depth of the triangle and the sphere is the same, that is, depth 1.
  • the human brain Since the triangle is clear, the human brain will think that the depth 1 is accurate for the triangle; but the ball is blurred, so the human brain will think that the depth 1 is inaccurate or not adjusted for the ball, and the human brain will Try to adjust the ciliary muscle to see the sphere clearly. In this way, the human brain will judge that the zoom depths of the triangle and the sphere are no longer the same depth, which is no longer the same as "the human brain thinks that the zoom depths of the two observed objects should be different based on the different convergence depths of the triangle and the sphere". Conflict relieves the fatigue of the human brain.
  • the display method provided by the embodiment of the present application can alleviate the user's fatigue when viewing the virtual environment through VR glasses, improve user experience, and does not need to rely on the support of special optical hardware such as stepping motors, and is low in cost and helpful
  • the equipment is light and miniaturized.
  • FIG. 7A and FIG. 7B are schematic diagrams of the first application scenario provided by the embodiment of the present application.
  • This application scenario takes a user wearing VR glasses to play a VR game as an example.
  • the VR glasses display an image 701 .
  • the image 701 may be an image generated by a VR game application, including objects such as guns, containers, and trees.
  • objects such as guns, containers, and trees.
  • the gun is at depth 1
  • the container is at depth 2
  • Depth of Field 3 > Depth of Field 2 > Depth of Field 1.
  • the user’s gaze point is on the gun (such as a scope)
  • the depth of field 3 where the tree is located is farther from depth of field 1
  • the depth of field 2 where the container is located is closer to depth of field 1
  • the definition of the tree on image 701 is clearer than that of the container low degree. Therefore, the tree in the VR game screen seen by the user wearing VR glasses is blurred, but the container is clear.
  • the sharpness of different objects on the image is different. Specifically, objects far away from the user's gaze point (such as trees) are relatively blurred, and objects relatively close to the user's gaze point (such as containers) are relatively clear.
  • the human brain will think that the convergence depth of the tree and the container is different; in addition, the definition of the tree and the container is different, the human brain will think that the zoom depth of the tree and the container should be different, which is different from the human brain. It is thought that the different depths of convergence of the tree and the container match, which can relieve brain fatigue.
  • the different depths of convergence of the tree and the container match, which can relieve brain fatigue.
  • the trees farther from the user's gaze point are blurred, and the containers closer to the user's gaze point are clear, so that the virtual environment seen by the human eye is more in line with the real situation.
  • the VR glasses display a frame of image (ie, image 701 ) as an example.
  • general VR glasses display an image stream (such as an image stream generated by a VR game application).
  • An image stream includes multiple frames of images.
  • the user's gaze point may change.
  • VR glasses can detect the user's gaze point in real time through the eye tracking module.
  • the gaze point changes, the clarity of the object is determined based on the new gaze point. For example, objects far from the new fixation point are blurred, and objects closer to the new fixation point are sharp.
  • One possible implementation is that while the user's gaze remains on the gun, the VR glasses display an image stream, and trees (objects far away from the user's gaze) are blurred on each frame of the image stream. That is to say, during the period when the user's gaze is on the gun, objects far away from the user's gaze are always in a blurred state. Although this can relieve fatigue, it will lose object (eg, tree) details, so that the user may miss some details and cause the game to fail.
  • object eg, tree
  • the VR glasses display the image stream.
  • tree can be high or low, and does not need to be in a blurry state all the time.
  • the VR glasses display an image stream, and the image stream includes an i-th frame, a j-th frame, and a k-th frame.
  • the tree (the object far away from the user's gaze point) is blurred in the i-th frame image
  • the tree is clear in the j-th frame image
  • the tree is blurred in the k-th frame image.
  • the VR glasses display the image of the i-th frame
  • the tree that the user sees is blurred, which can relieve fatigue.
  • the VR glasses display the image of the j-th frame, the tree that the user sees is clear.
  • the human brain will fuse the i-th frame image with the tree on the j-th frame image, so although the tree in the i-th frame image is blurred, it can still ensure that the details of the tree will not be lost in the human brain. That is to say, in Fig. 7B, when the VR glasses display the image stream, for objects (such as trees) that are far away from the user's gaze point, the definition may be high or low, and it does not need to be in a blurred state all the time. In this way, not only It can relieve fatigue and ensure that the human brain can capture enough details of objects far away from the user's gaze point, and the user experience is better.
  • an object such as a container
  • its definition may not change.
  • its definition of the tree since the definition of the tree varies from high to low, its definition must not exceed that of the container.
  • the tree and the container are clear on the image frame j in FIG. 7B , but the resolution of the tree is lower than or equal to that of the container.
  • the sharpness can also be high or low, as long as the sharpness of objects close to the user's gaze point on the same image is higher than or equal to that of objects far away from the user's gaze point degree can be.
  • the tree is blurred on the i-th frame image in Figure 7B
  • the container can also be blurred, but the blurring degree of the container is lower than that of the tree, so that the definition of the container is higher than that of the tree.
  • the user's gaze point is on a gun as an example.
  • the user's gaze point can be on the game equipment (such as a gun) currently held by the game character corresponding to the user.
  • the point of gaze can also be at the game character corresponding to the opponent in the game, or at the building, or at the body part of the game character corresponding to the user, etc., and the embodiments of the present application do not give examples one by one.
  • FIG. 8A and FIG. 8B are examples of the second application scenario provided by the embodiment of the present application.
  • the user wears VR glasses for VR driving as an example.
  • the VR glasses display an image 801 .
  • the image 801 may be an image generated by a VR driving application.
  • the image 801 includes vehicle information such as a steering wheel and a monitor, and also includes roads, trees on the roads, vehicles in front, and the like.
  • the depth of field 2 where the tree is located is greater than the depth of field 1 where the vehicle in front is located. Therefore, the sharpness of the trees in image 801 is lower than that of the vehicle ahead. That is, objects with a larger depth of field are blurred, and objects with a smaller depth of field are sharper.
  • This application scenario is different from the application scenarios shown in Fig. 7A and Fig. 7B.
  • the user's gaze point shall prevail. Objects far away from the user's gaze point are blurred, and objects close to the user's gaze point are clear. Objects are clear and uncorrelated with the user's gaze point.
  • the VR glasses display a frame of image (that is, image 801 ) as an example.
  • the VR glasses can display an image stream (such as an image stream generated by a VR driving application).
  • the image stream includes multiple frames of images.
  • a possible implementation manner is that distant objects on each image frame in the image stream are blurred. Although this method can relieve fatigue, it will lose the details of distant objects.
  • the clarity of distant objects (such as trees) in the image stream can be high or low, and it does not need to be constantly fuzzy state.
  • the VR glasses display an image stream, and the image stream includes an i-th frame, a j-th frame, and a k-th frame.
  • the tree on the i-th frame image is blurred
  • the tree on the j-th frame image is clear
  • the tree on the k-th frame image is blurred.
  • the VR glasses display the i-th frame of image the tree that the user sees is blurred, which can relieve fatigue.
  • the VR glasses display the j-th frame of image the tree that the user sees is clear.
  • the tree on the i-th frame image and the j-th frame image will be fused, so although the tree in the i-th frame image is blurred, it can still ensure that the details of the tree will not be lost in the human mind. That is to say, in FIG. 8B , objects with larger depths of field in the image stream displayed by the VR glasses may have high or low clarity, and do not need to be in a blur state all the time. In this way, it can not only relieve fatigue, but also ensure that the details of distant objects can be captured in the human brain, and the user experience is better.
  • a nearby object eg, a vehicle in front
  • its clarity may also remain unchanged.
  • the sharpness of the distant object varies from high to low
  • the sharpness thereof must not exceed the sharpness of the near object.
  • the tree on the image frame j in FIG. 8B is clear, and the vehicle in front is also clear, but the clarity of the tree is lower than or equal to that of the vehicle in front.
  • the sharpness of nearby objects can also be high or low, as long as the sharpness of nearby objects on the same image is higher than or equal to the sharpness of distant objects.
  • the tree on the i-th frame image in Figure 8B is blurred, and the vehicle in front can also be blurred, but the blurring degree of the vehicle in front is lower than the blurring degree of the tree, so that the clarity of the tree is lower than that of the vehicle in front .
  • objects with a larger depth of field have higher blurring and lower definition, and objects with smaller depth of field have lower blurring and higher definition.
  • the tree is blurred and the vehicle in front is clear (the depth of field 2 where the tree is located is greater than the depth of field 1 where the vehicle in front is located) as an example.
  • the depth of field 2 is greater than the depth of field 1 object blurring.
  • the depth of field 3 may also be used as the criterion, and the objects at the depth of field 1 and the depth of field 2 that are greater than the depth of field 3 can be blurred.
  • the depth of field 3 is the depth of field where the user is currently driving the vehicle as an example.
  • the depth of field 3 can also be the depth of field where the steering wheel of the user is currently driving the vehicle, or the user is currently driving the vehicle. Depth of field where the windshield is, and so on.
  • the VR game application is taken as an example, and the user's gaze point shall prevail. Objects that are closer to the point of view are sharp, and, when the image stream is displayed, the sharpness of objects in the image stream that are farther from the user's gaze point increases and decreases.
  • the second application scenario taking the VR driving application as an example, the distant objects are blurred and the nearby objects are clear, and when the image stream is displayed, the clarity of the distant objects in the image stream increases or decreases. drop.
  • Objects far away from the user's gaze point are blurred, and objects close to the user's gaze point are clear.
  • the mode is called the first eye protection mode, and the mode in which the distant objects are blurred and the near objects are clear in the second application scene is called the second eye protection mode.
  • the above are the two application scenarios listed in this application, mainly taking VR games and VR driving as examples.
  • the first application scenario such as VR game application
  • the second application scenario can also apply the solution of the first application scenario to achieve the user's gaze point as the standard, Objects farther from the user's gaze are blurred, and objects closer to the user's gaze are sharp.
  • the technical solutions provided by the embodiments of the present application can be applied to other application scenarios, such as VR viewing of cars, VR viewing of houses, VR chatting, VR teaching, VR theaters, and any other scenarios that need to display a virtual environment to the user.
  • FIG. 9 shows a schematic structural diagram of a VR wearable device provided by an embodiment of the present application by taking a VR wearable device (such as VR glasses) as an example.
  • the VR wearable device 100 may include a processor 110, a memory 120, a sensor module 130 (which may be used to acquire the user's posture), a microphone 140, buttons 150, an input and output interface 160, a communication module 170, a camera 180, battery 190 , optical display module 1100 , eye tracking module 1200 and so on.
  • the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the VR wearable device 100 .
  • the VR wearable device 100 may include more or fewer components than shown in the illustration, or combine certain components, or separate certain components, or arrange different components.
  • the illustrated components can be realized in hardware, software or a combination of software and hardware.
  • the processor 110 is generally used to control the overall operation of the VR wearable device 100, and may include one or more processing units, for example: the processor 110 may include an application processor (application processor, AP), a modem processor, a graphics processor (graphics processing unit, GPU), image signal processor (image signal processor, ISP), video processing unit (video processing unit, VPU) controller, memory, video codec, digital signal processor (digital signal processor, DSP ), baseband processor, and/or neural network processor (neural-network processing unit, NPU), etc. Wherein, different processing units may be independent devices, or may be integrated in one or more processors.
  • application processor application processor, AP
  • modem processor graphics processor
  • ISP image signal processor
  • video processing unit video processing unit
  • VPU video processing unit
  • memory video codec
  • digital signal processor digital signal processor
  • DSP digital signal processor
  • baseband processor baseband processor
  • neural network processor neural-network processing unit
  • a memory may also be provided in the processor 110 for storing instructions and data.
  • the memory in processor 110 is a cache memory.
  • the memory may hold instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to use the instruction or data again, it can be called directly from the memory. Repeated access is avoided, and the waiting time of the processor 110 is reduced, thus improving the efficiency of the system.
  • the processor 110 may be used to control the optical power of the VR wearable device 100 .
  • the processor 110 may be used to control the optical power of the optical display module 1100 to realize the function of adjusting the optical power of the VR wearable device 100 .
  • the processor 110 can adjust the relative positions of the various optical devices (such as lenses, etc.) When the human eye is imaging, the position of the corresponding virtual image plane can be adjusted. In this way, the effect of controlling the optical power of the VR wearable device 100 is achieved.
  • processor 110 may include one or more interfaces.
  • the interface may include an integrated circuit (inter-integrated circuit, I2C) interface, a universal asynchronous receiver/transmitter (universal asynchronous receiver/transmitter, UART) interface, a mobile industry processor interface (mobile industry processor interface, MIPI), a general input and output (general -purpose input/output, GPIO) interface, subscriber identity module (subscriber identity module, SIM) interface, and/or universal serial bus (universal serial bus, USB) interface, serial peripheral interface (serial peripheral interface, SPI) interface etc.
  • I2C integrated circuit
  • MIPI mobile industry processor interface
  • GPIO general input and output
  • subscriber identity module subscriber identity module
  • SIM subscriber identity module
  • USB serial peripheral interface
  • SPI serial peripheral interface
  • the processor 110 may perform blurring processing to different degrees on objects at different depths of field, so that objects at different depths of field have different sharpness.
  • the I2C interface is a bidirectional synchronous serial bus, including a serial data line (serial data line, SDA) and a serial clock line (derail clock line, SCL).
  • processor 110 may include multiple sets of I2C buses.
  • the UART interface is a universal serial data bus used for asynchronous communication.
  • the bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication.
  • a UART interface is generally used to connect the processor 110 and the communication module 170 .
  • the processor 110 communicates with the Bluetooth module in the communication module 170 through the UART interface to realize the Bluetooth function.
  • the MIPI interface can be used to connect the processor 110 with the display screen in the optical display module 1100 , the camera 180 and other peripheral devices.
  • the GPIO interface can be configured by software.
  • the GPIO interface can be configured as a control signal or as a data signal.
  • the GPIO interface can be used to connect the processor 110 with the camera 180 , the display screen in the optical display module 1100 , the communication module 170 , the sensor module 130 , the microphone 140 and so on.
  • the GPIO interface can also be configured as an I2C interface, I2S interface, UART interface, MIPI interface, etc.
  • the camera 180 may collect images including real objects
  • the processor 110 may fuse the images collected by the camera with the virtual objects, and display the fused images through the optical display module 1100 .
  • the camera 180 may also collect images including human eyes.
  • the processor 110 performs eye tracking through the images.
  • the USB interface is an interface that conforms to the USB standard specification, specifically, it can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc.
  • the USB interface can be used to connect a charger to charge the VR wearable device 100, and can also be used to transmit data between the VR wearable device 100 and peripheral devices. It can also be used to connect headphones and play audio through them. This interface can also be used to connect other electronic devices, such as mobile phones.
  • the USB interface may be USB3.0, which is compatible with high-speed display port (DP) signal transmission, and can transmit video and audio high-speed data.
  • DP display port
  • the VR wearable device 100 may include a wireless communication function, for example, the VR wearable device 100 may receive images from other electronic devices (such as a VR host) for display.
  • the communication module 170 may include a wireless communication module and a mobile communication module.
  • the wireless communication function can be realized by an antenna (not shown), a mobile communication module (not shown), a modem processor (not shown), and a baseband processor (not shown).
  • Antennas are used to transmit and receive electromagnetic wave signals. Multiple antennas may be included in the VR wearable device 100, and each antenna may be used to cover a single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization of the antennas. For example: Antenna 1 can be multiplexed as a diversity antenna of a wireless local area network. In other embodiments, the antenna may be used in conjunction with a tuning switch.
  • the mobile communication module can provide applications on the VR wearable device 100 including second generation (2th generation, 2G) network/third generation (3th generation, 3G) network/fourth generation (4th generation, 4G) network/fifth generation (5th generation, 5G) network and other wireless communication solutions.
  • the mobile communication module may include at least one filter, switch, power amplifier, low noise amplifier (low noise amplifier, LNA) and the like.
  • the mobile communication module can receive electromagnetic waves through the antenna, filter and amplify the received electromagnetic waves, and send them to the modem processor for demodulation.
  • the mobile communication module can also amplify the signal modulated by the modem processor, and convert it into electromagnetic wave and radiate it through the antenna.
  • at least part of the functional modules of the mobile communication module may be set in the processor 110 .
  • at least part of the functional modules of the mobile communication module and at least part of the modules of the processor 110 may be set in the same device.
  • a modem processor may include a modulator and a demodulator.
  • the modulator is used for modulating the low-frequency baseband signal to be transmitted into a medium-high frequency signal.
  • the demodulator is used to demodulate the received electromagnetic wave signal into a low frequency baseband signal. Then the demodulator sends the demodulated low-frequency baseband signal to the baseband processor for processing.
  • the low-frequency baseband signal is passed to the application processor after being processed by the baseband processor.
  • the application processor outputs sound signals through audio equipment (not limited to speakers, etc.), or displays images or videos through the display screen in the optical display module 1100 .
  • the modem processor may be a stand-alone device. In some other embodiments, the modem processor may be independent from the processor 110, and be set in the same device as the mobile communication module or other functional modules.
  • the wireless communication module can provide wireless local area networks (wireless local area networks, WLAN) (such as wireless fidelity (Wireless fidelity, Wi-Fi) network), bluetooth (bluetooth, BT), global navigation satellite, etc. applied on the VR wearable device 100.
  • System global navigation satellite system, GNSS
  • frequency modulation frequency modulation, FM
  • near field communication technology near field communication, NFC
  • infrared technology infrared, IR
  • the wireless communication module may be one or more devices integrating at least one communication processing module.
  • the wireless communication module receives electromagnetic waves through the antenna, frequency-modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110 .
  • the wireless communication module can also receive the signal to be sent from the processor 110, frequency-modulate it, amplify it, and convert it into electromagnetic wave through the antenna to radiate out.
  • the antenna of the VR wearable device 100 is coupled to the mobile communication module, so that the VR wearable device 100 can communicate with the network and other devices through wireless communication technology.
  • the wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and/or IR technology, etc.
  • GNSS can include global positioning system (global positioning system, GPS), global navigation satellite system (global navigation satellite system, GLONASS), Beidou satellite navigation system (beidou navigation satellite system, BDS), quasi-zenith satellite system (quasi-zenith) satellite system (QZSS) and/or satellite based augmentation systems (SBAS).
  • GPS global positioning system
  • GLONASS global navigation satellite system
  • Beidou satellite navigation system beidou navigation satellite system, BDS
  • quasi-zenith satellite system quasi-zenith satellite system
  • QZSS quasi-zenith satellite system
  • SBAS satellite based augmentation systems
  • the VR wearable device 100 realizes the display function through the GPU, the optical display module 1100 , and the application processor.
  • the GPU is a microprocessor for image processing, and is connected to the optical display module 1100 and the application processor. GPUs are used to perform mathematical and geometric calculations for graphics rendering.
  • Processor 110 may include one or more GPUs that execute program instructions to generate or change display information.
  • the memory 120 may be used to store computer-executable program code, including instructions.
  • the processor 110 executes various functional applications and data processing of the VR wearable device 100 by executing instructions stored in the memory 120 .
  • the memory 120 may include an area for storing programs and an area for storing data.
  • the stored program area can store an operating system, at least one application program required by a function (such as a sound playing function, an image playing function, etc.) and the like.
  • the storage data area can store data (such as audio data, phone book, etc.) created during the use of the VR wearable device 100 .
  • the memory 120 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, flash memory device, universal flash storage (universal flash storage, UFS) and the like.
  • the VR wearable device 100 can implement audio functions through an audio module, a speaker, a microphone 140, an earphone interface, and an application processor. Such as music playback, recording, etc.
  • the audio module is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signal.
  • the audio module can also be used to encode and decode audio signals.
  • the audio module may be set in the processor 110 , or some functional modules of the audio module may be set in the processor 110 .
  • Loudspeakers also called “horns" are used to convert audio electrical signals into sound signals.
  • the wearable device 100 can listen to music through the speaker, or listen to hands-free calls.
  • the microphone 140 also called “microphone” or “microphone” is used to convert sound signals into electrical signals.
  • the VR wearable device 100 may be provided with at least one microphone 140 .
  • the VR wearable device 100 can be provided with two microphones 140, which can also implement a noise reduction function in addition to collecting sound signals.
  • the VR wearable device 100 can also be provided with three, four or more microphones 140 to collect sound signals, reduce noise, identify sound sources, and realize directional recording functions, etc.
  • the headphone jack is used to connect wired headphones.
  • the headphone interface can be a USB interface, or a 3.5mm (mm) open mobile terminal platform (OMTP) standard interface, a cellular telecommunications industry association of the USA (CTIA) standard interface .
  • mm 3.5mm
  • CTIA cellular telecommunications industry association of the USA
  • the VR wearable device 100 may include one or more buttons 150 , and these buttons may control the VR wearable device and provide users with access to functions on the VR wearable device 100 .
  • Keys 150 may be in the form of buttons, switches, dials, and touch or near-touch sensing devices such as touch sensors. Specifically, for example, the user can turn on the optical display module 1100 of the VR wearable device 100 by pressing a button.
  • the keys 150 include a power key, a volume key and the like.
  • the key 150 may be a mechanical key. It can also be a touch button.
  • the VR wearable device 100 can receive key input and generate key signal input related to user settings and function control of the VR wearable device 100 .
  • the VR wearable device 100 may include an input-output interface 160, and the input-output interface 160 may connect other devices to the VR wearable device 100 through suitable components.
  • Components may include, for example, audio/video jacks, data connectors, and the like.
  • the optical display module 1100 is used for presenting images to the user under the control of the processor 110 .
  • the optical display module 1100 can convert the real pixel image display into a near-eye projection virtual image display through one or several optical devices such as mirrors, transmission mirrors, or optical waveguides, so as to realize virtual interactive experience, or realize virtual and Interactive experience combined with reality.
  • the optical display module 1100 receives image data information sent by the processor 110 and presents corresponding images to the user.
  • the VR wearable device 100 may further include an eye tracking module 1200, which is used to track the movement of human eyes, and then determine the point of gaze of the human eyes.
  • the position of the pupil can be located by image processing technology, the coordinates of the center of the pupil can be obtained, and then the gaze point of the person can be calculated.
  • the following describes the display method of the embodiment of the present application by taking the VR wearable device shown in FIG. 9 as VR glasses as an example.
  • VR glasses display images to users, and different objects on the images have different clarity. For example, an object far from the user's gaze point on the image (referred to as a first object for convenience of description) is blurred, and an object close to the user's gaze point (referred to as a second object for convenience of description) is clear.
  • Embodiment 1 may be applicable to the application scenarios shown in FIG. 7A and FIG. 7B above.
  • the sharpness may include image resolution or scanning resolution, etc. In the following embodiments, the sharpness is described by taking the resolution of a displayed image as an example.
  • FIG. 11 is a schematic flowchart of an image generation method provided in Embodiment 1. As shown in Figure 11, the process includes:
  • the depth of each object can be automatically saved when the rendering pipeline is running, or can be calculated by relying on binocular vision, which is not limited in the embodiment of the present application.
  • S1103. Determine the blurring degree of the object according to the distance between the user's gaze point and the object.
  • the object when the distance between the depth of field where the object is located and the depth of field where the user's gaze is located is less than the preset distance, the object may not be blurred; when the distance between the depth of field where the object is located and the depth of field where the user's gaze is located is greater than the preset distance Blur the object at a distance.
  • the specific value of the preset distance is not limited in this embodiment of the present application.
  • the degree of blurring of different objects on the image may increase in order according to the distance between the depth of field where the object is located and the depth of field where the user's gaze point is located. For example, suppose the depth of field where the user gazes at is depth 1. The distance between the depth of field where object 1 is located and depth of field 1 is distance 1, the distance between the depth of field where object 2 is and depth 1 is distance 2, and the distance between the depth of field where object 3 is and depth 1 is distance 3. If distance 1 ⁇ distance 2 ⁇ distance 3, then the degree of blur of object 1 ⁇ the degree of blur of object 2 ⁇ the degree of blur of object 3, so that the sharpness of object 1>the sharpness of object 2>the sharpness of object 3. That is to say, in the virtual environment seen by the user's eyes, objects farther from the user's gaze point are more blurred, and objects closer to the user's gaze point are clearer.
  • the VR device may generate an image first, and all objects on the image have the same definition, and then use an image blurring algorithm to perform blurring processing to different degrees on different objects on the image.
  • the image blurring algorithm includes at least one of Gaussian blur, image down-sampling, defocus blur (defocus blur) algorithm based on deep learning, level of detail (level of detail, LOD) data structure, etc., and this application will not go into details .
  • the user's gaze point may change. As the user's gaze point changes, the sharpness of objects on the image adjusts accordingly. For example, continue to take Figure 10 as an example.
  • the clarity of each object is re-determined based on the new gaze point (ie, the tree). Objects far away from the point of view are blurred.
  • Figure 10 uses VR glasses to display a frame of image as an example. It can be understood that, generally, the VR glasses display an image stream, and the image stream includes multiple frames of images.
  • the VR image generation device uses the image stream generated by the process shown in FIG. 11 by default (that is, objects far away from the user’s gaze point on each frame of image are blurred) .
  • the VR image generation device uses the image stream generated by the process shown in FIG. 11 by default (that is, objects far away from the user’s gaze point on each frame of image are blurred) .
  • the VR image generation device takes the VR image generation device as a mobile phone as an example, when the mobile phone detects at least one of the connection of the VR glasses, the startup of the VR glasses, and the startup of the VR application (such as a VR game), the mobile phone starts to use the process shown in Figure 11 to generate images by default. stream, and then display it through VR glasses.
  • the VR image generation device uses the existing way to generate images by default (that is, all objects on the image have the same clarity), and when the instruction for starting the first eye protection mode is detected, the process shown in Figure 11 is used Generate an image.
  • the first eye protection mode please refer to the previous description. That is to say, all objects on the image displayed by the VR glasses at the beginning have the same clarity, and after the instruction for starting the first eye protection mode is detected, objects far away from the user's gaze point on the displayed image are blurred.
  • FIG. 12 before frame i+1, all objects on the image have the same definition.
  • the instruction for starting the first eye protection mode includes but is not limited to: detecting that the user triggers an operation for starting the first eye protection mode (for example, a VR application includes a button for starting the first eye protection mode, so The above operation may be an operation of clicking the button), the user's viewing time is greater than the preset time period, and the number of times the user blinks/squints within the preset time period is greater than the preset number of times.
  • the first eye protection mode is activated to relieve user fatigue.
  • a prompt message may also be output, the prompt message is used to prompt the user whether to switch to the first eye protection mode, after the user confirms the indication to switch to the first eye protection mode is detected , switch to the first eye protection mode.
  • objects far away from the user's gaze point in the image stream generated by the VR image generation device are blurred, which can relieve human brain fatigue, but it is easy to lose the details of objects far away from the user's gaze point.
  • the object far away from the user's gaze point is always blurred, and the user cannot obtain the details of the object;
  • the object at the foveated point will always be blurred, and the user will not be able to get the details of the object.
  • the definition of objects far away from the user's gaze point in the image stream generated by the VR image generation device may be high or low (see Figure 15 below for the specific generation process).
  • the image stream includes multiple cycles, and each cycle includes multiple frames of images, and in each cycle, the sharpness of objects far away from the user's gaze point on the image increases first and then decreases.
  • the resolution of the tree (object far away from the user's gaze point) on the image frame i is lower than that of the tree on the image frame j, and the tree on the image frame j
  • the sharpness of is higher than the sharpness of the tree on the kth frame image. That is, the sharpness of an object far away from the user's gaze point (that is, a tree) within a cycle shows a change trend of "fuzzy-clear-fuzzy".
  • the resolution of the tree on the image of the kth frame is lower than that of the tree on the image of the pth frame, and the resolution of the tree on the image of the pth frame is higher than that of the tree on the image of the qth frame, that is, the next During the cycle, the sharpness of objects far from the user's gaze point (that is, trees) also shows a "fuzzy-clear-fuzzy" change trend.
  • the two periods in FIG. 13 may be the same or different, without limitation.
  • this sharpness change trend can alleviate the fatigue of the human brain, and on the other hand, it can prevent the user from losing image details of objects far away from the user's gaze point.
  • the n frame, the kth frame is the m frame after the jth frame, the pth frame is the w frame after the kth frame, and the qth frame is the s frame after the pth frame.
  • n, m, q, s are integers greater than or equal to 1.
  • n, m, p, and s can be determined according to the user's visual dwell time and image refresh frame rate. Assume that the user's visual dwell time is T, and the image refresh frame rate is P. Wherein, the visual dwell time T may be any value within the range of 0.1s to 3s, or may be set by the user, which is not limited in this embodiment of the present application. Then, within T time, T/P frame images can be displayed, then n is less than or equal to T/P, m is less than or equal to T/P, q is less than or equal to T/P, and s is less than or equal to T/P.
  • the time difference between the display moment of the j-th frame image and the display moment of the i-th frame image is less than the user's visual dwell time.
  • the image information on the j-th frame image and the i-frame image can be combined Superposition, since the object far away from the user's gaze point on the i-th frame image is blurred, the j-th frame image's object far away from the user's gaze point is clear, and the superposition of the two images can ensure the accuracy of the object far away from the user's gaze point The details are sufficient.
  • the time difference between the display moment of the jth frame of image and the display moment of the kth frame of image is less than the user's visual dwell time, and will not be repeated here.
  • the following object is far from the user's gaze point is a lighthouse as an example to give a specific example. Please refer to Figure 14, the object (i.e. the lighthouse) far away from the user's gaze point on the image frame i is blurred, and the lighthouse on the image frame j is clear.
  • the sharpness of objects close to the user's gaze point in the image stream may not change.
  • the sharpness of the object that is, the mountain
  • the GPU outputs N frames of images (for the convenience of description, the images output by the GPU are referred to as original images), and the definition of all objects on the N frames of images output by the GPU is the same, and the first object (that is, far away from the user).
  • the objects at the fixation point, such as the tree in Fig. 13) are N frames of new pictures whose resolutions change alternately.
  • the new image of frame i is the blurred image of the first object on the original image of frame i;
  • the new image of frame j is the original image of frame j or the first object of the original image of frame j
  • the new image of the kth frame is the image after blurring the first object on the original image of the kth frame;
  • the new image of the pth frame is the original image of the pth frame or the original image of the pth frame
  • the first object in the i-th frame of the original image output by the GPU is blurred to obtain the i-th frame of the new image.
  • more details of the first object may be included in the new image of the jth frame.
  • One feasible way is to superimpose (or fuse) the new image of frame i and the original image of frame j output by the GPU to obtain the new image of frame j. Therefore, the definition of the first object on the new image of frame j It is higher than the original image of the jth frame and the new image of the ith frame.
  • only the image block in the area where the first object is located on the new image of the i-th frame can be combined with the original image of the j-th frame The image blocks in the region where the first object is located are superimposed.
  • VR glasses display images to users, and different objects on the images have different clarity. For example, an object with a larger depth of field (called a first object for convenience of description) on an image is blurred, and an object with a smaller depth of field (called a second object for convenience of description) is clear.
  • Embodiment 2 may be applicable to the application scenarios shown in FIG. 8A and FIG. 8B above.
  • the second depth of field where the mountain is located is greater than the first depth of field where the tree is located, so the mountain is blurred and the tree is clear. In this way, in the virtual environment that the user sees, the mountain is blurred and the tree is clear. .
  • FIG. 17 is a schematic flow diagram of the image generation method provided in the second embodiment. As shown in FIG. 17, the flow of the method includes:
  • the preset depth of field may be a specific depth of field value or a depth of field range, which is not limited in this embodiment of the present application.
  • the preset depth of field is used to judge which objects are distant objects and which objects are near objects. For example, an object whose depth of field is greater than the preset depth of field is a distant object, and an object whose depth of field is smaller than the preset depth of field is a near object.
  • distant objects may be blurred, but close-range objects may not be blurred.
  • There are multiple ways to determine the preset depth of field including but not limited to at least one of the following ways.
  • the preset depth of field may be determined according to a VR scene, and the preset depth of field varies with different VR scenes.
  • the VR scene includes but is not limited to at least one of VR games, VR viewing, VR teaching and the like.
  • the VR game includes game characters, and the preset depth of field can be determined according to the game characters.
  • the preset depth of field can be the depth of field of the game character corresponding to the user in the game scene, or the depth of field of the body parts of the game character corresponding to the user, or the depth of field of the game equipment currently held by the game character corresponding to the user .
  • the game character's arm is holding a gun, and the depth of field where the arm or gun is located can be determined as the preset depth of field.
  • the depth of field of the game character controlling the game can be used as the preset depth of field.
  • VR viewing includes a display screen, and the depth of field where the display screen is located can be determined as the preset depth of field.
  • VR teaching includes teaching equipment such as blackboards, display screens, and projections, and the depth of field where the teaching equipment is located can be determined as the preset depth of field.
  • the preset depth of field can be set by the user.
  • the user can set the preset depth of field on the VR glasses or an electronic device (such as a mobile phone) connected to the VR glasses.
  • the electronic device includes various VR applications, and different preset depths of field may be set for different VR applications.
  • the user can set the preset depth of field of the VR applications on the electronic device in batches, or can set individually for each VR application.
  • the preset depth of field can also be the default depth of field, which can be understood as the default setting of the VR glasses, or the default setting of the electronic device (such as a mobile phone) connected to the VR glasses, or the electronic device connected to the VR glasses ( For example, the VR application currently running on the mobile phone) is set by default, etc., which are not limited in this embodiment of the present application.
  • the preset depth of field may also be the depth of field where the virtual image plane is located. Taking FIG. 6C as an example, the depth of field where the dotted line surface is located is depth 1, so the preset depth of field may be depth 1.
  • the preset depth of field can also be based on the depth of field of the main object in the picture currently being displayed by the VR glasses.
  • the main object may include an object occupying the largest area in the screen, an object located in the center of the screen, or a virtual object (such as a UI interface) in the screen, and the like.
  • the image includes a tree, a house, and the sun. Assuming that the house is in the center, then the house is determined to be the main object, and the preset depth of field may be the depth of field where the house is located. Because the depth of field of the mountain is greater than that of the house, the mountain is blurred, and the depth of field of the tree is smaller than that of the house, so the tree is clear.
  • the preset depth of field is the depth of field where the user gazes.
  • the VR glasses may include an eye tracking module, through which the user's gaze point can be determined, and the depth of field at which the user's gaze point is determined is the preset depth of field.
  • the image includes trees, houses and the sun, assuming that the user's focus is on the house, then the preset depth of field may be the depth of field where the house is located.
  • the depth of field of the mountain is greater than that of the house, so the mountain is blurred, and the depth of field of the tree is smaller than that of the house, so the tree is clear.
  • the depth of each object can be automatically saved when the rendering pipeline is running, or can be calculated by relying on binocular vision, which is not limited in the embodiment of the present application.
  • S1703. Determine the blurring degree of the object according to the distance between the depth of the object and a preset depth of field.
  • the object when the depth of the object is less than or equal to the preset depth, the object may not be blurred; when the depth of the object is greater than the preset depth, the object needs to be blurred.
  • the blurring degrees of different objects on the image may increase sequentially from small to large depth of field.
  • the depth of field 1 where the object 1 is located ⁇ the depth of field 2 where the object 2 is located ⁇ the depth of field 3 where the object 3 is located
  • the degree of blurring of the object 1 ⁇ the degree of blurring of the object 2 ⁇ the degree of blurring of the object 3 .
  • the sharpness among objects in the foreground, objects in the middle, and objects in the foreground decreases in turn.
  • the VR device can first generate an image, and then use an image blurring algorithm to blur different objects on the image to different degrees.
  • the image blurring algorithm includes at least one of Gaussian blur, image down-sampling, defocus blur (defocus blur) algorithm based on deep learning, level of detail (LOD) data structure, etc. limited.
  • LOD is a multi-layer data structure.
  • the data structure can be understood as an image processing algorithm, and the multi-layer data structure includes a multi-layer image processing algorithm.
  • LOD0 to LOD3 are included; wherein, each layer in LOD0 to LOD3 corresponds to an image processing algorithm.
  • different layers in LOD0 to LOD3 correspond to different algorithms, specifically, the higher the layer, the simpler the corresponding image processing algorithm. For example, LOD3 has the highest level, and the corresponding image processing algorithm is the simplest; LOD0 has the lowest level, and the corresponding image processing algorithm is the most complex.
  • LODs can be used to generate 3D images.
  • LOD0 to LOD3 as an example; wherein, each layer in LOD0 to LOD3 can be used to generate a layer in a 3D image, and then different layers are used to synthesize a 3D image; wherein, different layers correspond to different depth ranges.
  • the image depth can be divided according to the number of LOD levels, for example, there are four layers LOD0 to LOD3, and the image depth can be divided into four ranges.
  • LOD0 corresponds to depth range 1, that is, the image processing algorithm corresponding to LOD0 is used to process layers in depth range 1
  • LOD1 corresponds to depth range 2, that is, the image processing algorithm corresponding to LOD1 is used to process layers in depth range 2 layer for processing
  • LOD2 corresponds to depth range 3, that is, the image processing algorithm corresponding to LOD2 is used to process layers in depth range 3
  • LOD3 corresponds to depth range 4, that is, the image processing algorithm corresponding to LOD3 is used to process layers in depth range 3 Layers within range 4 are processed. Because this application considers that objects with greater depth are more blurred.
  • a layer with a larger depth range corresponds to a higher-level LOD layer (the higher the LOD level, the simpler the algorithm, see the previous description).
  • a layer with a smaller depth range corresponds to a lower-level LOD layer, because the lower the level The more complex the algorithm corresponding to the lower LOD layer, the clearer the processed layer.
  • the depth range 1 is 0-0.3m, which corresponds to LOD0 (because the layer generated by LOD0 has the highest clarity).
  • the depth range 2 is 0.3-0.5m, which corresponds to LOD1 (the layer generated by LOD1 has a lower resolution than the layer generated by LOD0).
  • Depth range 3 is 0.5-0.8m, corresponding to LOD2 (the definition of the layer generated by LOD2 is lower than that of LOD1 layer).
  • the depth range 3 is 0.8-1m, corresponding to LOD3 (the definition of the layer generated by LOD3 is lower than that of the LOD2 layer). That is, as the depth increases, the sharpness of the layer becomes lower and lower.
  • layers corresponding to different depth ranges synthesize an image, and the image is displayed on a VR display device.
  • Figure 16 uses VR glasses to display a frame of image as an example. It can be understood that, generally, the VR glasses display an image stream, and the image stream includes multiple frames of images.
  • the VR image generating device uses the image stream generated by the process shown in FIG. 17 by default (that is, the distant objects on each frame of image are blurred).
  • the VR image generation device uses the image stream generated by the process shown in FIG. 17 by default (that is, the distant objects on each frame of image are blurred).
  • the mobile phone detects at least one of the connection of the VR glasses, the startup of the VR glasses, and the startup of the VR application (such as a VR game)
  • the mobile phone starts to use the process shown in Figure 17 to generate images by default. stream, and then display it through VR glasses.
  • the VR image generation device uses the existing method to generate images by default (that is, all objects on the image have the same clarity), and when detecting the indication for starting the second eye protection mode, use the process shown in Figure 17 to generate image.
  • the second eye protection mode please refer to the previous description. That is to say, the resolution of all objects on the image displayed by the VR glasses at the beginning is the same. Vague.
  • FIG. 18 before the i+1th frame, all objects on the image have the same definition.
  • distant objects such as mountains and the sun
  • the indication for starting the second mode includes but is not limited to: detecting that the user triggers an operation for starting the second mode (for example, a VR application includes a button for starting the second eye protection mode, and the operation can be is the operation of clicking the button), the viewing time of the user is greater than the preset duration, and the number of times the user blinks/squints within the preset duration is greater than the preset number of times.
  • prompt information may also be output, which is used to prompt the user whether to switch to the second eye protection mode. , switch to the second eye protection mode. Since the second eye protection mode is activated, distant objects (such as mountains) on the image are blurred, so the fatigue of the human brain is relieved, and the user experience is better.
  • the distant objects in the image stream generated by the VR image generation device are blurred, which can relieve the fatigue of the human brain, but it is easy to lose the details of the distant objects.
  • the distant object is always blurred, and the user cannot obtain the details of the object;
  • the distant object will always be blurred. is blurred, and the user cannot obtain the details of the object.
  • the clarity of distant objects in the image stream generated by the VR image generating device in the first manner or the second manner may be high or low.
  • the image stream includes multiple periods, and each period includes multiple frames of images, and in each period, the definition of the first object on the image increases first and then decreases.
  • the sharpness of distant objects (such as mountains) on the i-th frame image is lower than the sharpness of the j-th frame image on the mountain, and the definition of the j-th frame image on the mountain is The sharpness is higher than that of the kth frame image uphill. That is, the sharpness of distant objects shows a change trend of "fuzzy-clear-fuzzy" within a cycle.
  • the sharpness of the k-th frame image uphill is lower than that of the p-th frame image uphill, and the p-th frame image's sharpness of the uphill is higher than the qth frame image's sharpness of the uphill.
  • the sharpness of distant objects also shows a change trend of "fuzzy-clear-fuzzy".
  • the two periods in Fig. 19 may be the same or different, without limitation.
  • this sharpness change trend can alleviate the fatigue of the human brain, and on the other hand, it can prevent users from losing image details of distant objects.
  • n, m, q, s are integers greater than or equal to 1.
  • n, m, p, and s are all 1, that is, the j-th frame image is the next frame of the i-th frame image, the k-th frame image is the next frame of the j-th frame image, and the p-th frame is the k-th frame The next frame of , the qth frame is the next frame of the pth frame.
  • n, m, p, and s can be determined according to the user's visual dwell time and image refresh frame rate, which are the same as the implementation principle of Embodiment 1, and will not be repeated.
  • the definition of objects in the foreground in the image stream may not change.
  • the clarity of the tree may not change.
  • Embodiment 1 and Embodiment 2 can be implemented independently or in combination.
  • the VR image generation device may use the technical solution of Embodiment 1 by default (such as the first method or the second method in Embodiment 1), or use the technical solution of Embodiment 2 by default (such as the first method in Embodiment 2).
  • One method or the second method) or, the VR image generating device includes a switching button, through which the VR image generating device can be set to use the technical solution of the first embodiment or the technical solution of the second embodiment; or, the VR application includes a button, through which the user can set whether the VR application uses the technical solution of Embodiment 1 or the technical solution of Embodiment 2.
  • the VR glasses have two display screens, a first display screen and a second display screen.
  • the first display screen is used to present images to the user's left eye
  • the second display screen is used to present images to the user's right eye.
  • the display screen corresponding to the left eye is referred to as the left-eye display screen
  • the display screen corresponding to the right eye is referred to as the right-eye display screen.
  • the left-eye display and the right-eye display are used to display image streams, respectively.
  • the image stream may be an image stream generated using the method of Embodiment 1 (such as the image stream shown in FIG. 13 or 12 ), or an image stream generated using the method of Embodiment 2 (such as FIG. 18 or image stream shown in Figure 19).
  • the image stream displayed on the left-eye display screen and the image stream displayed on the right-eye display screen are both the image streams shown in FIG. 13 in the first embodiment.
  • the images displayed on the left-eye display screen and the right-eye display screen are synchronized.
  • the i-th frame of image is displayed on the left-eye display screen
  • the i-th frame of image is also displayed on the right-eye display screen. Since the object (for example, a tree) away from the user's gaze point on the i-th frame image is blurred, at this time, the trees seen by the left eye and the right eye are both blurred.
  • the left-eye display screen displays the j-th frame of image
  • the right-eye display screen also displays the j-th frame of image.
  • the trees seen by the left eye and the right eye are both clear at this time.
  • the left and right eye display screens display the i-th frame of image synchronously, objects far away from the user's gaze point on the image obtained by synthesizing the left and right eye display screen images in the human brain will be blurred, which can relieve fatigue.
  • the left and right eye display screens synchronously display the j-th frame of image, the image obtained by synthesizing the images of the left and right eye display screens in the human brain is clear, and the objects far away from the user's gaze point can be obtained. detail.
  • the sharpness change trend of objects far away from the user's gaze point in the image stream displayed on the left-eye display screen and the image stream displayed on the right-eye display screen is the same, and both are "blurry-clear-blurry-clear" change trends.
  • the sharpness change trends of objects far away from the user's gaze point in the image stream displayed on the left-eye display screen and the image stream displayed on the right-eye display screen may be opposite.
  • the sharpness of objects far away from the user's gaze in the image stream displayed on the left-eye display shows an alternating change of "fuzzy-clear-fuzzy-clear”
  • the sharpness of objects far away from the user's gaze in the image stream displayed on the right-eye display shows "Clear-blur-clear-blur” alternate change.
  • the i-th frame image is displayed on the left-eye display screen
  • the i-th frame image is also displayed on the right-eye display screen.
  • Objects far away from the user's gaze point (for example, trees) on the i-th frame image on the left-eye display screen are blurred, and the tree is clear in the i-th frame image on the right-eye display screen. Therefore, at this time, the tree seen by the left eye is blurred, and the tree seen by the right eye is clear.
  • the left-eye display screen displays the j-th frame of image
  • the right-eye display screen also displays the j-th frame of image.
  • the tree is clear in the jth frame image on the left-eye display screen, and the tree is blurred in the j-th frame image on the right-eye display screen.
  • the tree seen by the left eye is clear, and the tree seen by the right eye is blurred.
  • the left and right eye display screens display images synchronously (the i-th frame image or the j-th frame image)
  • the objects far away from the fixation point on the left-eye image are clear, and the objects far from the fixation point on the right-eye image are blurred.
  • it can alleviate the fatigue of the human brain, and the image obtained by superimposing the left-eye image and the right-eye image in the human brain will not be too blurred for the object far away from the gaze point, and avoid losing too much detail of the object.
  • both the image stream displayed on the left-eye display screen and the image stream displayed on the right-eye display screen are the image streams shown in FIG. 19 in the second embodiment.
  • the images displayed on the left-eye display screen and the right-eye display screen are synchronized.
  • the i-th frame image is displayed on the left-eye display screen
  • the i-th frame image is also displayed on the right-eye display screen. Since the distant objects (such as mountains and the sun) are blurred on the i-th frame image, at this time, the distant objects seen by the left eye and the right eye are both blurred.
  • the left-eye display screen displays the j-th frame of image
  • the right-eye display screen also displays the j-th frame of image.
  • the distant objects eg, the mountain and the sun
  • the distant objects seen by the left eye and the right eye are both clear.
  • the human brain synthesizes the images of the left and right eye display screens to obtain blurred distant objects on the image, which can relieve fatigue.
  • the left and right eye display screens synchronously display the j-th frame of image
  • the image obtained by combining the images of the left and right eye display screens in the human brain is clear, and the details of the distant object can be obtained.
  • the image stream displayed on the left-eye display screen and the image stream displayed on the right-eye display screen have the same changing trend of the sharpness of distant objects, both of which are "fuzzy-clear-fuzzy-clear".
  • the sharpness change trend of the distant object in the image stream displayed on the left-eye display screen and the image stream displayed on the right-eye display screen may be opposite.
  • the sharpness of distant objects in the image stream displayed on the left-eye display shows an alternating change of "blur-clear-blur-clear”
  • the image stream displayed on the right-eye display shows “clear-blur-clear-blur" " Alternate changes.
  • the i-th frame image when the i-th frame image is displayed on the left-eye display screen, the i-th frame image is also displayed on the right-eye display screen.
  • the distant objects (such as mountains and the sun) are blurred on the i-th frame image on the left-eye display screen, and the distant objects are clear in the i-th frame image on the right-eye display screen. Therefore, at this time, the distant objects seen by the left eye are blurred, and the distant objects seen by the right eye are clear.
  • the left-eye display screen displays the j-th frame of image
  • the right-eye display screen also displays the j-th frame of image.
  • the distant objects (such as mountains and the sun) are clear on the jth frame image on the left-eye display screen, and the distant objects (such as mountain and sun) are blurred in the j-th frame image on the right-eye display screen. Therefore, at this time, the distant objects seen by the left eye are clear, and the distant objects seen by the right eye are blurred.
  • the left-eye and right-eye display screens display images synchronously (i-th frame image or j-th frame image)
  • the distant objects on the left-eye image are clear, and the distant objects on the right-eye image are blurred, which can be achieved to a certain extent. Relieve the fatigue of the human brain, and the image of the distant object on the image obtained by superimposing the left eye image and the right eye image in the human brain will not be too blurred, and avoid losing too much detail of the object.
  • FIG. 24 shows an electronic device 2400 provided by this application.
  • the electronic device 2400 may be the aforementioned VR wearable device (eg, VR glasses).
  • the electronic device 2400 may include: one or more processors 2401; one or more memories 2402; a communication interface 2403, and one or more computer programs 2404, and each of the above devices may communicate through one or more bus 2405 connection.
  • the one or more computer programs 2404 are stored in the above-mentioned memory 2402 and configured to be executed by the one or more processors 2401, the one or more computer programs 2404 include instructions, and the above-mentioned instructions can be used to perform the above-mentioned Relevant steps of the mobile phone in the corresponding embodiment.
  • the communication interface 2403 is used to implement communication with other devices, for example, the communication interface may be a transceiver.
  • the methods provided in the embodiments of the present application are introduced from the perspective of an electronic device (such as a mobile phone) as an execution subject.
  • the electronic device may include a hardware structure and/or a software module, and realize the above-mentioned functions in the form of a hardware structure, a software module, or a hardware structure plus a software module. Whether one of the above-mentioned functions is executed in the form of a hardware structure, a software module, or a hardware structure plus a software module depends on the specific application and design constraints of the technical solution.
  • the terms “when” or “after” may be interpreted to mean “if” or “after” or “in response to determining" or “in response to detecting ".
  • the phrases “in determining” or “if detected (a stated condition or event)” may be interpreted to mean “if determining" or “in response to determining" or “on detecting (a stated condition or event)” or “in response to detecting (a stated condition or event)”.
  • relational terms such as first and second are used to distinguish one entity from another, without limiting any actual relationship and order between these entities.
  • references to "one embodiment” or “some embodiments” or the like in this specification means that a particular feature, structure, or characteristic described in connection with the embodiment is included in one or more embodiments of the present application.
  • appearances of the phrases “in one embodiment,” “in some embodiments,” “in other embodiments,” “in other embodiments,” etc. in various places in this specification are not necessarily All refer to the same embodiment, but mean “one or more but not all embodiments” unless specifically stated otherwise.
  • the terms “including”, “comprising”, “having” and variations thereof mean “including but not limited to”, unless specifically stated otherwise.
  • all or part of them may be implemented by software, hardware, firmware or any combination thereof.
  • software When implemented using software, it may be implemented in whole or in part in the form of a computer program product.
  • the computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on the computer, all or part of the processes or functions described in this embodiment will be generated.
  • the computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable devices.
  • the computer instructions may be stored in or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website, computer, server or data center Transmission to another website site, computer, server, or data center by wired (eg, coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (eg, infrared, wireless, microwave, etc.).
  • the computer-readable storage medium may be any available medium that can be accessed by a computer, or a data storage device such as a server or a data center integrated with one or more available media.
  • the available medium may be a magnetic medium (for example, a floppy disk, a hard disk, or a magnetic tape), an optical medium (for example, DVD), or a semiconductor medium (for example, a Solid State Disk (SSD)).
  • a magnetic medium for example, a floppy disk, a hard disk, or a magnetic tape
  • an optical medium for example, DVD
  • a semiconductor medium for example, a Solid State Disk (SSD)

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Optics & Photonics (AREA)
  • Human Computer Interaction (AREA)
  • Processing Or Creating Images (AREA)

Abstract

A display method and an electronic device, which are used to alleviate the feeling of human brain fatigue that occurs when a user wears VR glasses to watch a VR scene. The method comprises: displaying N image frames to a user by means of a display device, wherein on an i-th image frame among the N image frames, the definition of a first object positioned in a first depth of field is a first definition, on a j-th image frame among the N image frames, the definition of the first object positioned in the first depth of field is a second definition, and on a k-th image frame among the N image frames, the definition of the first object positioned in the first depth of field is a third definition, the first definition being less than the second definition, and the second definition being greater than the third definition, i, j, k all being positive integers smaller than N, and i＜j＜k; and the first depth of field is greater than a second depth of field, or the distance between the first depth of field and the depth of field in which the gaze of the user is located is greater than a first distance.

Description

一种显示方法与电子设备A display method and electronic device
相关申请的交叉引用Cross References to Related Applications
本申请要求在2021年07月21日提交中国专利局、申请号为202110824187.7、申请名称为“一种显示方法与电子设备”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。This application claims the priority of the Chinese patent application with the application number 202110824187.7 and the application title "A Display Method and Electronic Device" submitted to the China Patent Office on July 21, 2021, the entire contents of which are incorporated in this application by reference .
技术领域technical field
本申请涉及电子技术领域，尤其涉及一种显示方法与电子设备。The present application relates to the field of electronic technology, in particular to a display method and electronic equipment.
背景技术Background technique
虚拟现实(Virtual Reality，VR)技术是借助计算机及传感器技术创造的一种人机交互手段。VR技术综合了计算机图形技术、计算机仿真技术、传感器技术、显示技术等多种科学技术，可以创建虚拟环境，用户通过佩戴VR眼镜沉浸于虚拟环境中。虚拟环境是通过许多张三维图像不断刷新而呈现出来的，三维图像中包括处于不同景深的对象，给用户带来立体感。Virtual Reality (VR) technology is a means of human-computer interaction created with the help of computer and sensor technology. VR technology integrates computer graphics technology, computer simulation technology, sensor technology, display technology and other scientific technologies to create a virtual environment, and users can immerse themselves in the virtual environment by wearing VR glasses. The virtual environment is presented through continuous refreshing of many three-dimensional images, and the three-dimensional images include objects in different depths of field, giving users a sense of three-dimensionality.
现实生活中，人眼观看物体的时候会同时进行辐辏调节(双眼在观察物体时左眼视线和右眼视线均指向物体)和焦点调节(调节晶状体，将光线聚焦到视网膜上)，如图1所示。一般辐辏调节和焦距调节是一致的，这是人类正常的生理机制。In real life, when the human eye looks at an object, it will simultaneously perform convergence adjustment (the sight of the left eye and the right eye point to the object when both eyes are observing the object) and focus adjustment (adjust the lens to focus the light on the retina), as shown in Figure 1 shown. Generally, vergence adjustment and focal length adjustment are consistent, which is a normal physiological mechanism of human beings.
但是，在VR眼镜所呈现的虚拟环境中，用户看到的景物都是由VR眼镜的显示屏显示出来的。屏幕发出的光线并没有深度差异，所以经过焦距调节使得眼睛焦点就定在屏幕上。但用户实际所看到的虚拟环境中对象深度与显示屏距离用户的距离并不相同，所以辐辏调节与焦点调节不一致，这种现象称为视觉辐辏调节冲突(Vergence accommodation conflict，VAC)。VAC会给用户带来疲劳感和眩晕感，因为人脑无法准确的判断对象的真实深度，如果长时间如此，还会对用户的视力产生严重的影响。However, in the virtual environment presented by the VR glasses, the scenery seen by the user is displayed by the display screen of the VR glasses. The light emitted by the screen has no depth difference, so after adjusting the focal length, the focus of the eyes is fixed on the screen. However, the depth of objects in the virtual environment that the user actually sees is not the same as the distance from the display screen to the user, so the vergence adjustment is inconsistent with the focus adjustment. This phenomenon is called Vergence accommodation conflict (VAC). VAC will bring fatigue and dizziness to the user, because the human brain cannot accurately judge the true depth of the object, and if it does so for a long time, it will also have a serious impact on the user's vision.
发明内容Contents of the invention
本申请的目的在于提供了一种显示方法与电子设备，用于提升VR体验。The purpose of the present application is to provide a display method and an electronic device for improving VR experience.
第一方面，提供一种显示方法。该方法包括：通过显示设备向用户展示N帧图像；N为正整数；所述N帧图像中第i帧图像上，处于第一景深处的第一对象的清晰度为第一清晰度；所述N帧图像中第j帧图像上，处于所述第一景深处的所述第一对象的清晰度为第二清晰度；所述N帧图像中第k帧图像上，处于所述第一景深处的所述第一对象的清晰度为第三清晰度；其中，所述第一清晰度小于所述第二清晰度，所述第二清晰度大于所述第三清晰度，i、j、k均为小于N的正整数，i＜j＜k；其中，所述第一景深大于第二景深，或者，所述第一景深与所述用户注视点所在景深之间的距离大于第一距离。In a first aspect, a display method is provided. The method includes: displaying N frames of images to the user through a display device; N is a positive integer; on the i-th frame of the N frames of images, the definition of the first object in the first depth of field is the first definition; In the j-th frame of the N frames of images, the definition of the first object in the first depth of field is the second definition; in the k-th frame of the N frames of images, in the first The sharpness of the first object in the depth of field is the third sharpness; wherein, the first sharpness is smaller than the second sharpness, and the second sharpness is greater than the third sharpness, i, j , k are all positive integers less than N, i<j<k; wherein, the first depth of field is greater than the second depth of field, or, the distance between the first depth of field and the depth of field where the user gaze point is greater than the first distance.
需要说明的是，以VR显示设备为例，VR显示设备显示的图像流中每帧图像上所有对象的清晰度相同的话，会导致大脑无法准确的判断不同对象的深度，导致大脑出现疲劳感。在本申请实施例中，VR显示设备(如，VR眼镜)向用户展示的图像流(即，N帧图 像)中处于第一景深处的第一对象的清晰度有高有低。比如，景深较大的第一对象或者距离用户注视点较远的第一对象的清晰度有高有低。当VR显示设备显示第i帧或第k帧图像时，由于第一对象清晰度比较低，可以缓解人脑疲劳。当VR显示设备显示第j帧图像时，由于第j帧图像上第一对象清晰度较高，所以大脑能够摄取到第一对象的细节，比较第一对象的细节不丢失。因此，通过这种方式，既可以缓解人脑疲劳，又可以保证人脑摄取到对象的足够细节，用户体验较好。It should be noted that, taking a VR display device as an example, if all the objects on each frame of the image stream displayed by the VR display device have the same definition, the brain will not be able to accurately judge the depth of different objects, resulting in a sense of brain fatigue. In this embodiment of the present application, the definition of the first object in the first depth of field in the image stream (that is, N frames of images) displayed by the VR display device (for example, VR glasses) to the user varies from high to low. For example, the definition of the first object with a larger depth of field or the first object farther away from the user's gaze point varies from high to low. When the VR display device displays the i-th frame or the k-th frame of images, since the definition of the first object is relatively low, it can relieve human brain fatigue. When the VR display device displays the jth frame of image, since the first object has a higher definition on the jth frame of image, the brain can capture the details of the first object, and the details of the first object will not be lost. Therefore, in this way, the fatigue of the human brain can be relieved, and it can also ensure that the human brain can absorb enough details of the object, and the user experience is better.
在一种可能的设计中，在显示所述第i帧图像、所述第j帧图像以及所述第k帧图像期间，所述用户注视点不变。也就是说，用户佩戴VR眼镜观看虚拟环境时，如果用户注视点不发生变化，那么远离用户注视点的第一对象的清晰度有高有低(或者说有升有降)。比如，当VR眼镜显示第i帧或第k帧图像时，由于第一对象清晰度比较低，可以缓解人脑疲劳。当显示第j帧图像时，由于第j帧图像上第一对象是清晰的，这样可以避免丢失第一对象的细节，保证大脑中可以摄取到第一对象的足够细节。通过这种方式，既可以缓解人脑疲劳，又可以保证人脑摄取到对象的足够细节，用户体验较好。In a possible design, during the period of displaying the i-th frame image, the j-th frame image and the k-th frame image, the gaze point of the user remains unchanged. That is to say, when the user wears VR glasses to watch the virtual environment, if the user's gaze point does not change, the definition of the first object far away from the user's gaze point may be high or low (or rise or fall). For example, when the VR glasses display the i-th frame or the k-th frame of images, since the definition of the first object is relatively low, it can relieve human brain fatigue. When the j-th frame image is displayed, since the first object is clear on the j-th frame image, the details of the first object can be avoided from being lost, and sufficient details of the first object can be captured in the brain. In this way, it can not only alleviate the fatigue of the human brain, but also ensure that the human brain can absorb enough details of the object, and the user experience is better.
可以理解的是，当用户关注点发生变化时，第一景深对应变化。比如，当用户注视点在对象A时，假设对象A所在景深为0.5m，第一景深与对象A所在景深之间距离大于第一距离(比如，0.5m)，所以第一景深大于或等于1m；当用户注视点变化为对象B时，假设对象B所在景深为0.8m，第一景深与对象B所在景深之间距离大于第一距离(比如，0.5m)，所以第一景深变为大于或等于1.3m；因此，随着用户关注点的变化，第一景深发生变化。It can be understood that when the user's point of focus changes, the first depth of field changes accordingly. For example, when the user's gaze point is on object A, assuming that the depth of field where object A is located is 0.5m, the distance between the first depth of field and the depth of field where object A is located is greater than the first distance (for example, 0.5m), so the first depth of field is greater than or equal to 1m ; When the user's gaze point changes to object B, assuming that the depth of field where object B is located is 0.8m, the distance between the first depth of field and the depth of field where object B is located is greater than the first distance (for example, 0.5m), so the first depth of field becomes greater than or equals 1.3m; therefore, as the user's focus changes, the first depth of field changes.
在一种可能的设计中，所述第二景深包括：用户指定景深、用户注视点所在景深、***默认景深、虚像面所在景深、与虚拟场景对应的景深和所述第i帧图像上主体对象所在景深中的至少一种。也就是说，显示设备显示的图像流中景深大于第二景深的第一对象的清晰度有高有低。其中，第二景深可以有多种方式确定，比如，方式一，第二景深可以是根据VR场景确定，预设景深随VR场景的不同而不同。其中，所述VR场景包括但不限定于VR游戏、VR观影、VR教学等中的至少一种。方式二，第二景深可以是用户设置的。应理解，不同的VR应用可以设置不同的第二景深。方式三，第二景深还可以是默认景深。方式四，第二景深还可以是虚像面所在景深。方式五，第二景深还可以是VR显示设备当前正显示的画面中主体对象所在景深。方式六、第二景深是用户注视点所在景深。需要说明的是，以上列举了几种第二景深的确定方式，但本申请实施例不限定于以上方式，其它的确定第二景深的方式也是可行的。In a possible design, the second depth of field includes: the depth of field specified by the user, the depth of field of the user's gaze point, the default depth of field of the system, the depth of field of the virtual image plane, the depth of field corresponding to the virtual scene, and the subject object on the i-th frame image At least one of the depths of field. That is to say, in the image stream displayed by the display device, the definition of the first object whose depth of field is greater than the second depth of field varies from high to low. Wherein, the second depth of field may be determined in multiple ways, for example, method 1, the second depth of field may be determined according to a VR scene, and the preset depth of field varies with different VR scenes. Wherein, the VR scene includes but is not limited to at least one of VR games, VR viewing, VR teaching and the like. In the second manner, the second depth of field may be set by the user. It should be understood that different VR applications may set different second depths of field. Mode 3, the second depth of field may also be the default depth of field. Mode 4, the second depth of field may also be the depth of field where the virtual image plane is located. Mode 5, the second depth of field may also be the depth of field where the main object is located in the picture currently being displayed by the VR display device. Method 6. The second depth of field is the depth of field where the user's gaze point is located. It should be noted that several methods for determining the second depth of field are listed above, but the embodiments of the present application are not limited to the above methods, and other methods for determining the second depth of field are also feasible.
在一种可能的设计中，在显示所述N帧图像之前，所述方法还包括：检测到用户触发用于启动护眼模式的操作、用户观看时间大于第一时长、第二时长内用户眼睛眨眼/眯眼次数大于第一次数中的至少一项。举例来说，显示设备刚开始使用相同的清晰度进行显示，即显示的图像流中所有对象的清晰度相同。当检测到用于启动护眼模式的操作、用户观看时间大于第一时长、第二时长内用户眼睛眨眼/眯眼次数大于第一次数中的至少一项时，显示的图像流中第一景深处的第一对象的清晰度有升有降。这样可以缓解人脑疲劳，同时保证人脑摄取到对象的足够细节。而且，在用户需要的时候(比如，用户眼睛眨眼/眯眼次数增大)，启动本申请的技术方案(图像流中第一景深处的第一对象的清晰度有升有降)，节省图像处理(比如第i帧和第k帧图像上第一对象模糊化处理)带来的功耗。In a possible design, before displaying the N frames of images, the method further includes: detecting that the user triggers an operation for starting the eye protection mode, the user's viewing time is greater than the first duration, and the user's eyesight within the second duration At least one of the number of blinks/squints is greater than the first number. For example, the display device initially displays with the same resolution, that is, all objects in the displayed image stream have the same resolution. When at least one of the operation for starting the eye protection mode is detected, the user’s viewing time is greater than the first duration, and the number of blinks/squints of the user’s eyes within the second duration is greater than the first number, the first in the displayed image stream The sharpness of the first object in the depth of field goes up and down. This can relieve the fatigue of the human brain and at the same time ensure that the human brain can absorb enough details of the object. Moreover, when the user needs it (for example, the number of times the user blinks/squints increases), the technical solution of the present application is started (the definition of the first object at the first depth of field in the image stream increases or decreases), saving images. Power consumption caused by processing (such as blurring the first object on the i-th frame and the k-th frame image).
在一种可能的设计中，所述N帧图像中处于第三景深处的第二对象的清晰度相同。也 就是说，图像流中处于第一景深处的第一对象的清晰度有高有低，而处于第三景深处的第二对象的清晰度相同。In a possible design, the clarity of the second object in the third depth of field in the N frames of images is the same. That is to say, in the image stream, the first object at the first depth of field has higher or lower sharpness, while the second object at the third depth of field has the same sharpness.
比如，所述第三景深小于所述第一景深。即，景深较大的第一对象的清晰度有高有低，景深较小的第二对象的清晰度不变。一般来说，人眼看近处物体时，看到的细节越多越清楚，看远处物体时，看到的细节越少越模糊。因此，当显示设备显示的图像上远处对象模糊、近处对象清晰时，人感受到的虚拟环境比较符合真实情况。而且，显示设备显示的图像流中远处对象的清晰度有高有低，不是一直模糊状态，所以可以保证人脑获取到远处对象的足够细节。For example, the third depth of field is smaller than the first depth of field. That is, the definition of the first object with a larger depth of field varies from high to low, while the definition of the second object with a smaller depth of field remains unchanged. Generally speaking, when the human eye looks at nearby objects, the more details it sees, the clearer it is, and when looking at distant objects, the less details it sees, the more blurred. Therefore, when the distant objects are blurred and the nearby objects are clear on the image displayed by the display device, the virtual environment felt by people is more in line with the real situation. Moreover, the clarity of distant objects in the image stream displayed by the display device varies from high to low, and is not always in a blurred state, so it can ensure that the human brain can obtain sufficient details of distant objects.
再比如，所述第三景深与所述用户注视点所在景深之间的距离小于所述第一景深与所述用户注视点所在景深之间的距离。即，远离用户注视点的第一对象的清晰度有高有低，靠近用户注视点的第二对象的清晰度不变。可以理解的是，在现实环境中，人眼注视某个物体时，眼睛看到的该物体比较清晰，距离该物体较远的其它物体比较模糊。因此，显示设备显示的图像上远离用户注视点的对象模糊，靠近用户注视点的对象清晰，这样人眼所看到的虚拟环境比较符合真实情况。而且，显示设备显示的图像流中远离用户注视点的对象的清晰度有高有低，不是一直模糊状态，所以可以保证人脑获取到远离用户注视点的对象的足够细节。For another example, the distance between the third depth of field and the depth of field where the user gaze point is smaller than the distance between the first depth of field and the depth of field where the user gaze point is located. That is, the clarity of the first object far from the user's gaze point varies, and the clarity of the second object close to the user's gaze point remains unchanged. It can be understood that, in a real environment, when human eyes look at an object, the object seen by the eyes is relatively clear, and other objects far away from the object are relatively blurred. Therefore, objects far away from the user's gaze point on the image displayed by the display device are blurred, and objects close to the user's gaze point are clear, so that the virtual environment seen by human eyes is more in line with the real situation. Moreover, in the image stream displayed by the display device, the definition of objects far away from the user's gaze point varies from high to low, and is not always in a blurred state, so it can ensure that the human brain can obtain sufficient details of objects far away from the user's gaze point.
一种可实施方式为，所述第j帧图像的显示时间与所述第i帧图像的显示时间之间的时间间隔小于或等于所述用户的视觉停留时长；和/或，所述第k帧图像的显示时间与所述第j帧图像的显示时间之间的时间间隔小于或等于所述时间停留时长。需要说明的是，第i帧图像上第一对象的清晰度低，第j帧图像上第一对象的清晰度高，为了保证人脑获取到第一对象的细节，可以在用户的视觉停留时长内显示第i帧图像和第j帧图像，这样的话，人脑会融合第i帧图像和第j帧图像，保证人脑摄取到第一对象的足够细节。A possible implementation manner is that the time interval between the display time of the j-th frame of image and the display time of the i-th frame of image is less than or equal to the user's visual dwell time; and/or, the k-th The time interval between the display time of the frame image and the display time of the jth frame image is less than or equal to the time dwell time. It should be noted that the definition of the first object on the i-th frame of image is low, and the definition of the first object on the j-th frame of image is high. Display the i-th frame image and the j-th frame image, in this case, the human brain will fuse the i-th frame image and the j-th frame image to ensure that the human brain can capture enough details of the first object.
另一种可实施方式为，j＝i+n，其中，n大于或等于1，或者，n随着所述用户的时间停留时长与所述显示设备的图像刷新帧率的变化而变化；和/或，k＝j+m，其中，m大于或等于1，或者，m随着所述用户的时间停留时长与所述显示设备的图像刷新帧率的变化而变化。Another possible implementation manner is, j=i+n, wherein, n is greater than or equal to 1, or, n changes with the change of the user's time stay and the image refresh frame rate of the display device; and /or, k=j+m, wherein, m is greater than or equal to 1, or, m changes with the change of the user's time stay and the image refresh frame rate of the display device.
示例性的，j＝i+1，k＝j+1，即图像流中上一帧中第一对象模糊，下一帧中第一对象清晰，再下一帧中第一对象模糊。或者，n、m可以根据用户视觉停留时间、图像刷新帧率确定。假设用户视觉停留时间为T，图像刷新帧率为P。那么，在T时间内，可以显示T/P帧图像，那么n小于或等于T/P，m小于或等于T/P。因此，在用户的视觉停留时长内显示第i帧图像和第j帧图像，这样的话，人脑会融合第i帧图像和第j帧图像，保证人脑摄取到第一对象的足够细节。Exemplarily, j=i+1, k=j+1, that is, the first object in the previous frame in the image stream is blurred, the first object in the next frame is clear, and the first object in the next frame is blurred. Alternatively, n and m may be determined according to the user's visual dwell time and image refresh frame rate. Assume that the user's visual dwell time is T, and the image refresh frame rate is P. Then, within T time, T/P frame images can be displayed, then n is less than or equal to T/P, and m is less than or equal to T/P. Therefore, the i-th frame image and the j-th frame image are displayed within the user's visual dwell time. In this way, the human brain will fuse the i-th frame image and the j-th frame image to ensure that the human brain can capture enough details of the first object.
在一种可能的设计中，所述显示设备包括第一显示屏和第二显示屏，所述第一显示屏用于向用户左眼呈现图像，所述第二显示屏用于向用户右眼呈现图像；所述第一显示屏和所述第二显示屏上显示的图像同步。其中，第一显示屏和第二显示屏上显示的图像同步可以理解为第一显示屏显示第i帧图像，那么第二显示屏也显示第i帧图像，保证两个显示屏显示的图像顺序一致。In a possible design, the display device includes a first display screen and a second display screen, the first display screen is used to present images to the user's left eye, and the second display screen is used to display images to the user's right eye presenting an image; the images displayed on the first display screen and the second display screen are synchronized. Wherein, the synchronization of the images displayed on the first display screen and the second display screen can be understood as that the first display screen displays the i-th frame image, then the second display screen also displays the i-th frame image, ensuring the order of the images displayed on the two display screens unanimous.
一种方式为，第一显示屏和第二显示屏分别显示所述N帧图像；可以理解为，第一显示屏和第二显示屏显示同一组图像流，该图像流中第一对象的清晰度有高有低。由于两个显示屏显示的图像流相同，而且显示的图像同步，比如，同时显示第i帧，同时显示第j 帧等等。所以，第一显示屏上第一对象模糊时，第二显示屏上第一对象也是模糊的。也就是说，第一显示屏显示的图像流和第二显示屏显示的图像流中第一对象的清晰度变化趋势相同，比如都是“模糊-清晰-模糊-清晰”的变化趋势。One way is that the first display screen and the second display screen respectively display the N frames of images; it can be understood that the first display screen and the second display screen display the same group of image streams, and the first object in the image streams Clarity is high and low. Since the image streams displayed on the two display screens are the same, and the displayed images are synchronized, for example, the i-th frame and the j-th frame are displayed at the same time, and so on. Therefore, when the first object is blurred on the first display screen, the first object is also blurred on the second display screen. That is to say, the change trend of the clarity of the first object in the image stream displayed on the first display screen and the image stream displayed on the second display screen is the same, for example, both have a change trend of "blurry-clear-blurry-clear".
第二种方式，所述第一显示屏显示所述N帧图像；所述第二显示屏显示另外N帧图像；所述另外N帧图像与所述N帧图像的图像内容相同；所述另外N帧图像中第i帧图像上，处于第一景深处的所述第一对象的清晰度为第四清晰度；所述另外N帧图像中第j帧图像上，处于所述第一景深处的所述第一对象的清晰度为第五清晰度；所述另外N帧图像中第k帧图像上，处于所述第一景深处的所述第一对象的清晰度为第六清晰度；其中，所述第四清晰度大于所述第五清晰度，所述第四清晰度小于所述第六清晰度。由于两个显示屏显示的图像同步，比如，同时显示第i帧，同时显示第j帧等等。所以，第一显示屏上第一对象模糊时，第二显示屏上第一对象是清晰的。也就是说，第一显示屏显示的图像流和第二显示屏显示的图像流中第一对象的清晰度变化趋势可以相反。比如，第一显示屏显示的图像流中第一对象的清晰度呈“模糊-清楚-模糊-清楚”的交替变化，第二显示屏显示的图像流中远处对象呈“清楚-模糊-清楚-模糊”的交替变化。In the second way, the first display screen displays the N frames of images; the second display screen displays another N frames of images; the other N frames of images have the same image content as the N frames of images; the other On the i-th image in the N frames of images, the definition of the first object in the first depth of field is the fourth definition; in the j-th image in the other N frames of images, in the first depth of field The definition of the first object is the fifth definition; the definition of the first object in the first depth of field on the k-th frame image in the other N frames of images is the sixth definition; Wherein, the fourth definition is greater than the fifth definition, and the fourth definition is smaller than the sixth definition. Since the images displayed on the two display screens are synchronized, for example, the i-th frame and the j-th frame are displayed at the same time, and so on. Therefore, when the first object is blurred on the first display screen, the first object is clear on the second display screen. That is to say, the change trend of the definition of the first object in the image stream displayed on the first display screen and the image stream displayed on the second display screen may be opposite. For example, the sharpness of the first object in the image stream displayed on the first display screen changes alternately from "blur-clear-fuzzy-clear", and the distant objects in the image stream displayed on the second display screen appear "clear-fuzzy-clear-clear-clear". Fuzzy" alternately.
示例性的，所述第四清晰度大于所述第一清晰度；和/或，所述第五清晰度小于所述第二清晰度；和/或，所述第六清晰度大于所述第三清晰度。Exemplarily, the fourth definition is greater than the first definition; and/or, the fifth definition is smaller than the second definition; and/or, the sixth definition is greater than the first Three clarity.
比如，当左、右眼对应的显示屏同步显示图像(比如，第i帧图像)时，左眼图像上第一对象模糊，右眼图像上第一对象清晰。再比如，当左、右眼对应的显示屏同步显示图像(比如，第j帧图像)时，左眼图像上第一对象清晰，右眼图像上第一对象模糊。这种方式，一定程度上可以缓解人脑疲劳，而且人脑中将左眼图像和右眼图像叠加得到的图像上第一对象的不至于太过模糊，避免丢失该对象的太多细节。For example, when the display screens corresponding to the left and right eyes synchronously display images (for example, the i-th frame image), the first object on the left-eye image is blurred, and the first object on the right-eye image is clear. For another example, when the display screens corresponding to the left and right eyes synchronously display images (for example, the j-th frame image), the first object on the left-eye image is clear, and the first object on the right-eye image is blurred. This method can alleviate the fatigue of the human brain to a certain extent, and the first object on the image obtained by superimposing the left-eye image and the right-eye image in the human brain will not be too blurred, so as to avoid losing too many details of the object.
本申请实施例提供的显示方法可以适用于各种应用场景。比如游戏应用(如，VR游戏应用)、模拟驾驶(如，VR驾驶)、模拟教学(如，VR教学)等等。以下以VR游戏和VR驾驶为例。The display method provided by the embodiment of the present application may be applicable to various application scenarios. For example, game applications (eg, VR game applications), simulated driving (eg, VR driving), simulated teaching (eg, VR teaching), and the like. Take VR games and VR driving as examples below.
第一种应用场景，所述N帧图像是与游戏相关的图像；所述游戏可以是VR游戏。比如，所述N帧图像是VR游戏应用产生的图像。所述第二景深包括：游戏场景中，所述用户对应的游戏角色所在景深，或者，所述用户对应的游戏角色上身体部位(比如手臂)所在景深，或者，所述用户对应的游戏角色当前所持游戏装备(比如***)所在景深；和/或，所述用户注视点所在景深包括：游戏场景中，游戏对方对应的游戏角色所在景深，或者，建筑物所在景深，或者，所述用户对应的游戏角色上身体部位所在景深，或者，所述用户对应的游戏角色当前所持游戏装备所在景深。因此，游戏场景中，景深大于第二景深的第一对象或远离用户注视点的第一对象的清晰度可以有高有低，不仅可以缓解疲劳感，又能保证人脑中摄取到第一对象的足够细节，保证游戏(如,VR游戏)体验。In the first application scenario, the N frames of images are images related to games; the games may be VR games. For example, the N frames of images are images generated by a VR game application. The second depth of field includes: in the game scene, the depth of field where the game character corresponding to the user is located, or the depth of field where the body part (such as an arm) of the game character corresponding to the user is located, or the current depth of field of the game character corresponding to the user. The depth of field where the game equipment held (such as guns) is located; and/or, the depth of field where the user’s gaze point is located includes: in the game scene, the depth of field where the game character corresponding to the game opponent is located, or the depth of field where the building is located, or, the depth of field where the user’s corresponding The depth of field of the body parts on the game character, or the depth of field of the game equipment currently held by the game character corresponding to the user. Therefore, in the game scene, the definition of the first object whose depth of field is greater than the second depth of field or the first object far away from the user's gaze point can be high or low, which can not only relieve fatigue, but also ensure that the first object's image is captured in the human brain. Enough details to ensure the experience of games (eg, VR games).
第二种应用场景，所述N帧图像是与车辆驾驶相关的图像；比如所述N帧图像是VR驾驶应用产生的图像。所述第二景深包括：车辆驾驶场景中，所述用户当前驾驶车辆所在景深，或者，所述用户当前驾驶车辆上方向盘所在景深，或者，所述用户当前驾驶车辆上挡风玻璃所在景深；和/或，所述用户注视点所在景深，包括：车辆驾驶场景中，行驶道路上其它用户驾驶车辆(比如行驶在用户当前驾驶车辆前方的车辆)，或者，行驶道路的路边设置对象(比如，道路边上的树、指示牌等等)。因此，车辆驾驶场景中，景深大于第二景深的第一对象或远离用户注视点的第一对象的清晰度可以有高有低，不仅可以缓解疲 劳感，又能保证人脑中摄取到第一对象的足够细节，保证车辆驾驶(如，VR驾驶)体验。In the second application scenario, the N frames of images are images related to vehicle driving; for example, the N frames of images are images generated by a VR driving application. The second depth of field includes: in the vehicle driving scene, the depth of field of the vehicle currently driven by the user, or the depth of field of the steering wheel of the vehicle currently driven by the user, or the depth of field of the windshield of the vehicle currently driven by the user; and /or, the depth of field where the user's gaze point is located includes: in the vehicle driving scene, other users driving vehicles on the driving road (such as a vehicle driving in front of the user's current driving vehicle), or objects set on the roadside of the driving road (such as, trees along the road, signs, etc.). Therefore, in the vehicle driving scene, the definition of the first object whose depth of field is greater than the second depth of field or the first object far away from the user's gaze point can be high or low, which can not only relieve fatigue, but also ensure that the first object is captured in the human brain enough details to ensure the vehicle driving (eg, VR driving) experience.
在一种可能的设计中，所述第i帧图像是对第i帧原图上所述第一对象做模糊化处理后的图像；所述第j帧图像是第j帧原图或对第j帧原图上所述第一对象作清晰化处理后的图像；所述第k帧图像是对第k帧原图上所述第一对象做模糊化处理后的图像；其中，所述第i帧原图、第j帧原图、第k帧原图上所有对象的清晰度相同；其中，所述第j帧图像是对第j帧原图上所述第一对象作清晰化处理后的图像，包括：所述第j帧图像是所述第i帧图像和所述第j帧原图融合得到的图像；或者，所述第j帧图像是对所述第i帧图像作模糊化处理时被丢失的图像信息和所述第j帧原图融合得到的图像。因此，第j帧图像上第一对象的清晰度高，保证人脑能够摄取到第一对象的足够细节，不影响用户体验。In a possible design, the image in the i-th frame is an image after blurring the first object on the original image in the i-th frame; the image in the j-th frame is the original image in the j-th frame or The image of the first object on the j-frame original image after clearing processing; the k-th frame image is an image after blurring the first object on the k-th frame of the original image; wherein, the first The definition of all objects on the i-frame original image, the j-th frame original image, and the k-th frame original image is the same; wherein, the j-th frame image is after the definition processing of the first object on the j-th frame original image image, including: the image of the jth frame is an image obtained by fusing the image of the ith frame and the original image of the jth frame; or, the image of the jth frame is blurred on the image of the ith frame An image obtained by fusing the image information lost during processing with the original image of the jth frame. Therefore, the definition of the first object on the jth frame image is high, ensuring that the human brain can capture enough details of the first object without affecting user experience.
在一种可能的设计中，所述第j帧图像是所述第i帧图像和所述第j帧原图融合得到的图像，包括：所述第j帧图像上所述第一对象所在区域内的图像块是第一图像块和第二图像块融合得到的；其中，所述第一图像块是所述第i帧图像上所述第一对象所在区域内的图像块，所述第二图像块是所述第j帧原图上所述第一对象所在区域内的图像块。通过这种方式，在将第i帧图像和第j帧原图融合时，可以只将第i帧图像上第一对象所在的区域内的图像块与第j帧原图上第一对象所在区域内的图像块融合，不需要第i帧图像和第j帧原图整张图像融合，提升效率。In a possible design, the j-th frame image is an image obtained by fusing the i-th frame image and the j-th frame original image, including: the area where the first object is located on the j-th frame image The image block in is obtained by fusion of the first image block and the second image block; wherein, the first image block is an image block in the area where the first object is located on the ith frame image, and the second image block is The image block is an image block in the area where the first object is located on the original image of the jth frame. In this way, when fusing the i-th frame image with the j-th frame original image, only the image block in the area where the first object is located on the i-th frame image can be combined with the j-th frame original image where the first object is located. The fusion of the image blocks in the frame does not require the entire image fusion of the i-th frame image and the j-th frame original image to improve efficiency.
第二方面，还提供一种电子设备，包括：In a second aspect, an electronic device is also provided, including:
处理器，存储器，以及，一个或多个程序；processor, memory, and, one or more programs;
其中，所述一个或多个程序被存储在所述存储器中，所述一个或多个程序包括指令，当所述指令被所述处理器执行时，使得所述电子设备执行如上述第一方面提供的方法步骤。Wherein, the one or more programs are stored in the memory, the one or more programs include instructions, and when the instructions are executed by the processor, the electronic device performs the above-mentioned first aspect The method steps provided.
第三方面，提供一种计算机可读存储介质，所述计算机可读存储介质用于存储计算机程序，当所述计算机程序在计算机上运行时，使得所述计算机执行如上述第一方面提供的方法。In the third aspect, a computer-readable storage medium is provided, the computer-readable storage medium is used to store a computer program, and when the computer program is run on a computer, the computer executes the method as provided in the above-mentioned first aspect .
第四方面，提供一种计算机程序产品，包括计算机程序，当所述计算机程序在计算机上运行时，使得所述计算机执行如上述第一方面提供的方法。In a fourth aspect, a computer program product is provided, including a computer program, and when the computer program is run on a computer, the computer is made to execute the method provided in the first aspect above.
第五方面，提供一种电子设备上的图形用户界面，所述电子设备具有显示屏、存储器、以及处理器，所述处理器用于执行存储在所述存储器中的一个或多个计算机程序，所述图形用户界面包括所述电子设备执行上述第一方面提供的方法时显示的图形用户界面。In a fifth aspect, a graphical user interface on an electronic device is provided, the electronic device has a display screen, a memory, and a processor, the processor is used to execute one or more computer programs stored in the memory, the The graphical user interface includes a graphical user interface displayed when the electronic device executes the method provided in the first aspect above.
第六方面，本申请实施例还提供一种芯片***，所述芯片***与电子设备中的存储器耦合，用于调用存储器中存储的计算机程序并执行本申请实施例第一方面的技术方案，本申请实施例中“耦合”是指两个部件彼此直接或间接地结合。In the sixth aspect, the embodiment of the present application further provides a chip system, the chip system is coupled with the memory in the electronic device, and is used to call the computer program stored in the memory and execute the technical solution of the first aspect of the embodiment of the present application. "Coupling" in the embodiments of the application means that two components are directly or indirectly combined with each other.
上述第二方面至第六方面的有益效果，参见第一方面的有益效果，不重复赘述。For the above beneficial effects of the second aspect to the sixth aspect, refer to the beneficial effects of the first aspect, which will not be repeated.
附图说明Description of drawings
图1为本申请一实施例提供的视觉辐辏调节冲突的一种示意图；FIG. 1 is a schematic diagram of vergence-accommodation conflict provided by an embodiment of the present application;
图2为本申请一实施例提供的VR***的示意图；FIG. 2 is a schematic diagram of a VR system provided by an embodiment of the present application;
图3A为本申请一实施例提供的人眼会聚的原理示意图；FIG. 3A is a schematic diagram of the principle of human eye convergence provided by an embodiment of the present application;
图3B为本申请一实施例提供的人眼结构的示意图；Fig. 3B is a schematic diagram of the human eye structure provided by an embodiment of the present application;
图4为本申请一实施例提供的人眼睫状肌对晶状体调节的示意图；Fig. 4 is a schematic diagram of the adjustment of the human eye ciliary muscle to the lens provided by an embodiment of the present application;
图5为本申请一实施例提供的VR眼镜的一种示意图；Fig. 5 is a schematic diagram of VR glasses provided by an embodiment of the present application;
图6A至图6C为本申请一实施例提供的VR眼镜显示的图像对应的虚像面的示意图；6A to 6C are schematic diagrams of virtual image planes corresponding to images displayed by VR glasses provided by an embodiment of the present application;
图7A至图7B为本申请一实施例提供的第一种应用场景的示意图；7A to 7B are schematic diagrams of a first application scenario provided by an embodiment of the present application;
图8A至图8B为本申请一实施例提供的第二种应用场景的示意图；8A to 8B are schematic diagrams of a second application scenario provided by an embodiment of the present application;
图9为本申请一实施例提供的VR穿戴设备的结构示意图；FIG. 9 is a schematic structural diagram of a VR wearable device provided by an embodiment of the present application;
图10为本申请一实施例提供的VR虚拟环境中用户注视点清晰，非用户注视点模糊的示意图；FIG. 10 is a schematic diagram of clear gaze points of users and blurred gaze points of non-users in a VR virtual environment provided by an embodiment of the present application;
图11为本申请一实施例提供的图像生成原理的一种流程示意图；FIG. 11 is a schematic flow chart of an image generation principle provided by an embodiment of the present application;
图12至图13为本申请一实施例提供的图像流的一种示意图；12 to 13 are schematic diagrams of image streams provided by an embodiment of the present application;
图14为本申请一实施例提供的用户人脑获取图像的示意图；Fig. 14 is a schematic diagram of an image acquired by a user's human brain provided by an embodiment of the present application;
图15为本申请一实施例提供的图像流生成过程的示意图；FIG. 15 is a schematic diagram of an image stream generation process provided by an embodiment of the present application;
图16为本申请一实施例提供的VR虚拟环境中远景对象模糊、近景对象清晰的示意图；FIG. 16 is a schematic diagram of fuzzy objects in the foreground and clear objects in the foreground in the VR virtual environment provided by an embodiment of the present application;
图17为本申请一实施例提供的图像生成原理的另一种流程示意图；Fig. 17 is another schematic flowchart of the principle of image generation provided by an embodiment of the present application;
图18至图19为本申请一实施例提供的图像流的另一种示意图；18 to 19 are another schematic diagram of an image stream provided by an embodiment of the present application;
图20至图23为本申请一实施例提供的显示设备上左眼显示屏和右眼显示屏显示图像流的示意图；20 to 23 are schematic diagrams of image streams displayed on the left-eye display screen and the right-eye display screen on the display device provided by an embodiment of the present application;
图24为本申请一实施例提供的电子设备的结构示意图。FIG. 24 is a schematic structural diagram of an electronic device provided by an embodiment of the present application.
具体实施方式detailed description
以下，对本申请实施例中的部分用语进行解释说明，以便于本领域技术人员理解。In the following, some terms used in the embodiments of the present application are explained, so as to facilitate the understanding of those skilled in the art.
(1)本申请实施例涉及的至少一个，包括一个或者多个；其中，多个是指大于或者等于两个。另外，需要理解的是，在本申请的描述中，“第一”、“第二”等词汇，仅用于区分描述的目的，而不能理解为明示或暗示相对重要性，也不能理解为明示或暗示顺序。比如，第一对象和第二对象并不代表二者的重要程度，或者代表二者的顺序，是为了区分对象。(1) At least one of the embodiments of the present application involves one or more; wherein, a plurality means greater than or equal to two. In addition, it should be understood that in the description of this application, words such as "first" and "second" are only used for the purpose of distinguishing descriptions, and cannot be understood as express or implied relative importance, nor can they be understood as express or imply order. For example, the first object and the second object do not represent the importance of the two, or represent the order of the two, but to distinguish the objects.
在本申请实施例中，“和/或”，是一种描述关联对象的关联关系，表示可以存在三种关系，例如，A和/或B，可以表示：单独存在A，同时存在A和B，单独存在B这三种情况。另外，本文中字符“/”，一般表示前后关联对象是一种“或”的关系。In the embodiment of this application, "and/or" is an association relationship describing associated objects, indicating that there may be three relationships, for example, A and/or B, which may mean: A exists alone, and A and B exist at the same time , there are three cases of B alone. In addition, the character "/" in this article generally indicates that the contextual objects are an "or" relationship.
(2)虚拟现实(Virtual Reality，VR)技术是借助计算机及传感器技术创造的一种人机交互手段。VR技术综合了计算机图形技术、计算机仿真技术、传感器技术、显示技术等多种科学技术，可以创建虚拟环境。虚拟环境包括由计算机生成的、并实时动态播放的三维立体逼真图像为用户带来视觉感知；而且，除了计算机图形技术所生成的视觉感知外，还有听觉、触觉、力觉、运动等感知，甚至还包括嗅觉和味觉等，也称为多感知；此外，还可以检测用户的头部转动，眼睛、手势、或其他人体行为动作，由计算机来处理与用户的动作相适应的数据，并对用户的动作实时响应，并分别反馈到用户的五官，进而形式虚拟环境。示例性的，用户佩戴VR穿戴设备可以看到VR游戏界面，通过手势、手柄等操作，可以与VR游戏界面交互，仿佛身处游戏中。(2) Virtual Reality (VR) technology is a means of human-computer interaction created with the help of computer and sensor technology. VR technology integrates computer graphics technology, computer simulation technology, sensor technology, display technology and other science and technology to create a virtual environment. The virtual environment includes three-dimensional realistic images generated by computers and dynamically played in real time to bring visual perception to users; moreover, in addition to the visual perception generated by computer graphics technology, there are also perceptions such as hearing, touch, force, and movement. It even includes the sense of smell and taste, also known as multi-sensing; in addition, it can also detect the user's head rotation, eyes, gestures, or other human behaviors, and the computer will process the data that is suitable for the user's actions, and The user's actions respond in real time and are fed back to the user's facial features respectively, thereby forming a virtual environment. Exemplarily, the user can see the VR game interface by wearing the VR wearable device, and can interact with the VR game interface through gestures, handles, and other operations, as if in a game.
(3)增强现实(Augmented Reality，AR)技术是指将计算机生成的虚拟对象叠加到真实世界的场景之上，从而实现对真实世界的增强。也就是说，AR技术中需要采集真实世界的场景，然后在真实世界上增加虚拟环境。(3) Augmented Reality (AR) technology refers to superimposing computer-generated virtual objects on real-world scenes to enhance the real world. In other words, AR technology needs to collect real-world scenes, and then add a virtual environment to the real world.
因此，VR技术与AR技术的区别在于，VR技术创建的是完全的虚拟环境，用户看到 的全部是虚拟对象；而AR技术是在真实世界上叠加了虚拟对象，即既包括真实世界中对象也包括虚拟对象。比如，用户佩戴透明眼镜，通过该眼镜可以看到周围的真实环境，而且该眼镜上还可以显示虚拟对象，这样，用户既可以看到真实对象也可以看到虚拟对象。Therefore, the difference between VR technology and AR technology is that VR technology creates a complete virtual environment, and all users see are virtual objects; while AR technology superimposes virtual objects on the real world, that is, it includes objects in the real world. Also includes dummy objects. For example, the user wears transparent glasses, through which the real environment around can be seen, and virtual objects can also be displayed on the glasses, so that the user can see both real objects and virtual objects.
(4)混合现实技术(Mixed Reality，MR)，是通过在虚拟环境中引入现实场景信息(或称为真实场景信息)，将虚拟环境、现实世界和用户之间搭起一个交互反馈信息的桥梁，从而增强用户体验的真实感。具体来说，把现实对象虚拟化，(比如，使用摄像头来扫描现实对象进行三维重建，生成虚拟对象)，经过虚拟化的真实对象引入到虚拟环境中，这样，用户在虚拟环境中可以看到真实对象。(4) Mixed reality technology (Mixed Reality, MR) is to build a bridge of interactive feedback information between the virtual environment, the real world and users by introducing real scene information (or called real scene information) into the virtual environment. , thereby enhancing the realism of the user experience. Specifically, the real object is virtualized (for example, using a camera to scan the real object for 3D reconstruction to generate a virtual object), and the virtualized real object is introduced into the virtual environment, so that the user can see in the virtual environment real object.
需要说明的是，本申请实施例提供的技术方案可以适用于VR场景、AR场景或MR场景中。当然，除了VR、AR和MR之外还可以适用于其它场景。比如，裸眼3D场景(裸眼3D显示屏、裸眼3D投影等)、影院(如3D电影)、电子设备中的VR软件等，总之，可以适用于任何需要显示三维图像的场景。It should be noted that the technical solution provided by the embodiment of the present application may be applicable to a VR scene, an AR scene or an MR scene. Of course, in addition to VR, AR and MR, it can also be applied to other scenarios. For example, glasses-free 3D scenes (glasses-free 3D display, glasses-free 3D projection, etc.), theaters (such as 3D movies), VR software in electronic equipment, etc., in short, can be applied to any scene that needs to display three-dimensional images.
为了方便描述，下文主要以VR场景为例进行介绍。For the convenience of description, the following mainly introduces the VR scene as an example.
示例性的，请参见图2，为本申请实施例VR***的示意图。VR***中包括VR穿戴设备100，以及图像生成设备200。其中，图像生成设备200包括主机(例如VR主机)或服务器(例如VR服务器)。VR穿戴设备100与VR主机或VR服务器连接(有线连接或无线连接)。VR主机或VR服务器可以是具有较大计算能力的设备。例如，VR主机可以是手机、平板电脑、笔记本电脑等设备，VR服务器可以是云服务器等。VR主机或VR服务器负责生成图像等，然后将图像发送给VR穿戴设备100显示，用户佩戴VR穿戴设备100可以看到图像。示例性的，VR穿戴设备100可以是头戴式设备(Head Mounted Display，HMD)，比如眼镜、头盔等。可选的，图2中VR***中也可以不包括图像生成设备200。比如，VR穿戴设备100本地具有图像生成能力，无需从图像生成设备200(VR主机或VR服务器)获取图像进行显示。总之，VR穿戴设备100可以显示三维图像，由于三维图像上不同对象的景深(参见下文介绍)不同，所以通过三维图像可以为用户展示虚拟环境。For example, please refer to FIG. 2 , which is a schematic diagram of a VR system according to an embodiment of the present application. The VR system includes a VR wearable device 100 and an image generating device 200 . Wherein, the image generating device 200 includes a host (such as a VR host) or a server (such as a VR server). The VR wearable device 100 is connected (wired connection or wireless connection) with a VR host or a VR server. The VR host or VR server may be a device with relatively large computing power. For example, the VR host can be a device such as a mobile phone, a tablet computer, or a notebook computer, and the VR server can be a cloud server, etc. The VR host or VR server is responsible for generating images, etc., and then sends the images to the VR wearable device 100 for display, and the user wearing the VR wearable device 100 can see the images. Exemplarily, the VR wearable device 100 may be a head mounted device (Head Mounted Display, HMD), such as glasses, a helmet, and the like. Optionally, the VR system in FIG. 2 may not include the image generating device 200 . For example, the VR wearable device 100 has image generation capabilities locally, and does not need to acquire images from the image generation device 200 (VR host or VR server) for display. In a word, the VR wearable device 100 can display a three-dimensional image. Since different objects have different depths of field (refer to the introduction below) on the three-dimensional image, the virtual environment can be shown to the user through the three-dimensional image.
(5)景深(Depth of Field，简称DOF)(5) Depth of Field (DOF for short)
三维图像包括不同图像深度的对象。比如，VR穿戴设备显示三维图像，用户佩戴VR穿戴设备看到的是三维场景(即虚拟环境)，该三维场景中不同对象到用户眼睛的距离不同，呈现立体感。因此，图像深度可以理解为三维图像上对象与用户眼睛之间的距离，图像深度越大，视觉上距离用户眼睛越远，看上去像是远景，图像深度越小，视觉上距离用户眼睛越近，看上去像是近景。图像深度还可以称为“景深”。A three-dimensional image includes objects at different image depths. For example, a VR wearable device displays a three-dimensional image, and the user wearing the VR wearable device sees a three-dimensional scene (that is, a virtual environment). Different objects in the three-dimensional scene have different distances from the user's eyes, presenting a three-dimensional effect. Therefore, the image depth can be understood as the distance between the object on the 3D image and the user's eyes. The larger the image depth, the farther away from the user's eyes visually, which looks like a distant view. The smaller the image depth, the closer to the user's eyes visually. , which looks like a close-up. Image depth may also be referred to as "depth of field".
为了能够对VR穿戴设备向用户展示虚拟环境进行清楚的说明，以下首先对人眼视觉产生机制进行简单说明。In order to be able to clearly explain the virtual environment displayed to the user by the VR wearable device, the following first briefly describes the mechanism of human vision generation.
可以理解的是，在实际场景中，用户在观看物体时，人眼可以通过获取实际场景中的光信号，并对该光信号在大脑中进行处理，实现视觉感受。其中，该实际场景中的光信号可以包括来自不同物体的反射光，和/或光源直接发出的光信号。由于实际场景的光信号中可以携带有实际场景中的各个物体的相关信息(如大小，位置，颜色等)，因此，大脑可以通过对光信号进行处理，就能够获取实际场景中的物体的信息，即获取视觉感受。It can be understood that, in an actual scene, when a user views an object, the human eye can obtain a light signal in the actual scene and process the light signal in the brain to realize visual experience. Wherein, the optical signal in the actual scene may include reflected light from different objects, and/or an optical signal directly emitted by a light source. Since the light signal of the actual scene can carry the relevant information (such as size, position, color, etc.) of each object in the actual scene, the brain can obtain the information of the object in the actual scene by processing the light signal , that is, to obtain visual experience.
需要说明的是，由于左眼和右眼在观看同一物体时，视线角度稍有不同。因此左眼和右眼看到的景象实际上是有差异的。比如，左眼可以获取与左眼视线方向垂直的人眼焦点 所在平面的二维图像(以下简称为左眼图像)的光信号。同理，右眼可以获取与右眼视线方向垂直的人眼焦点所在平面的二维图像(以下简称为右眼图像)的光信号。左眼图像与右眼图像稍有不同。大脑可以通过对左眼图像和右眼图像的光信号进行处理，从而获取当前场景中不同物体的相关信息。It should be noted that when the left eye and the right eye watch the same object, the viewing angles are slightly different. Therefore, the scene seen by the left eye and the right eye is actually different. For example, the left eye can obtain the light signal of the two-dimensional image (hereinafter referred to as the left eye image) on the plane where the focus of the human eye is perpendicular to the line of sight of the left eye. Similarly, the right eye can obtain the light signal of the two-dimensional image (hereinafter referred to as the right eye image) on the plane where the focus of the human eye is perpendicular to the line of sight of the right eye. The image for the left eye is slightly different from the image for the right eye. The brain can obtain information about different objects in the current scene by processing the light signals of the left-eye image and the right-eye image.
此外，用户还可以通过获取实际场景中不同物体的深度，获取立体视觉感受。立体视觉感受也可称为双目立体视觉。In addition, users can also obtain stereoscopic vision experience by obtaining the depth of different objects in the actual scene. Stereo vision experience can also be called binocular stereo vision.
示例性的，用户在观看实际场景中的物体时，会经历汇聚(vergence)和变焦(accommodation)两个过程。其中，汇聚也可以称为会聚，本申请对此名称不限定。Exemplarily, when viewing an object in an actual scene, the user will experience two processes of convergence (vergence) and zoom (accommodation). Wherein, converging may also be referred to as converging, which is not limited in this application.
一、会聚1. Convergence
会聚可以理解为调节人眼视线使其指向物体。示例性的，如图3A所示，双眼在观察实际场景中的物体(图3A中以物体是三角形为例)时，可以通过控制人眼附近的肌肉，使得左眼和右眼的视线分别向物体转动(指向物体)。Convergence can be understood as adjusting the line of sight of the human eye so that it points to the object. Exemplarily, as shown in FIG. 3A, when the two eyes are observing an object in an actual scene (the object is a triangle in FIG. 3A as an example), the sight lines of the left eye and the right eye can be respectively directed to the The object turns (points to the object).
辐辏角度(Vergence angle)和辐辏深度(Vergence distance)Vergence angle (Vergence angle) and Vergence depth (Vergence distance)
继续如图3A所示，双眼在观察物体时左眼视线和右眼视线所形成的夹角称为辐辏角度θ。大脑可以通过获取双眼的辐辏角度θ，来判断该物体的深度，即辐辏深度。可以理解的是，被观察物体距离人眼越近，辐辏角度θ就越大，辐辏深度越小。对应的，被观察物体距离人眼越远，辐辏角度θ就越小，辐辏深度越大。Continuing as shown in FIG. 3A , the angle formed by the line of sight of the left eye and the line of sight of the right eye when the two eyes observe an object is called the convergence angle θ. The brain can judge the depth of the object by obtaining the convergence angle θ of the eyes, that is, the convergence depth. It can be understood that the closer the observed object is to the human eye, the larger the convergence angle θ and the smaller the convergence depth. Correspondingly, the farther the observed object is from the human eye, the smaller the convergence angle θ and the greater the convergence depth.
二、变焦2. Zoom
变焦可以理解为人眼在观察物体时，调节人眼到正确的焦距。一般，大脑通过睫状肌控制晶状体调节到正确的焦距。示例性的，结合图3B以及图4进行说明。图3B为人眼的组成示意图。如图3B所示，人眼中可以包括晶状体和睫状肌，以及位于眼底的视网膜。其中，晶状体可以起到变焦透镜的作用，对射入人眼的光线进行汇聚处理。以便将入射光线汇聚到人眼眼底的视网膜上，使得实际场景中的景物能够在视网膜上成清晰的像。睫状肌可以用于调节晶状体的形态，比如睫状肌可以通过收缩或放松，调节晶状体的屈光度，达到调整晶状体焦距的效果。从而使得实际场景中不同距离的物体，都可以通过晶状体清晰地在视网膜上的成像。作为一种示例，参考图4，为人眼在观察不同距离的物体时，睫状肌对晶状体的调节示意。如图4中的(a)所示，在人眼观察较远处的物体时，以该物体为非光源为例。来自该物体表面的反射光线可以接***行光。此时，睫状肌可以将晶状体的状态控制在如图4中(a)所示的状态，如睫状肌放松，控制晶状体呈扁平状，屈光度较小，从而使得平行入射光可以通过晶状体之后汇聚在眼底的视网膜上。而在人眼观察较近的物体时，结合图4中的(b)，以该物体为非光源为例。来自该物体表面的反射光可以按照如图4中的(b)所示的光路入射人眼。此时睫状肌可以将晶状体的状态处于图4中的(b)所示的状态，如睫状肌收缩，晶状体凸起，屈光度变大，从而使得如图4中的(b)所示的入射光可以通过晶状体之后汇聚在眼底的视网膜上。也就是说，在人眼观察不同距离物体时，睫状肌的收缩或放松状态是不同的。Zooming can be understood as adjusting the human eye to the correct focal length when observing an object. Normally, the brain controls the lens to adjust to the correct focus through the ciliary muscle. Exemplarily, it will be described in conjunction with FIG. 3B and FIG. 4 . Figure 3B is a schematic diagram of the composition of the human eye. As shown in FIG. 3B , the human eye may include a lens and ciliary muscle, as well as a retina located in the fundus. Among them, the crystalline lens can function as a zoom lens, converging the light rays entering the human eye. In order to converge the incident light onto the retina of the human eye fundus, so that the scene in the actual scene can form a clear image on the retina. The ciliary muscle can be used to adjust the shape of the lens. For example, the ciliary muscle can adjust the diopter of the lens by contracting or relaxing, so as to achieve the effect of adjusting the focal length of the lens. Therefore, objects at different distances in the actual scene can be clearly imaged on the retina through the lens. As an example, refer to FIG. 4 , which is a diagram illustrating the adjustment of the ciliary muscle to the lens when the human eye observes objects at different distances. As shown in (a) of FIG. 4 , when the human eye observes a distant object, take the object as a non-light source as an example. The reflected rays from the surface of this object can be close to parallel rays. At this time, the ciliary muscle can control the state of the lens to the state shown in Figure 4 (a), such as the ciliary muscle relaxes, and controls the lens to be flat and the diopter is small, so that parallel incident light can pass through the lens Converge on the retina of the fundus. When the human eye observes a relatively close object, in combination with (b) in FIG. 4 , take the object as a non-light source as an example. The reflected light from the surface of the object may enter the human eye according to the optical path shown in (b) in FIG. 4 . At this time, the ciliary muscle can keep the state of the lens in the state shown in (b) in Figure 4, as the ciliary muscle contracts, the lens protrudes, and the diopter becomes larger, so that the lens shown in (b) in Figure 4 Incident light can pass through the lens and then converge on the retina in the fundus of the eye. That is to say, when the human eye observes objects at different distances, the contraction or relaxation state of the ciliary muscle is different.
变焦深度(Accommodation distance)Zoom depth (Accommodation distance)
上面提到过，人眼在观察不同距离物体时，睫状肌的收缩或放松状态是不同的，所以大脑可以在人眼观察物体时，通过当前睫状肌的收缩或放松状态，判断该物体的深度。该深度可以称为变焦深度。As mentioned above, when the human eye observes objects at different distances, the contraction or relaxation state of the ciliary muscle is different, so the brain can judge the object through the contraction or relaxation state of the current ciliary muscle when the human eye observes the object. depth. This depth may be referred to as zoom depth.
因此，在会聚过程中人脑中会根据辐辏角度θ确定出辐辏深度，在变焦时，人脑中会 根据睫状肌的收缩或放松状态，确定变焦深度，辐辏深度和变焦深度都表征物体距离用户眼镜之间的距离。一般，在现实场景中，用户观察物体时，辐辏深度和变焦深度是协调的或者说是一致的。Therefore, during the convergence process, the human brain will determine the convergence depth according to the convergence angle θ. When zooming, the human brain will determine the zoom depth according to the contraction or relaxation state of the ciliary muscle. Both the convergence depth and the zoom depth represent the object distance The distance between the user's glasses. Generally, in a real scene, when a user observes an object, the depth of convergence and the depth of zoom are coordinated or consistent.
如果辐辏深度和变焦深度所指示的物体的深度不一致时，大脑无法准确判断物体的深度，就会出现疲劳感，影响用户体验。一般，辐辏深度和变焦深度所指示的物体的深度不一致也可称为视觉辐辏调节冲突(Vergence accommodation conflict，VAC)。If the depth of the object indicated by the convergence depth and the zoom depth are inconsistent, the brain cannot accurately judge the depth of the object, and a sense of fatigue will occur, which will affect the user experience. Generally, the depth inconsistency of the object indicated by the vergence depth and the zoom depth may also be called vergence accommodation conflict (Vergence accommodation conflict, VAC).
目前，大多数的VR产品(如，没有结合变焦面或多焦面技术的VR产品)向用户展示虚拟环境时，大多会出现VAC。At present, when most VR products (for example, VR products without zoom plane or multi-focal plane technology) display a virtual environment to users, most of them will have VAC.
[根据细则91更正 29.09.2022]　
示例性的，以通过VR技术向用户展示虚拟环境的电子设备是VR眼镜为例。图5为一种VR眼镜的示意图。如图5所示，在VR眼镜中可以设置有2个显示屏(如显示屏501以及显示屏502，显示屏501和显示屏502可以是独立的显示屏，或者，显示屏501和显示屏502可以是同一块显示屏上的不同显示区域)，每个显示屏都具有显示功能。每个显示屏都可以通过对应的目镜，用于向用户的一个人眼(如左眼或右眼)显示对应的内容。其中，显示屏501对应目镜503，显示屏502对应目镜504。比如，在显示屏501上，可以显示虚拟环境对应的左眼图像。该左眼图像的光线可以通过目镜503，汇聚在左眼处，从而使得左眼看到左眼图像。同理，在显示屏502上，可以显示虚拟环境对应的右眼图像。该右眼图像的光线可以通过目镜504，汇聚在右眼处，从而使得右眼看到右眼图像。由此，大脑可以通过对左眼图像和右眼图像进行融合，从而使得用户看到左眼图像和右眼图像对应的虚拟环境中的物体。[Corrected 29.09.2022 under Rule 91]
Exemplarily, an electronic device that displays a virtual environment to a user through a VR technology is VR glasses as an example. Fig. 5 is a schematic diagram of VR glasses. As shown in Figure 5, two display screens (such as display screen 501 and display screen 502) can be set in VR glasses, and display screen 501 and display screen 502 can be independent display screens, or display screen 501 and display screen 502 can be different display areas on the same display screen), and each display screen has a display function. Each display screen can be used to display corresponding content to one eye (such as left eye or right eye) of the user through a corresponding eyepiece. Wherein, the display screen 501 corresponds to the eyepiece 503 , and the display screen 502 corresponds to the eyepiece 504 . For example, on the display screen 501, a left-eye image corresponding to the virtual environment may be displayed. The light of the left-eye image can pass through the eyepiece 503 and converge at the left eye, so that the left eye can see the left-eye image. Similarly, on the display screen 502, the right-eye image corresponding to the virtual environment may be displayed. The light of the right-eye image can pass through the eyepiece 504 and converge at the right eye, so that the right eye sees the right-eye image. Thus, the brain can fuse the left-eye image and the right-eye image, so that the user can see objects in the virtual environment corresponding to the left-eye image and the right-eye image.
需要说明的是，由于目镜的汇聚作用，人眼看到的图像实际上是对应显示屏上显示的图像在如图6A所示的虚像面600上的图像。例如，左眼看到的左眼图像可以是在虚像面600上左眼图像对应的虚像。又如，右眼看到的右眼图像可以是在虚像面600上右眼图像对应的虚像。It should be noted that due to the converging effect of the eyepiece, the image seen by human eyes is actually an image corresponding to the image displayed on the display screen on the virtual image plane 600 as shown in FIG. 6A . For example, the left-eye image seen by the left eye may be a virtual image corresponding to the left-eye image on the virtual image plane 600 . As another example, the right-eye image seen by the right eye may be a virtual image corresponding to the right-eye image on the virtual image plane 600 .
示例性的，结合图6B。以图像上的物体是三角形为例，图像在显示屏上显示。在人眼观察对应的显示屏时，为了能够看清楚显示屏上的图像，那么睫状肌就会调整双眼的晶状体，使得虚像面600上的图像可以通过晶状体汇聚在视网膜上。因此，变焦距离可以是虚像面600到人眼的距离(如图6B所示的深度1)。而VR眼镜向用户展示的虚拟环境中的物体往往并不在虚像面600上。比如，图6B中虚拟环境中的被观察物体三角形(由于是虚拟环境，所以三角形使用虚线表示)不在虚像面600上。用户可以通过旋转眼球，将双眼视线汇聚到虚拟环境中的三角形上。辐辏角度θ如图6B的标识所示。这样，辐辏深度就应当为虚拟环境下，被观察物体(即三角形)的深度。比如，该辐辏深度可以为如图6B所示的深度2。For example, refer to FIG. 6B. Taking the object on the image as a triangle as an example, the image is displayed on the display screen. When human eyes observe the corresponding display screen, in order to see the images on the display screen clearly, the ciliary muscle will adjust the lenses of both eyes so that the images on the virtual image surface 600 can converge on the retina through the lenses. Therefore, the zoom distance may be the distance from the virtual image plane 600 to the human eye (depth 1 as shown in FIG. 6B ). However, the objects in the virtual environment displayed by the VR glasses to the user are often not on the virtual image plane 600 . For example, the observed object triangle in the virtual environment in FIG. 6B (because it is a virtual environment, the triangle is represented by a dotted line) is not on the virtual image plane 600 . By rotating the eyeballs, users can focus their eyes on the triangle in the virtual environment. The angle of convergence θ is shown as a label in FIG. 6B . In this way, the depth of convergence should be the depth of the observed object (ie, triangle) in the virtual environment. For example, the depth of convergence may be depth 2 as shown in FIG. 6B .
可以看到，此时深度1和深度2并不一致。这样，就会使得大脑无法准确判断被观察物体的深度，从而引发大脑疲劳感，影响用户体验。It can be seen that depth 1 and depth 2 are not consistent at this time. In this way, the brain cannot accurately judge the depth of the observed object, thereby causing brain fatigue and affecting user experience.
上面是以一个被观察物体为例的，一般情况下，虚拟环境中包括多个被观察物体。如图6C，被观察物体包括两个，其中，被观察物体1以三角形(虚线)为例，被观察物体2以圆球(虚线)为例。对于每个被观察物体，都会存在辐辏深度与变焦深度不同的情况。The above is an example of an observed object. Generally, a virtual environment includes multiple observed objects. As shown in FIG. 6C , there are two observed objects, wherein the observed object 1 is a triangle (dotted line) as an example, and the observed object 2 is a sphere (dashed line) as an example. For each observed object, there will be cases where the depth of convergence and the depth of zoom are different.
此外，目前的VR技术中，一张图像上所有虚拟物体(即被观察物体)的清晰度是相同的。继续以图6C为例，被观察物体以三角形和圆球为例进行说明。三角形和圆球在同一张图像上显示，且清晰度相同，而且三角形和圆球对应的虚像面600在同一深度(即深 度1)，所以人脑基于同一虚像面会认为两个被观察物体的变焦深度应该相同。但实际上两个被观察物体的辐辏深度是不同的，三角形的辐辏深度是深度2，圆球的辐辏深度是深度3，人脑基于不同辐辏深度会认为这两个被观察物体的变焦深度应该是不同的，与人脑基于同一虚像面认为两个被观察物体变焦深度相同有冲突，所以人脑无法准确的判断物体深度，就会出现疲劳感。因此，当虚拟环境中存在多个对象，且不同对象具有相同清晰度会加重用户疲劳感，影响用户体验。In addition, in the current VR technology, the clarity of all virtual objects (that is, observed objects) on an image is the same. Continuing to take FIG. 6C as an example, the observed objects are described by taking triangles and spheres as examples. The triangle and the sphere are displayed on the same image with the same definition, and the virtual image plane 600 corresponding to the triangle and the sphere are at the same depth (that is, depth 1), so the human brain will consider the zoom of the two observed objects based on the same virtual image plane The depth should be the same. But in fact, the convergence depth of the two observed objects is different. The convergence depth of the triangle is depth 2, and the convergence depth of the sphere is depth 3. Based on the different convergence depths, the human brain will think that the zoom depth of the two observed objects should be It is different. It conflicts with the human brain’s belief that the zoom depth of two observed objects is the same based on the same virtual image surface, so the human brain cannot accurately judge the depth of the object, and a sense of fatigue will appear. Therefore, when there are multiple objects in the virtual environment, and different objects have the same definition, user fatigue will be aggravated and user experience will be affected.
为了解决VR技术中的VAC，存在一种解决方案为，调整虚像面600的位置，以使变焦深度和辐辏深度一致，解决因变焦深度和辐辏深度不一致而带来的大脑疲劳感。以图6B为例，这种技术可以将虚像面600调整到虚拟环境中被观察物体(如三角形)所在景深处，使得变焦深度和辐辏深度一致。以图6C为例，这种技术可以调整三角形对应的虚像面到三角形所在景深处，将圆球对应的虚像面调整到圆形所在景深处，这样的话，三角形对应的变焦深度和辐辏深度一致了，圆球对应的变焦深度和辐辏深度也一致了，从而克服了VAC。但是，这种调整虚像面位置的技术需要一定的光学硬件的支撑，比如步进电机等，一方面，额外的光学硬件会增加成本，另一方面，增加光学硬件会增大VR眼镜的体积，难以应用在轻小型VR穿戴设备上。In order to solve the VAC in VR technology, there is a solution to adjust the position of the virtual image plane 600 so that the zoom depth and the convergence depth are consistent, so as to solve the brain fatigue caused by the inconsistency between the zoom depth and the convergence depth. Taking FIG. 6B as an example, this technology can adjust the virtual image plane 600 to the depth of field where the observed object (such as a triangle) is located in the virtual environment, so that the zoom depth and the convergence depth are consistent. Take Figure 6C as an example, this technology can adjust the virtual image surface corresponding to the triangle to the depth of field where the triangle is located, and adjust the virtual image surface corresponding to the sphere to the depth of field where the circle is located. In this way, the zoom depth corresponding to the triangle is consistent with the depth of convergence , the zoom depth and convergence depth corresponding to the ball are also consistent, thus overcoming VAC. However, this technology of adjusting the position of the virtual image plane requires the support of certain optical hardware, such as stepping motors. On the one hand, additional optical hardware will increase the cost; on the other hand, adding optical hardware will increase the volume of VR glasses. Difficult to apply to light and small VR wearable devices.
为了解决上述技术问题，本申请实施例提供一种显示方法。比如，VR显示设备(如VR眼镜)向用户展示图像，图像上不同对象的清晰度不同，有的对象清晰，有的对象模糊。示例性的，以图6C为例，本申请实施例提供的显示方法中，可以设置圆球和三角形的清晰度不同。比如，圆球模糊、三角形清晰。人脑基于同一虚像面会认为三角形和圆球的变焦深度是相同的即深度1。由于三角形清晰，人脑会认为该深度1对于三角形来说是准确的；而圆球模糊，所以人脑会认为该深度1对于圆球来说是不准确的或未调整好的，人脑会试图调整睫状肌以清晰的看到圆球。这样，人脑会判断出三角形和圆球的变焦深度不再是同一个深度，这与“人脑基于三角形和圆球的不同辐辏深度认为这两个被观察物体的变焦深度应该不同”不再冲突，缓解了人脑的疲劳感。In order to solve the above technical problem, an embodiment of the present application provides a display method. For example, a VR display device (such as VR glasses) displays an image to a user, and different objects on the image have different resolutions, some objects are clear, and some objects are blurred. Exemplarily, taking FIG. 6C as an example, in the display method provided by the embodiment of the present application, the clarity of the sphere and the triangle can be set to be different. For example, a sphere is blurred and a triangle is clear. Based on the same virtual image surface, the human brain will think that the zoom depth of the triangle and the sphere is the same, that is, depth 1. Since the triangle is clear, the human brain will think that the depth 1 is accurate for the triangle; but the ball is blurred, so the human brain will think that the depth 1 is inaccurate or not adjusted for the ball, and the human brain will Try to adjust the ciliary muscle to see the sphere clearly. In this way, the human brain will judge that the zoom depths of the triangle and the sphere are no longer the same depth, which is no longer the same as "the human brain thinks that the zoom depths of the two observed objects should be different based on the different convergence depths of the triangle and the sphere". Conflict relieves the fatigue of the human brain.
进一步的，一般来说，人眼看近处物体时，看到的细节越多越清楚，看远处物体时，看到的细节越少越模糊(这一原理也被称为人眼成像的金字塔原理)。继续以图6C为例，VR眼镜展示的虚拟环境中远处的圆球模糊、近处的三角形清晰，这样，人感受到的虚拟环境比较符合真实情况(近景清晰，远景模糊)。Furthermore, generally speaking, when the human eye looks at nearby objects, the more details it sees, the clearer it is, and when looking at distant objects, the details it sees are less and more blurred (this principle is also known as the pyramid principle of human eye imaging. ). Continuing to take Figure 6C as an example, in the virtual environment displayed by the VR glasses, the distant spheres are blurred and the nearby triangles are clear. In this way, the virtual environment perceived by people is more in line with the real situation (close view is clear, distant view is blurred).
因此，本申请实施例提供的显示方法，能够缓解用户通过VR眼镜观看虚拟环境时产生的疲劳感，提升用户体验，而且不需要依赖步进电机等特殊光学硬件的支撑，成本低、有助于设备轻小型化。Therefore, the display method provided by the embodiment of the present application can alleviate the user's fatigue when viewing the virtual environment through VR glasses, improve user experience, and does not need to rely on the support of special optical hardware such as stepping motors, and is low in cost and helpful The equipment is light and miniaturized.
示例性的，图7A和图7B为本申请实施例提供的第一种应用场景的示意图。该应用场景以用户佩戴VR眼镜进行VR游戏为例。Exemplarily, FIG. 7A and FIG. 7B are schematic diagrams of the first application scenario provided by the embodiment of the present application. This application scenario takes a user wearing VR glasses to play a VR game as an example.
如图7A所示，VR眼镜显示图像701。图像701可以是VR游戏应用生成的图像，其中包括***、集装箱、树等对象。假设***在景深1、集装箱在景深2、树在景深3。景深3>景深2>景深1。假设图像701中用户注视点在***(如瞄准镜)，由于树所在的景深3距离景深1较远，集装箱所在的景深2距离景深1较近，所以图像701上树的清晰度比集装箱的清晰度低。因此，用户佩戴VR眼睛看到的VR游戏画面中树是模糊的，集装箱是 清晰的。也就是说，该应用场景中，图像上不同对象的清晰度不同。具体地，距离用户注视点远的对象(如树)比较模糊，距离用户注视点较近的对象(如集装箱)比较清晰。As shown in FIG. 7A , the VR glasses display an image 701 . The image 701 may be an image generated by a VR game application, including objects such as guns, containers, and trees. Suppose the gun is at depth 1, the container is at depth 2, and the tree is at depth 3. Depth of Field 3 > Depth of Field 2 > Depth of Field 1. Assuming that in image 701, the user’s gaze point is on the gun (such as a scope), since the depth of field 3 where the tree is located is farther from depth of field 1, and the depth of field 2 where the container is located is closer to depth of field 1, the definition of the tree on image 701 is clearer than that of the container low degree. Therefore, the tree in the VR game screen seen by the user wearing VR glasses is blurred, but the container is clear. That is to say, in this application scenario, the sharpness of different objects on the image is different. Specifically, objects far away from the user's gaze point (such as trees) are relatively blurred, and objects relatively close to the user's gaze point (such as containers) are relatively clear.
由于树和集装箱的景深不同，所以人脑会认为树和集装箱的辐辏深度不同；再加上树和集装箱的清晰度不同，人脑会认为树和集装箱的变焦深度应该不同，这与人脑所认为的树和集装箱的辐辏深度不同相匹配，能够缓解大脑疲劳。进一步来讲，在现实环境中，人眼注视某个物体时，眼睛看到的该物体比较清晰，距离该物体较远的其它物体比较模糊。上述应用场景中，距离用户注视点较远的树模糊，距离用户注视点较近的集装箱清晰，这样人眼所看到的虚拟环境比较符合真实情况。Because the depth of field of the tree and the container is different, the human brain will think that the convergence depth of the tree and the container is different; in addition, the definition of the tree and the container is different, the human brain will think that the zoom depth of the tree and the container should be different, which is different from the human brain. It is thought that the different depths of convergence of the tree and the container match, which can relieve brain fatigue. Further speaking, in a real environment, when human eyes look at an object, the object seen by the eyes is relatively clear, and other objects farther away from the object are relatively blurred. In the above application scenarios, the trees farther from the user's gaze point are blurred, and the containers closer to the user's gaze point are clear, so that the virtual environment seen by the human eye is more in line with the real situation.
需要说明的是，图7A中以VR眼镜显示一帧图像(即图像701)为例，需要说明的是，一般VR眼镜显示图像流(如VR游戏应用生成的图像流)。图像流包括多帧图像。在显示多帧图像时，用户的注视点可能会发生变化，VR眼镜可以通过眼动跟踪模组实时检测用户注视点，当注视点发生变化时，基于新的注视点确定对象的清晰度。比如，距离新的注视点远的对象模糊，距离新的注视点较近的对象清晰。It should be noted that, in FIG. 7A , the VR glasses display a frame of image (ie, image 701 ) as an example. It should be noted that general VR glasses display an image stream (such as an image stream generated by a VR game application). An image stream includes multiple frames of images. When displaying multiple frames of images, the user's gaze point may change. VR glasses can detect the user's gaze point in real time through the eye tracking module. When the gaze point changes, the clarity of the object is determined based on the new gaze point. For example, objects far from the new fixation point are blurred, and objects closer to the new fixation point are sharp.
为了方便描述，下文以该应用场景中用户注视点保持在***处为例介绍VR眼镜显示图像流的过程。For the convenience of description, the process of displaying image streams in VR glasses is described below by taking the user's gaze point on the gun in this application scenario as an example.
一种可实现方式为，用户注视点保持在***的期间内，VR眼镜显示图像流，图像流中每帧图像上都将树(远离用户注视点的对象)模糊化。也就是说，用户注视点在***的期间内，远离用户注视点的对象一直处于模糊状态。这样虽然能够缓解疲劳感，但是会丢失对象(如，树)细节，使得用户可能会错过一些细节导致游戏失败。One possible implementation is that while the user's gaze remains on the gun, the VR glasses display an image stream, and trees (objects far away from the user's gaze) are blurred on each frame of the image stream. That is to say, during the period when the user's gaze is on the gun, objects far away from the user's gaze are always in a blurred state. Although this can relieve fatigue, it will lose object (eg, tree) details, so that the user may miss some details and cause the game to fail.
为了既能够缓解疲劳感，又能保证人脑中摄取到对象(如，树)的细节，用户注视点保持在***的期间内，VR眼镜显示图像流，图像流中远离用户注视点的对象(如，树)的清晰度可以有高有低，不需要一直处于模糊状态。In order to not only relieve fatigue, but also ensure that the details of objects (such as trees) are captured in the human brain, during the period when the user's gaze point is kept on the gun, the VR glasses display the image stream. , tree) can be high or low, and does not need to be in a blurry state all the time.
示例性的，如图7B，VR眼镜显示图像流，图像流包括第i帧、第j帧、第k帧。其中，第i帧图像中树(远离用户注视点的对象)是模糊的，第j帧图像中树是清晰的，第k帧图像中树是模糊的。这样的话，当VR眼镜显示第i帧图像时，用户看到的树是模糊的，可以以缓解疲劳感，当VR眼睛显示第j帧图像时，用户看到的树是清晰的，由于视觉停留效应，人脑中会将第i帧图像和第j帧图像上的树融合，所以虽然第i帧图像中的树模糊化了，但是仍然能保证人脑中不会丢失树的细节。也就是说，图7B中，VR眼镜显示图像流时，对于距离用户注视点较远的对象(如树)，其清晰度有高有低，不需要一直处于模糊状态，通过这种方式，不仅可以缓解疲劳感，又能保证人脑中摄取到远离用户注视点的对象的足够细节，用户体验较好。Exemplarily, as shown in FIG. 7B , the VR glasses display an image stream, and the image stream includes an i-th frame, a j-th frame, and a k-th frame. Among them, the tree (the object far away from the user's gaze point) is blurred in the i-th frame image, the tree is clear in the j-th frame image, and the tree is blurred in the k-th frame image. In this way, when the VR glasses display the image of the i-th frame, the tree that the user sees is blurred, which can relieve fatigue. When the VR glasses display the image of the j-th frame, the tree that the user sees is clear. Effect, the human brain will fuse the i-th frame image with the tree on the j-th frame image, so although the tree in the i-th frame image is blurred, it can still ensure that the details of the tree will not be lost in the human brain. That is to say, in Fig. 7B, when the VR glasses display the image stream, for objects (such as trees) that are far away from the user's gaze point, the definition may be high or low, and it does not need to be in a blurred state all the time. In this way, not only It can relieve fatigue and ensure that the human brain can capture enough details of objects far away from the user's gaze point, and the user experience is better.
可选的，图7B中对于靠近用户注视点的对象(如集装箱)，其清晰度可以不变。示例性的，由于树的清晰度有高有低的变化，但其清晰度不得超过集装箱的清晰度。比如，图7B中第j帧图像上树是清晰的，集装箱也是清晰的，但是树的清晰度低于或等于集装箱的清晰度。或者，对于靠近用户注视点的对象(如集装箱)，其清晰度也可以有高有低，只要同一张图像上靠近用户注视点的对象的清晰度高于或等于远离用户注视点的对象的清晰度即可。比如，图7B中第i帧图像上树模糊化了，集装箱也可以模糊化，但是集装箱模糊化程度低于树的模糊化程度，以使集装箱的清晰度高于树的清晰度。简单来说，与用户注视点距离越远的对象，模糊化程度越高，清晰度越低，与用户注视点距离越近的对象，模糊化程度越低，清晰度越高。Optionally, in FIG. 7B , for an object (such as a container) close to the user's gaze point, its definition may not change. Exemplarily, since the definition of the tree varies from high to low, its definition must not exceed that of the container. For example, the tree and the container are clear on the image frame j in FIG. 7B , but the resolution of the tree is lower than or equal to that of the container. Or, for objects close to the user's gaze point (such as containers), the sharpness can also be high or low, as long as the sharpness of objects close to the user's gaze point on the same image is higher than or equal to that of objects far away from the user's gaze point degree can be. For example, the tree is blurred on the i-th frame image in Figure 7B, and the container can also be blurred, but the blurring degree of the container is lower than that of the tree, so that the definition of the container is higher than that of the tree. To put it simply, the farther the distance from the user's gaze point, the higher the degree of blurring and the lower the clarity, and the closer the distance to the user's gaze point, the lower the blurring degree and the higher the clarity.
需要说明的是，图7A和图7B中，以用户注视点在***为例，换言之，用户注视点可以在该用户对应的游戏角色当前所持游戏装备(如***)上，可以理解的是，用户注视点还可以在游戏对方对应的游戏角色处，或者，建筑物处，或者，该用户对应的游戏角色上身体部位处，等等，本申请实施例不一一举例。It should be noted that in FIG. 7A and FIG. 7B , the user's gaze point is on a gun as an example. In other words, the user's gaze point can be on the game equipment (such as a gun) currently held by the game character corresponding to the user. The point of gaze can also be at the game character corresponding to the opponent in the game, or at the building, or at the body part of the game character corresponding to the user, etc., and the embodiments of the present application do not give examples one by one.
示例性的，图8A和图8B为本申请实施例提供的第二种应用场景的示例。该应用场景中用户佩戴VR眼镜进行VR驾驶为例。Exemplarily, FIG. 8A and FIG. 8B are examples of the second application scenario provided by the embodiment of the present application. In this application scenario, the user wears VR glasses for VR driving as an example.
如图8A所示，VR眼镜显示图像801。示例性的，图像801可以是VR驾驶应用生成的图像。图像801中包括方向盘、显示器等车辆信息、还包括道路、位于道路上的树和前方车辆等等。其中，树所在的景深2大于前方车辆所在的景深1。因此，图像801中树的清晰度低于前方车辆的清晰度。也就是说，景深较大的对象比较模糊，景深较小的对象比较清晰。该应用场景与图7A和图7B所示的应用场景不同。前文图7A和图7B所示的应用场景中以用户注视点为准，远离用户注视点的对象模糊，靠近用户注视点的对象清晰，图8A和图8B的应用场景中远处对象模糊，近处对象清晰，与用户注视点不相关。As shown in FIG. 8A , the VR glasses display an image 801 . Exemplarily, the image 801 may be an image generated by a VR driving application. The image 801 includes vehicle information such as a steering wheel and a monitor, and also includes roads, trees on the roads, vehicles in front, and the like. Wherein, the depth of field 2 where the tree is located is greater than the depth of field 1 where the vehicle in front is located. Therefore, the sharpness of the trees in image 801 is lower than that of the vehicle ahead. That is, objects with a larger depth of field are blurred, and objects with a smaller depth of field are sharper. This application scenario is different from the application scenarios shown in Fig. 7A and Fig. 7B. In the application scenarios shown in Figure 7A and Figure 7B above, the user's gaze point shall prevail. Objects far away from the user's gaze point are blurred, and objects close to the user's gaze point are clear. Objects are clear and uncorrelated with the user's gaze point.
继续参见图8A，由于树和前方车辆的景深不同，所以人脑会认为树和前方车辆的辐辏深度不同；再加上树和前方车辆的清晰度不同，人脑会认为树和前方车辆的变焦深度应该不同，这与人脑所认为的树和前方车辆的辐辏深度不同相匹配，能缓解大脑疲劳。进一步来讲，在现实环境中，人眼在看近处物体时，看到的细节越多越清楚，当看远处物体时，看到的细节越少越模糊。因此，图8A的虚拟环境中近处对象清晰，远处对象模糊，这样人眼所看到的虚拟环境比较符合真实情况。Continue to refer to Figure 8A. Since the depth of field of the tree and the vehicle in front are different, the human brain will think that the depth of convergence between the tree and the vehicle in front is different; in addition, the resolution of the tree and the vehicle in front is different, and the human brain will think that the zoom of the tree and the vehicle in front is different. The depth should be different, which matches the different convergence depths of the tree and the vehicle in front that the human brain thinks, and can relieve brain fatigue. Furthermore, in a real environment, when the human eye looks at nearby objects, the more details it sees, the clearer it is. When looking at distant objects, the details it sees are less and blurred. Therefore, in the virtual environment shown in FIG. 8A , nearby objects are clear and distant objects are blurred, so that the virtual environment seen by human eyes is more in line with the real situation.
需要说明的是，图8A中以VR眼镜显示一帧图像(即图像801)为例，可以理解的是，VR眼镜可以显示图像流(如VR驾驶应用生成的图像流)。图像流中包括多帧图像。一种可能的实现方式为，图像流中每帧图像上的远处对象都模糊化。这种方式虽然可以缓解疲劳感，但是会丢失远处对象的细节。It should be noted that, in FIG. 8A , the VR glasses display a frame of image (that is, image 801 ) as an example. It can be understood that the VR glasses can display an image stream (such as an image stream generated by a VR driving application). The image stream includes multiple frames of images. A possible implementation manner is that distant objects on each image frame in the image stream are blurred. Although this method can relieve fatigue, it will lose the details of distant objects.
为了既能够缓解疲劳感，又能保证人脑中摄取到远处对象的细节，VR眼镜显示图像流时，图像流中远处对象(如，树)的清晰度可以有高有低，不需要一直处于模糊状态。In order to relieve fatigue and ensure that the details of distant objects are captured in the human brain, when VR glasses display image streams, the clarity of distant objects (such as trees) in the image stream can be high or low, and it does not need to be constantly fuzzy state.
示例性的，如图8B，VR眼镜显示图像流，图像流包括第i帧、第j帧、第k帧。其中，第i帧图像上树是模糊的，第j帧图像上树是清晰的，第k帧图像上树是模糊的。当VR眼睛显示第i帧图像时，用户看到的树是模糊的，可以缓解疲劳感，当VR眼镜显示第j帧图像时，用户看到的树是清晰的，由于视觉停留效应，人脑中会将第i帧图像和第j帧图像上的树融合，所以虽然第i帧图像中的树模糊化了，但是仍然能保证人脑中不会丢失树的细节。也就是说，图8B中，VR眼镜显示的图像流中景深较大的对象其清晰度有高有低，不需要一直处于模糊状态。通过这种方式，不仅可以缓解疲劳感，又能保证人脑中摄取到远处对象的细节，用户体验较好。Exemplarily, as shown in FIG. 8B , the VR glasses display an image stream, and the image stream includes an i-th frame, a j-th frame, and a k-th frame. Among them, the tree on the i-th frame image is blurred, the tree on the j-th frame image is clear, and the tree on the k-th frame image is blurred. When the VR glasses display the i-th frame of image, the tree that the user sees is blurred, which can relieve fatigue. When the VR glasses display the j-th frame of image, the tree that the user sees is clear. In this method, the tree on the i-th frame image and the j-th frame image will be fused, so although the tree in the i-th frame image is blurred, it can still ensure that the details of the tree will not be lost in the human mind. That is to say, in FIG. 8B , objects with larger depths of field in the image stream displayed by the VR glasses may have high or low clarity, and do not need to be in a blur state all the time. In this way, it can not only relieve fatigue, but also ensure that the details of distant objects can be captured in the human brain, and the user experience is better.
可选的，图8B中，对于近处对象(如，前方车辆)，其清晰度也可以不变。示例性的，由于远处对象的清晰度有高有低的变化，但其清晰度不得超出近处对象的清晰度。比如，图8B中第j帧图像上树是清晰的，前方车辆也是清晰的，但是树的清晰度低于或等于前方车辆的清晰度。或者，近处对象(如前方车辆)，其清晰度也可以有高有低，只要同一张图像上近处对象的清晰度高于或等于远处对象的清晰度即可。比如，图8B中第i帧图像上树模糊化了，前方车辆也可以模糊化，但是前方车辆的糊化程度低于树的模糊化程度，以使树的清晰度低于前方车辆的清晰度。简单来说，景深越大的对象，模糊化程度越高，清 晰度越低，景深越小的对象，模糊化程度越低，清晰度越高。Optionally, in FIG. 8B , for a nearby object (eg, a vehicle in front), its clarity may also remain unchanged. Exemplarily, since the sharpness of the distant object varies from high to low, the sharpness thereof must not exceed the sharpness of the near object. For example, the tree on the image frame j in FIG. 8B is clear, and the vehicle in front is also clear, but the clarity of the tree is lower than or equal to that of the vehicle in front. Alternatively, the sharpness of nearby objects (such as vehicles in front) can also be high or low, as long as the sharpness of nearby objects on the same image is higher than or equal to the sharpness of distant objects. For example, the tree on the i-th frame image in Figure 8B is blurred, and the vehicle in front can also be blurred, but the blurring degree of the vehicle in front is lower than the blurring degree of the tree, so that the clarity of the tree is lower than that of the vehicle in front . To put it simply, objects with a larger depth of field have higher blurring and lower definition, and objects with smaller depth of field have lower blurring and higher definition.
需要说明的是，上面的实施例中，以树模糊，前方车辆清晰(树所在的景深2大于前方车辆所在的景深1)为例，换言之，以景深1为准，大于景深1的景深2处的对象模糊化。在另一些实施例中，如图8A，在车辆驾驶场景中，还可以以景深3为准，大于景深3的景深1和景深2处的对象都可以模糊化。可以理解的是，图8A中以景深3是用户当前驾驶车辆所在景深为例，可以理解是的，景深3还可以是该用户当前驾驶车辆上方向盘所在景深，或者，所述用户当前驾驶车辆上挡风玻璃所在景深，等等。It should be noted that in the above embodiment, the tree is blurred and the vehicle in front is clear (the depth of field 2 where the tree is located is greater than the depth of field 1 where the vehicle in front is located) as an example. In other words, the depth of field 2 is greater than the depth of field 1 object blurring. In other embodiments, as shown in FIG. 8A , in the vehicle driving scene, the depth of field 3 may also be used as the criterion, and the objects at the depth of field 1 and the depth of field 2 that are greater than the depth of field 3 can be blurred. It can be understood that, in FIG. 8A, the depth of field 3 is the depth of field where the user is currently driving the vehicle as an example. It can be understood that the depth of field 3 can also be the depth of field where the steering wheel of the user is currently driving the vehicle, or the user is currently driving the vehicle. Depth of field where the windshield is, and so on.
以上介绍了两种应用场景，第一种应用场景(图7A和图7B)中，以VR游戏应用为例，且以用户注视点为准，距离用户注视点较远的对象模糊，距离用户注视点较近的对象清晰，而且，当显示图像流时，图像流中距离用户注视点较远的对象的清晰度有升有降。第二种应用场景(图8A和图8B)中，以VR驾驶应用为例，远处对象模糊，近处对象清晰，而且，当显示图像流时，图像流中远处对象的清晰度有升有降。这两种方式都可以起到缓解视觉疲劳感的效果，为了方便描述，下文将第一种应用场景中以用户注视点为准，远离用户注视点的对象模糊，靠近用户注视点的对象清晰的模式称为第一种护眼模式，将第二种应用场景中远处对象模糊、近处对象清晰的模式称为第二种护眼模式。Two application scenarios have been introduced above. In the first application scenario (Figure 7A and Figure 7B), the VR game application is taken as an example, and the user's gaze point shall prevail. Objects that are closer to the point of view are sharp, and, when the image stream is displayed, the sharpness of objects in the image stream that are farther from the user's gaze point increases and decreases. In the second application scenario (Figure 8A and Figure 8B), taking the VR driving application as an example, the distant objects are blurred and the nearby objects are clear, and when the image stream is displayed, the clarity of the distant objects in the image stream increases or decreases. drop. These two methods can relieve visual fatigue. For the convenience of description, the first application scenario is based on the user's gaze point. Objects far away from the user's gaze point are blurred, and objects close to the user's gaze point are clear. The mode is called the first eye protection mode, and the mode in which the distant objects are blurred and the near objects are clear in the second application scene is called the second eye protection mode.
以上是本申请列举的两种应用场景，主要以VR游戏和VR驾驶为例，需要说明的是，上述两种应用场景可以结合，比如第一种应用场景(如，VR游戏应用)也可以应用第二种应用场景的方案实现远处对象模糊，近处对象清晰，或者，第二种应用场景(如，VR驾驶应用)也可以应用第一种应用场景的方案实现以用户注视点为准，距离用户注视点较远的对象模糊，距离用户注视点较近的对象清晰。此外，本申请实施例提供的技术方案可以适用于其它应用场景，比如，VR看车、VR看房，VR聊天、VR教学，VR影院等任何需要像用户展示虚拟环境的场景。The above are the two application scenarios listed in this application, mainly taking VR games and VR driving as examples. It should be noted that the above two application scenarios can be combined. For example, the first application scenario (such as VR game application) can also be applied. The solution of the second application scenario achieves blurred objects in the distance and clear objects nearby, or, the second application scenario (for example, VR driving application) can also apply the solution of the first application scenario to achieve the user's gaze point as the standard, Objects farther from the user's gaze are blurred, and objects closer to the user's gaze are sharp. In addition, the technical solutions provided by the embodiments of the present application can be applied to other application scenarios, such as VR viewing of cars, VR viewing of houses, VR chatting, VR teaching, VR theaters, and any other scenarios that need to display a virtual environment to the user.
下面介绍本申请相关的设备。The equipment related to this application is introduced below.
示例性的，请参考图9，以VR穿戴设备(如VR眼镜)为例，示出了本申请实施例提供的一种VR穿戴设备的结构示意图。如图9所示，VR穿戴设备100可以包括处理器110，存储器120，传感器模块130(可以用于获取用户的姿态)，麦克风140，按键150，输入输出接口160，通信模块170，摄像头180，电池190、光学显示模组1100以及眼动追踪模组1200等。For example, please refer to FIG. 9 , which shows a schematic structural diagram of a VR wearable device provided by an embodiment of the present application by taking a VR wearable device (such as VR glasses) as an example. As shown in FIG. 9 , the VR wearable device 100 may include a processor 110, a memory 120, a sensor module 130 (which may be used to acquire the user's posture), a microphone 140, buttons 150, an input and output interface 160, a communication module 170, a camera 180, battery 190 , optical display module 1100 , eye tracking module 1200 and so on.
可以理解的是，本申请实施例示意的结构并不构成对VR穿戴设备100的具体限定。在本申请另一些实施例中，VR穿戴设备100可以包括比图示更多或更少的部件，或者组合某些部件，或者拆分某些部件，或者不同的部件布置。图示的部件可以以硬件，软件或软件和硬件的组合实现。It can be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the VR wearable device 100 . In other embodiments of the present application, the VR wearable device 100 may include more or fewer components than shown in the illustration, or combine certain components, or separate certain components, or arrange different components. The illustrated components can be realized in hardware, software or a combination of software and hardware.
处理器110通常用于控制VR穿戴设备100的整体操作，可以包括一个或多个处理单元，例如：处理器110可以包括应用处理器(application processor，AP)，调制解调处理器，图形处理器(graphics processing unit，GPU)，图像信号处理器(image signal processor，ISP)，视频处理单元(video processing unit，VPU)控制器，存储器，视频编解码器，数字信号处理器(digital signal processor，DSP)，基带处理器，和/或神经网络处理器(neural-network processing unit，NPU)等。其中，不同的处理单元可以是独立的器件，也可以集成在一个或多个处理器中。The processor 110 is generally used to control the overall operation of the VR wearable device 100, and may include one or more processing units, for example: the processor 110 may include an application processor (application processor, AP), a modem processor, a graphics processor (graphics processing unit, GPU), image signal processor (image signal processor, ISP), video processing unit (video processing unit, VPU) controller, memory, video codec, digital signal processor (digital signal processor, DSP ), baseband processor, and/or neural network processor (neural-network processing unit, NPU), etc. Wherein, different processing units may be independent devices, or may be integrated in one or more processors.
处理器110中还可以设置存储器，用于存储指令和数据。在一些实施例中，处理器110中的存储器为高速缓冲存储器。该存储器可以保存处理器110刚用过或循环使用的指令或数据。如果处理器110需要再次使用该指令或数据，可从所述存储器中直接调用。避免了重复存取，减少了处理器110的等待时间，因而提高了***的效率。A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. The memory may hold instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to use the instruction or data again, it can be called directly from the memory. Repeated access is avoided, and the waiting time of the processor 110 is reduced, thus improving the efficiency of the system.
[根据细则91更正 29.09.2022]　
在本申请的一些实施例中，处理器110可以用于控制VR穿戴设备100的光焦度。示例性的，处理器110可以用于控制光学显示模组1100的光焦度，实现对VR穿戴设备100的光焦度的调整的功能。例如，处理器110可以通过调整光学显示模组1100中各个光学器件(如透镜等)之间的相对位置，使得光学显示模组1100的光焦度得到调整，进而使得光学显示模组1100在向人眼成像时，对应的虚像面的位置可以得到调整。从而达到控制VR穿戴设备100的光焦度的效果。[Corrected 29.09.2022 under Rule 91]
In some embodiments of the present application, the processor 110 may be used to control the optical power of the VR wearable device 100 . Exemplarily, the processor 110 may be used to control the optical power of the optical display module 1100 to realize the function of adjusting the optical power of the VR wearable device 100 . For example, the processor 110 can adjust the relative positions of the various optical devices (such as lenses, etc.) When the human eye is imaging, the position of the corresponding virtual image plane can be adjusted. In this way, the effect of controlling the optical power of the VR wearable device 100 is achieved.
在一些实施例中，处理器110可以包括一个或多个接口。接口可以包括集成电路(inter-integrated circuit，I2C)接口，通用异步收发传输器(universal asynchronous receiver/transmitter，UART)接口，移动产业处理器接口(mobile industry processor interface，MIPI)，通用输入输出(general-purpose input/output，GPIO)接口，用户标识模块(subscriber identity module，SIM)接口，和/或通用串行总线(universal serial bus，USB)接口，串行外设接口(serial peripheral interface，SPI)接口等。In some embodiments, processor 110 may include one or more interfaces. The interface may include an integrated circuit (inter-integrated circuit, I2C) interface, a universal asynchronous receiver/transmitter (universal asynchronous receiver/transmitter, UART) interface, a mobile industry processor interface (mobile industry processor interface, MIPI), a general input and output (general -purpose input/output, GPIO) interface, subscriber identity module (subscriber identity module, SIM) interface, and/or universal serial bus (universal serial bus, USB) interface, serial peripheral interface (serial peripheral interface, SPI) interface etc.
在一些实施例中，处理器110可以对不同景深处的对象做不同程度的模糊化处理，以使不同景深处的对象的清晰度不同。In some embodiments, the processor 110 may perform blurring processing to different degrees on objects at different depths of field, so that objects at different depths of field have different sharpness.
I2C接口是一种双向同步串行总线，包括一根串行数据线(serial data line，SDA)和一根串行时钟线(derail clock line，SCL)。在一些实施例中，处理器110可以包含多组I2C总线。The I2C interface is a bidirectional synchronous serial bus, including a serial data line (serial data line, SDA) and a serial clock line (derail clock line, SCL). In some embodiments, processor 110 may include multiple sets of I2C buses.
UART接口是一种通用串行数据总线，用于异步通信。该总线可以为双向通信总线。它将要传输的数据在串行通信与并行通信之间转换。在一些实施例中，UART接口通常被用于连接处理器110与通信模块170。例如：处理器110通过UART接口与通信模块170中的蓝牙模块通信，实现蓝牙功能。The UART interface is a universal serial data bus used for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, a UART interface is generally used to connect the processor 110 and the communication module 170 . For example: the processor 110 communicates with the Bluetooth module in the communication module 170 through the UART interface to realize the Bluetooth function.
MIPI接口可以被用于连接处理器110与光学显示模组1100中的显示屏，摄像头180等***器件。The MIPI interface can be used to connect the processor 110 with the display screen in the optical display module 1100 , the camera 180 and other peripheral devices.
GPIO接口可以通过软件配置。GPIO接口可以被配置为控制信号，也可被配置为数据信号。在一些实施例中，GPIO接口可以用于连接处理器110与摄像头180，光学显示模组1100中的显示屏，通信模块170，传感器模块130，麦克风140等。GPIO接口还可以被配置为I2C接口，I2S接口，UART接口，MIPI接口等。可选的，摄像头180可以采集包括真实对象的图像，处理器110可以将摄像头采集的图像与虚拟对象融合，通过光学显示模组1100现实融合得到的图像。可选的，摄像头180还可以采集包括人眼的图像。处理器110通过所述图像进行眼动追踪。The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 with the camera 180 , the display screen in the optical display module 1100 , the communication module 170 , the sensor module 130 , the microphone 140 and so on. The GPIO interface can also be configured as an I2C interface, I2S interface, UART interface, MIPI interface, etc. Optionally, the camera 180 may collect images including real objects, the processor 110 may fuse the images collected by the camera with the virtual objects, and display the fused images through the optical display module 1100 . Optionally, the camera 180 may also collect images including human eyes. The processor 110 performs eye tracking through the images.
USB接口是符合USB标准规范的接口，具体可以是Mini USB接口，Micro USB接口，USB Type C接口等。USB接口可以用于连接充电器为VR穿戴设备100充电，也可以用于VR穿戴设备100与***设备之间传输数据。也可以用于连接耳机，通过耳机播放音频。该接口还可以用于连接其他电子设备，例如手机等。USB接口可以是USB3.0，用于兼容高速显示接口(display port，DP)信号传输，可以传输视音频高速数据。The USB interface is an interface that conforms to the USB standard specification, specifically, it can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface can be used to connect a charger to charge the VR wearable device 100, and can also be used to transmit data between the VR wearable device 100 and peripheral devices. It can also be used to connect headphones and play audio through them. This interface can also be used to connect other electronic devices, such as mobile phones. The USB interface may be USB3.0, which is compatible with high-speed display port (DP) signal transmission, and can transmit video and audio high-speed data.
[根据细则91更正 29.09.2022]　
可以理解的是，本申请实施例示意的各模块间的接口连接关系，只是示意性说明，并 不构成对VR穿戴设备100的结构限定。在本申请另一些实施例中，穿戴设备100也可以采用上述实施例中不同的接口连接方式，或多种接口连接方式的组合。[Corrected 29.09.2022 under Rule 91]
It can be understood that the interface connection relationship between modules shown in the embodiment of the present application is only a schematic illustration, and does not constitute a structural limitation of the VR wearable device 100 . In other embodiments of the present application, the wearable device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.
另外，VR穿戴设备100可以包含无线通信功能，比如，VR穿戴设备100可以从其它电子设备(比如VR主机)接收图像进行显示。通信模块170可以包含无线通信模块和移动通信模块。无线通信功能可以通过天线(未示出)、移动通信模块(未示出)，调制解调处理器(未示出)以及基带处理器(未示出)等实现。天线用于发射和接收电磁波信号。VR穿戴设备100中可以包含多个天线，每个天线可用于覆盖单个或多个通信频带。不同的天线还可以复用，以提高天线的利用率。例如：可以将天线1复用为无线局域网的分集天线。在另外一些实施例中，天线可以和调谐开关结合使用。In addition, the VR wearable device 100 may include a wireless communication function, for example, the VR wearable device 100 may receive images from other electronic devices (such as a VR host) for display. The communication module 170 may include a wireless communication module and a mobile communication module. The wireless communication function can be realized by an antenna (not shown), a mobile communication module (not shown), a modem processor (not shown), and a baseband processor (not shown). Antennas are used to transmit and receive electromagnetic wave signals. Multiple antennas may be included in the VR wearable device 100, and each antenna may be used to cover a single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization of the antennas. For example: Antenna 1 can be multiplexed as a diversity antenna of a wireless local area network. In other embodiments, the antenna may be used in conjunction with a tuning switch.
移动通信模块可以提供应用在VR穿戴设备100上的包括第二代(2th generation，2G)网络/第三代(3th generation，3G)网络/***(4th generation，4G)网络/第五代(5th generation，5G)网络等无线通信的解决方案。移动通信模块可以包括至少一个滤波器，开关，功率放大器，低噪声放大器(low noise amplifier，LNA)等。移动通信模块可以由天线接收电磁波，并对接收的电磁波进行滤波，放大等处理，传送至调制解调处理器进行解调。移动通信模块还可以对经调制解调处理器调制后的信号放大，经天线转为电磁波辐射出去。在一些实施例中，移动通信模块的至少部分功能模块可以被设置于处理器110中。在一些实施例中，移动通信模块的至少部分功能模块可以与处理器110的至少部分模块被设置在同一个器件中。The mobile communication module can provide applications on the VR wearable device 100 including second generation (2th generation, 2G) network/third generation (3th generation, 3G) network/fourth generation (4th generation, 4G) network/fifth generation (5th generation, 5G) network and other wireless communication solutions. The mobile communication module may include at least one filter, switch, power amplifier, low noise amplifier (low noise amplifier, LNA) and the like. The mobile communication module can receive electromagnetic waves through the antenna, filter and amplify the received electromagnetic waves, and send them to the modem processor for demodulation. The mobile communication module can also amplify the signal modulated by the modem processor, and convert it into electromagnetic wave and radiate it through the antenna. In some embodiments, at least part of the functional modules of the mobile communication module may be set in the processor 110 . In some embodiments, at least part of the functional modules of the mobile communication module and at least part of the modules of the processor 110 may be set in the same device.
调制解调处理器可以包括调制器和解调器。其中，调制器用于将待发送的低频基带信号调制成中高频信号。解调器用于将接收的电磁波信号解调为低频基带信号。随后解调器将解调得到的低频基带信号传送至基带处理器处理。低频基带信号经基带处理器处理后，被传递给应用处理器。应用处理器通过音频设备(不限于扬声器等)输出声音信号，或通过光学显示模组1100中的显示屏显示图像或视频。在一些实施例中，调制解调处理器可以是独立的器件。在另一些实施例中，调制解调处理器可以独立于处理器110，与移动通信模块或其他功能模块设置在同一个器件中。A modem processor may include a modulator and a demodulator. Wherein, the modulator is used for modulating the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low frequency baseband signal. Then the demodulator sends the demodulated low-frequency baseband signal to the baseband processor for processing. The low-frequency baseband signal is passed to the application processor after being processed by the baseband processor. The application processor outputs sound signals through audio equipment (not limited to speakers, etc.), or displays images or videos through the display screen in the optical display module 1100 . In some embodiments, the modem processor may be a stand-alone device. In some other embodiments, the modem processor may be independent from the processor 110, and be set in the same device as the mobile communication module or other functional modules.
无线通信模块可以提供应用在VR穿戴设备100上的包括无线局域网(wireless local area networks，WLAN)(如无线保真(wireless fidelity，Wi-Fi)网络)，蓝牙(bluetooth，BT)，全球导航卫星***(global navigation satellite system，GNSS)，调频(frequency modulation，FM)，近距离无线通信技术(near field communication，NFC)，红外技术(infrared，IR)等无线通信的解决方案。无线通信模块可以是集成至少一个通信处理模块的一个或多个器件。无线通信模块经由天线接收电磁波，将电磁波信号调频以及滤波处理，将处理后的信号发送到处理器110。无线通信模块还可以从处理器110接收待发送的信号，对其进行调频，放大，经天线转为电磁波辐射出去。The wireless communication module can provide wireless local area networks (wireless local area networks, WLAN) (such as wireless fidelity (Wireless fidelity, Wi-Fi) network), bluetooth (bluetooth, BT), global navigation satellite, etc. applied on the VR wearable device 100. System (global navigation satellite system, GNSS), frequency modulation (frequency modulation, FM), near field communication technology (near field communication, NFC), infrared technology (infrared, IR) and other wireless communication solutions. The wireless communication module may be one or more devices integrating at least one communication processing module. The wireless communication module receives electromagnetic waves through the antenna, frequency-modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110 . The wireless communication module can also receive the signal to be sent from the processor 110, frequency-modulate it, amplify it, and convert it into electromagnetic wave through the antenna to radiate out.
在一些实施例中，VR穿戴设备100的天线和移动通信模块耦合，使得VR穿戴设备100可以通过无线通信技术与网络以及其他设备通信。该无线通信技术可以包括全球移动通讯***(global system for mobile communications，GSM)，通用分组无线服务(general packet radio service，GPRS)，码分多址接入(code division multiple access，CDMA)，宽带码分多址(wideband code division multiple access，WCDMA)，时分码分多址(time-division code division multiple access，TD-SCDMA)，长期演进(long term evolution，LTE)，BT，GNSS，WLAN，NFC，FM，和/或IR技术等。GNSS可以包括全球卫星定位***(global  positioning system，GPS)，全球导航卫星***(global navigation satellite system，GLONASS)，北斗卫星导航***(beidou navigation satellite system，BDS)，准天顶卫星***(quasi-zenith satellite system，QZSS)和/或星基增强***(satellite based augmentation systems，SBAS)。In some embodiments, the antenna of the VR wearable device 100 is coupled to the mobile communication module, so that the VR wearable device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and/or IR technology, etc. GNSS can include global positioning system (global positioning system, GPS), global navigation satellite system (global navigation satellite system, GLONASS), Beidou satellite navigation system (beidou navigation satellite system, BDS), quasi-zenith satellite system (quasi-zenith) satellite system (QZSS) and/or satellite based augmentation systems (SBAS).
VR穿戴设备100通过GPU，光学显示模组1100，以及应用处理器等实现显示功能。GPU为图像处理的微处理器，连接光学显示模组1100和应用处理器。GPU用于执行数学和几何计算，用于图形渲染。处理器110可包括一个或多个GPU，其执行程序指令以生成或改变显示信息。The VR wearable device 100 realizes the display function through the GPU, the optical display module 1100 , and the application processor. The GPU is a microprocessor for image processing, and is connected to the optical display module 1100 and the application processor. GPUs are used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or change display information.
[根据细则91更正 29.09.2022]　
存储器120可以用于存储计算机可执行程序代码，该可执行程序代码包括指令。处理器110通过运行存储在存储器120的指令，从而执行VR穿戴设备100的各种功能应用以及数据处理。存储器120可以包括存储程序区和存储数据区。其中，存储程序区可存储操作***，至少一个功能所需的应用程序(比如声音播放功能，图像播放功能等)等。存储数据区可存储VR穿戴设备100使用过程中所创建的数据(比如音频数据，电话本等)等。此外，存储器120可以包括高速随机存取存储器，还可以包括非易失性存储器，例如至少一个磁盘存储器件，闪存器件，通用闪存存储器(universal flash storage，UFS)等。[Corrected 29.09.2022 under Rule 91]
The memory 120 may be used to store computer-executable program code, including instructions. The processor 110 executes various functional applications and data processing of the VR wearable device 100 by executing instructions stored in the memory 120 . The memory 120 may include an area for storing programs and an area for storing data. Wherein, the stored program area can store an operating system, at least one application program required by a function (such as a sound playing function, an image playing function, etc.) and the like. The storage data area can store data (such as audio data, phone book, etc.) created during the use of the VR wearable device 100 . In addition, the memory 120 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, flash memory device, universal flash storage (universal flash storage, UFS) and the like.
VR穿戴设备100可以通过音频模块，扬声器，麦克风140，耳机接口，以及应用处理器等实现音频功能。例如音乐播放，录音等。音频模块用于将数字音频信息转换成模拟音频信号输出，也用于将模拟音频输入转换为数字音频信号。音频模块还可以用于对音频信号编码和解码。在一些实施例中，音频模块可以设置于处理器110中，或将音频模块的部分功能模块设置于处理器110中。扬声器，也称“喇叭”，用于将音频电信号转换为声音信号。穿戴设备100可以通过扬声器收听音乐，或收听免提通话。The VR wearable device 100 can implement audio functions through an audio module, a speaker, a microphone 140, an earphone interface, and an application processor. Such as music playback, recording, etc. The audio module is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signal. The audio module can also be used to encode and decode audio signals. In some embodiments, the audio module may be set in the processor 110 , or some functional modules of the audio module may be set in the processor 110 . Loudspeakers, also called "horns", are used to convert audio electrical signals into sound signals. The wearable device 100 can listen to music through the speaker, or listen to hands-free calls.
麦克风140，也称“话筒”，“传声器”，用于将声音信号转换为电信号。VR穿戴设备100可以设置至少一个麦克风140。在另一些实施例中，VR穿戴设备100可以设置两个麦克风140，除了采集声音信号，还可以实现降噪功能。在另一些实施例中，VR穿戴设备100还可以设置三个，四个或更多麦克风140，实现采集声音信号，降噪，还可以识别声音来源，实现定向录音功能等。The microphone 140, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. The VR wearable device 100 may be provided with at least one microphone 140 . In other embodiments, the VR wearable device 100 can be provided with two microphones 140, which can also implement a noise reduction function in addition to collecting sound signals. In some other embodiments, the VR wearable device 100 can also be provided with three, four or more microphones 140 to collect sound signals, reduce noise, identify sound sources, and realize directional recording functions, etc.
耳机接口用于连接有线耳机。耳机接口可以是USB接口，也可以是3.5毫米(mm)的开放移动穿戴设备平台(open mobile terminal platform，OMTP)标准接口，美国蜂窝电信工业协会(cellular telecommunications industry association of the USA，CTIA)标准接口。The headphone jack is used to connect wired headphones. The headphone interface can be a USB interface, or a 3.5mm (mm) open mobile terminal platform (OMTP) standard interface, a cellular telecommunications industry association of the USA (CTIA) standard interface .
[根据细则91更正 29.09.2022]　
在一些实施例中，VR穿戴设备100可以包括一个或多个按键150，这些按键可以控制VR穿戴设备，为用户提供访问VR穿戴设备100上的功能。按键150的形式可以是按钮、开关、刻度盘和触摸或近触摸传感设备(如触摸传感器)。具体的，例如，用户可以通过按下按钮来打开VR穿戴设备100的光学显示模组1100。按键150包括开机键，音量键等。按键150可以是机械按键。也可以是触摸式按键。VR穿戴设备100可以接收按键输入，产生与VR穿戴设备100的用户设置以及功能控制有关的键信号输入。[Corrected 29.09.2022 under Rule 91]
In some embodiments, the VR wearable device 100 may include one or more buttons 150 , and these buttons may control the VR wearable device and provide users with access to functions on the VR wearable device 100 . Keys 150 may be in the form of buttons, switches, dials, and touch or near-touch sensing devices such as touch sensors. Specifically, for example, the user can turn on the optical display module 1100 of the VR wearable device 100 by pressing a button. The keys 150 include a power key, a volume key and the like. The key 150 may be a mechanical key. It can also be a touch button. The VR wearable device 100 can receive key input and generate key signal input related to user settings and function control of the VR wearable device 100 .
在一些实施例中，VR穿戴设备100可以包括输入输出接口160，输入输出接口160可以通过合适的组件将其他装置连接到VR穿戴设备100。组件例如可以包括音频/视频插孔，数据连接器等。In some embodiments, the VR wearable device 100 may include an input-output interface 160, and the input-output interface 160 may connect other devices to the VR wearable device 100 through suitable components. Components may include, for example, audio/video jacks, data connectors, and the like.
光学显示模组1100用于在处理器110的控制下，为用户呈现图像。光学显示模组1100可以通过反射镜、透射镜或光波导等中的一种或几种光学器件，将实像素图像显示转化为近眼投影的虚拟图像显示，实现虚拟的交互体验，或实现虚拟与现实相结合的交互体验。 例如，光学显示模组1100接收处理器110发送的图像数据信息，并向用户呈现对应的图像。The optical display module 1100 is used for presenting images to the user under the control of the processor 110 . The optical display module 1100 can convert the real pixel image display into a near-eye projection virtual image display through one or several optical devices such as mirrors, transmission mirrors, or optical waveguides, so as to realize virtual interactive experience, or realize virtual and Interactive experience combined with reality. For example, the optical display module 1100 receives image data information sent by the processor 110 and presents corresponding images to the user.
在一些实施例中，VR穿戴设备100还可以包括眼动跟踪模组1200，眼动跟踪模组1200用于跟踪人眼的运动，进而确定人眼的注视点。如，可以通过图像处理技术，定位瞳孔位置，获取瞳孔中心坐标，进而计算人的注视点。In some embodiments, the VR wearable device 100 may further include an eye tracking module 1200, which is used to track the movement of human eyes, and then determine the point of gaze of the human eyes. For example, the position of the pupil can be located by image processing technology, the coordinates of the center of the pupil can be obtained, and then the gaze point of the person can be calculated.
为了方便理解，以下以图9所示的VR穿戴设备是VR眼镜为例，对本申请实施例的显示方法的进行说明。For the convenience of understanding, the following describes the display method of the embodiment of the present application by taking the VR wearable device shown in FIG. 9 as VR glasses as an example.
实施例一Embodiment one
VR眼镜向用户展示图像，图像上不同对象的清晰度不同。比如，图像上远离用户注视点的对象(为了方便描述称为第一对象)模糊，靠近用户注视点的对象(为了方便描述称为第二对象)清晰。本实施例一可以适用于前文图7A和图7B所示的应用场景。VR glasses display images to users, and different objects on the images have different clarity. For example, an object far from the user's gaze point on the image (referred to as a first object for convenience of description) is blurred, and an object close to the user's gaze point (referred to as a second object for convenience of description) is clear. Embodiment 1 may be applicable to the application scenarios shown in FIG. 7A and FIG. 7B above.
示例性的，请参见图10所示，假设用户注视点在房子，而山所在景深与房子所在景深之间的距离L1小于树所在景深与房子所在景深之间的距离L2，所以山的清晰度高于树的清晰度。也就是，用户注视点在房子时，距离房子较近的山清晰，距离房子较远的树模糊。其中，清晰度可以包括图像分辨率或扫描分辨率等，在以下实施例中，清晰度以所显示图像的分辨率为例进行说明。As an example, please refer to Figure 10, assuming that the user’s focus is on the house, and the distance L1 between the depth of field where the mountain is located and the depth of field where the house is located is smaller than the distance L2 between the depth of field where the tree is located and the depth of field where the house is located, so the clarity of the mountain Higher than tree clarity. That is, when the user looks at the house, the mountains closer to the house are clear, and the trees farther away from the house are blurred. Wherein, the sharpness may include image resolution or scanning resolution, etc. In the following embodiments, the sharpness is described by taking the resolution of a displayed image as an example.
示例性的，图11为实施例一提供的图像生成方法的流程示意图。如图11，所述流程包括：Exemplarily, FIG. 11 is a schematic flowchart of an image generation method provided in Embodiment 1. As shown in Figure 11, the process includes:
S1101，确定用户注视点。S1101. Determine a gaze point of a user.
其中，用户注视点的确定过程前文已描述过，不重复赘述。Wherein, the process of determining the user's gaze point has been described above, and will not be repeated here.
S1102，确定待绘制的不同对象的深度。S1102. Determine the depths of different objects to be drawn.
其中，对于各物体的深度，可在渲染管线运行时自动保存，也可依赖双目视觉进行计算，本申请实施例不作限定。Wherein, the depth of each object can be automatically saved when the rendering pipeline is running, or can be calculated by relying on binocular vision, which is not limited in the embodiment of the present application.
S1103，根据用户注视点与所述对象之间的距离，确定所述对象的模糊程度。S1103. Determine the blurring degree of the object according to the distance between the user's gaze point and the object.
示例性的，当对象所在景深与用户注视点所在景深之间的距离小于预设距离时，可以不对该对象进行模糊化处理；当对象所在景深与用户注视点所在景深之间的距离大于预设距离时，对该对象进行模糊处理。其中，预设距离的具体取值，本申请实施例不作限定。Exemplarily, when the distance between the depth of field where the object is located and the depth of field where the user's gaze is located is less than the preset distance, the object may not be blurred; when the distance between the depth of field where the object is located and the depth of field where the user's gaze is located is greater than the preset distance Blur the object at a distance. Wherein, the specific value of the preset distance is not limited in this embodiment of the present application.
或者，图像上不同对象的模糊化程度可以按照该对象所在景深与用户注视点所在景深之间的距离从小到大依次增加。比如，假设用户注视点所在景深为景深1。对象1所在景深与景深1之间的距离为距离1，对象2所在景深与景深1之间的距离为距离2，对象3所在景深与景深1之间的距离为距离3。如果距离1<距离2<距离3，那么对象1的模糊程度<对象2的模糊程度<对象3的模糊程度，这样，对象1的清晰度>对象2的清晰度>对象3的清晰度。也就是说，用户眼睛所看到的虚拟环境中，距离用户注视点越远的对象越模糊，距离用户注视点越近的对象越清晰。Alternatively, the degree of blurring of different objects on the image may increase in order according to the distance between the depth of field where the object is located and the depth of field where the user's gaze point is located. For example, suppose the depth of field where the user gazes at is depth 1. The distance between the depth of field where object 1 is located and depth of field 1 is distance 1, the distance between the depth of field where object 2 is and depth 1 is distance 2, and the distance between the depth of field where object 3 is and depth 1 is distance 3. If distance 1<distance 2<distance 3, then the degree of blur of object 1<the degree of blur of object 2<the degree of blur of object 3, so that the sharpness of object 1>the sharpness of object 2>the sharpness of object 3. That is to say, in the virtual environment seen by the user's eyes, objects farther from the user's gaze point are more blurred, and objects closer to the user's gaze point are clearer.
S1104，生成图像，并对图像上所述对象做模糊化处理。S1104. Generate an image, and perform blurring processing on the object on the image.
一种可能的实施方式为，VR设备可以先生成图像，该图像上所有对象的清晰度相同，然后利用图像模糊化算法对图像上的不同对象做不同程度的模糊化处理。其中，图像模糊化算法包括高斯模糊、图像下采样、基于深度学习的离焦模糊(defocus blur)算法、细节层次(level of detail，LOD)数据结构等中的至少一种，本申请不多赘述。A possible implementation manner is that the VR device may generate an image first, and all objects on the image have the same definition, and then use an image blurring algorithm to perform blurring processing to different degrees on different objects on the image. Wherein, the image blurring algorithm includes at least one of Gaussian blur, image down-sampling, defocus blur (defocus blur) algorithm based on deep learning, level of detail (level of detail, LOD) data structure, etc., and this application will not go into details .
可以理解的是，VR眼镜显示图像时，用户注视点可能发生变化。当用户注视点发生变化时，图像上对象的清晰度相应的调整。比如，继续以图10为例，当用户注视点由房子变为树时，基于新的注视点(即树)重新确定各个对象的清晰度，即距离新的注视点近的对象清晰，距离新的注视点远的对象模糊。It is understandable that when the VR glasses display images, the user's gaze point may change. As the user's gaze point changes, the sharpness of objects on the image adjusts accordingly. For example, continue to take Figure 10 as an example. When the user's gaze point changes from a house to a tree, the clarity of each object is re-determined based on the new gaze point (ie, the tree). Objects far away from the point of view are blurred.
图10以VR眼镜显示一帧图像为例进行介绍。可以理解的是，一般情况下，VR眼镜显示图像流，图像流包括多帧图像。Figure 10 uses VR glasses to display a frame of image as an example. It can be understood that, generally, the VR glasses display an image stream, and the image stream includes multiple frames of images.
第一种方式，VR图像生成设备(比如图2中的图像生成设备200)默认使用图11所示的流程生成的图像流(即，每帧图像上远离用户注视点的对象作模糊化处理)。比如，以VR图像生成设备是手机为例，手机在检测到连接VR眼镜、VR眼镜开机、VR应用(如VR游戏)启动中的至少一种时，默认开始使用图11所示的流程生成图像流，然后通过VR眼镜进行显示。In the first way, the VR image generation device (such as the image generation device 200 in FIG. 2 ) uses the image stream generated by the process shown in FIG. 11 by default (that is, objects far away from the user’s gaze point on each frame of image are blurred) . For example, taking the VR image generation device as a mobile phone as an example, when the mobile phone detects at least one of the connection of the VR glasses, the startup of the VR glasses, and the startup of the VR application (such as a VR game), the mobile phone starts to use the process shown in Figure 11 to generate images by default. stream, and then display it through VR glasses.
第二种方式，VR图像生成设备默认使用现有方式生成图像(即，图像上所有对象清晰度相同)，当检测到用于启动第一护眼模式的指示时，使用图11所示的流程生成图像。其中，第一护眼模式请参见前文描述。也就是说，VR眼镜刚开始显示的图像上所有对象的清晰度相同，在检测到用于启动第一护眼模式的指示后，显示的图像上远离用户注视点的对象模糊。示例性的，请参见图12，在第i+1帧之前，图像上所有对象的清晰度相同。当VR眼镜检测用于启动第一护眼种模式的指示时，第i+1帧至第N帧图像上远离用户注视点的对象(如树)模糊化。其中，用于启动第一护眼模式的指示包括但不限定于：检测到用户触发用于启动第一护眼模式的操作(如VR应用中包括用于启动第一护眼模式的按钮，所述操作可以是点击所述按钮的操作)、用户观看时间大于预设时长、预设时长内用户眼睛眨眼/眯眼次数大于预设次数中的至少一种。以预设时长内用户眼睛眨眼/眯眼次数大于预设次数为例，如果用户眼睛有不适感(如疲劳感)，可能会不停眨眼或眯眼以缓解这种不适感，所以当检测到用户眨眼/眯眼次数较多时，启动第一护眼模式，以缓解用户疲劳感。In the second way, the VR image generation device uses the existing way to generate images by default (that is, all objects on the image have the same clarity), and when the instruction for starting the first eye protection mode is detected, the process shown in Figure 11 is used Generate an image. For the first eye protection mode, please refer to the previous description. That is to say, all objects on the image displayed by the VR glasses at the beginning have the same clarity, and after the instruction for starting the first eye protection mode is detected, objects far away from the user's gaze point on the displayed image are blurred. For example, please refer to FIG. 12 , before frame i+1, all objects on the image have the same definition. When the VR glasses detect the indication for activating the first eye protection mode, objects (such as trees) far away from the user's gaze point on the i+1th to Nth frame images are blurred. Wherein, the instruction for starting the first eye protection mode includes but is not limited to: detecting that the user triggers an operation for starting the first eye protection mode (for example, a VR application includes a button for starting the first eye protection mode, so The above operation may be an operation of clicking the button), the user's viewing time is greater than the preset time period, and the number of times the user blinks/squints within the preset time period is greater than the preset number of times. Take the user's eyes blinking/squinting times greater than the preset number of times within the preset time period as an example, if the user's eyes feel uncomfortable (such as fatigue), they may keep blinking or squinting to relieve the discomfort, so when detected When the user blinks/squints more times, the first eye protection mode is activated to relieve user fatigue.
由于启动第一护眼模式之后，图像上远离用户注视点的对象(如树)模糊化，所以人脑疲劳感得到缓解，用户体验较好。可选的，在启动第一护眼模式之前，还可以输出提示信息，该提示信息用于提示用户是否切换到第一护眼模式，在检测到用户确认切换到第一护眼模式的指示后，切换到第一护眼模式。Since the first eye protection mode is activated, objects (such as trees) far away from the user's gaze point on the image are blurred, so the fatigue of the human brain is relieved, and the user experience is better. Optionally, before starting the first eye protection mode, a prompt message may also be output, the prompt message is used to prompt the user whether to switch to the first eye protection mode, after the user confirms the indication to switch to the first eye protection mode is detected , switch to the first eye protection mode.
上面的第一种方式和第二种方式，VR图像生成设备生成的图像流中远离用户注视点的对象作模糊化处理，可以缓解人脑疲劳，但是容易丢失远离用户注视点的对象的细节。比如，第一种方式中，远离用户注视点的对象始终是模糊的，用户无法获得该对象的细节；第二种方式中，在检测到用于启动第一护眼模式的指示后，远离用户注视点的对象将一直是模糊的，用户也无法获得该对象的细节。In the first and second methods above, objects far away from the user's gaze point in the image stream generated by the VR image generation device are blurred, which can relieve human brain fatigue, but it is easy to lose the details of objects far away from the user's gaze point. For example, in the first way, the object far away from the user's gaze point is always blurred, and the user cannot obtain the details of the object; The object at the foveated point will always be blurred, and the user will not be able to get the details of the object.
为了既能够缓解疲劳感，又能获取远离用户注视点的对象的细节。上述第一种方式或第二种方式中VR图像生成设备生成的图像流中远离用户注视点的对象的清晰度可以有高有低(具体生成过程请参见后文图15)。比如，图像流包括多个周期，每个周期内包括多帧图像，且每个周期内图像上远离用户注视点的对象的清晰度先升后降。In order to not only relieve fatigue, but also obtain details of objects far away from the user's gaze point. In the first mode or the second mode above, the definition of objects far away from the user's gaze point in the image stream generated by the VR image generation device may be high or low (see Figure 15 below for the specific generation process). For example, the image stream includes multiple cycles, and each cycle includes multiple frames of images, and in each cycle, the sharpness of objects far away from the user's gaze point on the image increases first and then decreases.
示例性的，请参见图13，一个周期内，第i帧图像上树(远离用户注视点的对象)的清晰度低于第j帧图像上的树的清晰度，第j帧图像上的树的清晰度高于第k帧图像上树 的清晰度。即，一个周期内远离用户注视点的对象(即树)的清晰度呈“模糊-清晰-模糊”的变化趋势。下一个周期内，第k帧图像上树的清晰度低于第p帧图像上树的清晰度，第p帧图像上树的清晰度高于第q帧图像上树的清晰度，即下一个周期内，远离用户注视点的对象(即树)的清晰度也是呈“模糊-清晰-模糊”的变化趋势。图13中的两个周期可以相同或不同，不作限定。这种清晰度变化趋势一方面可以缓解人脑疲劳感，另一方面可以避免用户丢失远离用户注视点的对象的图像细节。For example, please refer to Figure 13. In one period, the resolution of the tree (object far away from the user's gaze point) on the image frame i is lower than that of the tree on the image frame j, and the tree on the image frame j The sharpness of is higher than the sharpness of the tree on the kth frame image. That is, the sharpness of an object far away from the user's gaze point (that is, a tree) within a cycle shows a change trend of "fuzzy-clear-fuzzy". In the next cycle, the resolution of the tree on the image of the kth frame is lower than that of the tree on the image of the pth frame, and the resolution of the tree on the image of the pth frame is higher than that of the tree on the image of the qth frame, that is, the next During the cycle, the sharpness of objects far from the user's gaze point (that is, trees) also shows a "fuzzy-clear-fuzzy" change trend. The two periods in FIG. 13 may be the same or different, without limitation. On the one hand, this sharpness change trend can alleviate the fatigue of the human brain, and on the other hand, it can prevent the user from losing image details of objects far away from the user's gaze point.
示例性的，继续参见图13，i、j、k、p满足：j＝i+n，k＝j+m，p＝k+w，q＝p+s，第j帧是第i帧之后的n帧，第k帧是第j帧之后的m帧，第p帧是第k帧之后的w帧，第q帧是第p帧之后的s帧。其中，n、m、q、s为大于或等于1的整数。Exemplarily, continue to refer to FIG. 13, i, j, k, p satisfy: j=i+n, k=j+m, p=k+w, q=p+s, and the jth frame is after the ith frame The n frame, the kth frame is the m frame after the jth frame, the pth frame is the w frame after the kth frame, and the qth frame is the s frame after the pth frame. Wherein, n, m, q, s are integers greater than or equal to 1.
一种示例为，j＝i+1，k＝j+1，p＝k+1，q＝p+1，即，第j帧图像是第i帧图像的下一帧，第k帧图像是第j帧图像的下一帧，第p帧是第k帧的下一帧，第q帧是第p帧的下一帧。One example is, j=i+1, k=j+1, p=k+1, q=p+1, that is, the jth frame image is the next frame of the i frame image, and the kth frame image is The next frame of the j-th frame image, the p-th frame is the next frame of the k-th frame, and the q-th frame is the next frame of the p-th frame.
另一种示例为，n、m、p、s可以根据用户视觉停留时间、图像刷新帧率确定。假设用户视觉停留时间为T，图像刷新帧率为P。其中，视觉停留时间T可以是0.1s至3s范围内的任一值，或者，可以是用户设置的，本申请实施例不作限定。那么，在T时间内，可以显示T/P帧图像，那么n小于或等于T/P，m小于或等于T/P，q小于或等于T/P，s小于或等于T/P。换句话说，第j帧图像的显示时刻与第i帧图像的显示时刻之间的时间差小于用户视觉停留时间，这样的话，人脑中可以将第j帧图像和第i帧图像上的图像信息叠加，由于第i帧图像上远离用户注视点的对象是模糊的，第j帧图像上远离用户注视点的对象是清晰的，两张图像的叠加，可以保证人脑中远离用户注视点的对象的细节足够。同理，第j帧图像的显示时刻与第k帧图像的显示时刻之间的时间差小于用户视觉停留时间，不重复赘述。为了方便理解，以下远离用户注视点的对象是灯塔为例，给出一个具体示例。请参见图14，第i帧图像上的远离用户注视点的对象(即灯塔)模糊，第j帧图像上灯塔清晰，用户眼睛观看第i帧图像和第j帧图像时，因为视觉停留效应，人脑中可以形成的灯塔的清晰度比第i帧图像上灯塔的清晰度高，所以不会因为第i帧图像上灯塔模糊而导致细节丢失。Another example is that n, m, p, and s can be determined according to the user's visual dwell time and image refresh frame rate. Assume that the user's visual dwell time is T, and the image refresh frame rate is P. Wherein, the visual dwell time T may be any value within the range of 0.1s to 3s, or may be set by the user, which is not limited in this embodiment of the present application. Then, within T time, T/P frame images can be displayed, then n is less than or equal to T/P, m is less than or equal to T/P, q is less than or equal to T/P, and s is less than or equal to T/P. In other words, the time difference between the display moment of the j-th frame image and the display moment of the i-th frame image is less than the user's visual dwell time. In this way, the image information on the j-th frame image and the i-frame image can be combined Superposition, since the object far away from the user's gaze point on the i-th frame image is blurred, the j-th frame image's object far away from the user's gaze point is clear, and the superposition of the two images can ensure the accuracy of the object far away from the user's gaze point The details are sufficient. Similarly, the time difference between the display moment of the jth frame of image and the display moment of the kth frame of image is less than the user's visual dwell time, and will not be repeated here. To facilitate understanding, the following object is far from the user's gaze point is a lighthouse as an example to give a specific example. Please refer to Figure 14, the object (i.e. the lighthouse) far away from the user's gaze point on the image frame i is blurred, and the lighthouse on the image frame j is clear. When the user's eyes watch the image frame i and frame j, because of the visual dwell effect The clarity of the lighthouse that can be formed in the human brain is higher than that of the lighthouse on the i-th frame image, so details will not be lost due to the blurring of the lighthouse on the i-th frame image.
可选的，继续以图13为例，图像流中靠近用户注视点的对象的清晰度可以不变。比如，图13中，靠近用户注视点的对象(即山)的清晰度保持不变。Optionally, continuing to take FIG. 13 as an example, the sharpness of objects close to the user's gaze point in the image stream may not change. For example, in FIG. 13 , the sharpness of the object (that is, the mountain) close to the user's gaze point remains unchanged.
示例性的，以下介绍图13所示的图像流的生成过程。Exemplarily, the following describes the process of generating the image stream shown in FIG. 13 .
GPU输出N帧图像(为了方便描述，将GPU输出的图像称为原图)，GPU输出的N帧图像上所有对象的清晰度是相同的，通过N帧原图生成第一对象(即远离用户注视点的对象，如图13中的树)的清晰度交替变化的N帧新图。下文以通过第i帧原图、第j帧原图、第k帧原图和第p帧原图(比如，j＝i+1，k＝j+1，p＝k+1)为例，介绍生成第i帧新图、第j帧新图、第k帧新图和第p帧新图的过程。The GPU outputs N frames of images (for the convenience of description, the images output by the GPU are referred to as original images), and the definition of all objects on the N frames of images output by the GPU is the same, and the first object (that is, far away from the user The objects at the fixation point, such as the tree in Fig. 13), are N frames of new pictures whose resolutions change alternately. The following takes the original image of the i-th frame, the original image of the j-th frame, the original image of the k-th frame, and the original image of the p-th frame (for example, j=i+1, k=j+1, p=k+1) as an example, Introduce the process of generating a new image of the i-th frame, a new image of the j-th frame, a new image of the k-th frame, and a new image of the p-th frame.
简单来说，第i帧新图是对第i帧原图上第一对象做模糊化处理后的图像；第j帧新图是第j帧原图或对第j帧原图上第一对象作清晰化处理后的图像；第k帧新图是对第k帧原图上第一对象做模糊化处理后的图像；第p帧新图是第p帧原图或对第p帧原图上第一对象作清晰化处理后的图像。To put it simply, the new image of frame i is the blurred image of the first object on the original image of frame i; the new image of frame j is the original image of frame j or the first object of the original image of frame j The image after clearing processing; the new image of the kth frame is the image after blurring the first object on the original image of the kth frame; the new image of the pth frame is the original image of the pth frame or the original image of the pth frame The image after the sharpening process of the first object.
示例性的，如图15，对GPU输出的第i帧原图上的第一对象做模糊化处理，得到第i帧新图。为了避免第一对象的细节丢失，第j帧新图上可以包括第一对象的更多细节。一 种可行方式为，将第i帧新图和GPU输出的第j帧原图叠加(或者称为融合)，得到第j帧新图，所以，第j帧新图上第一对象的清晰度高于第j帧原图以及第i帧新图。另一种可行方式为，将对第i帧原图作模糊化处理时丢失的图像信息与GPU输出的第j帧原图叠加，得到第j帧新图。如果j＝i+1，即将上一帧图像作模糊化处理所丢失的图像信息补偿到下一帧图像中。可选的，为了提升效率，在将第i帧新图和第j帧原图叠加时，可以只将第i帧新图上第一对象所在的区域内的图像块与第j帧原图上第一对象所在区域内的图像块叠加。Exemplarily, as shown in FIG. 15 , the first object in the i-th frame of the original image output by the GPU is blurred to obtain the i-th frame of the new image. In order to avoid the loss of details of the first object, more details of the first object may be included in the new image of the jth frame. One feasible way is to superimpose (or fuse) the new image of frame i and the original image of frame j output by the GPU to obtain the new image of frame j. Therefore, the definition of the first object on the new image of frame j It is higher than the original image of the jth frame and the new image of the ith frame. Another feasible way is to superimpose the image information lost when the i-th frame of the original image is blurred with the j-th frame of the original image output by the GPU to obtain the j-th frame of the new image. If j=i+1, the image information lost by blurring the previous frame image is compensated to the next frame image. Optionally, in order to improve efficiency, when superimposing the new image of the i-th frame and the original image of the j-th frame, only the image block in the area where the first object is located on the new image of the i-th frame can be combined with the original image of the j-th frame The image blocks in the region where the first object is located are superimposed.
类似的，请继续参见图15，对GPU输出的第k帧图像上第一对象做模糊化处理，得到第k帧新图。为了避免第一对象的细节丢失，将第k帧新图与GPU输出的第p帧原图叠加，得到第p帧新图，或者，将对第k原图作模糊化处理时所丢失的图像信息与第p帧原图叠加得到第p帧新图。可选的，为了提升效率，在将第k帧新图与第p帧原图叠加时，可以只需将第k帧新图上第一对象所在区域内的图像块与第p帧原图上第一对象所在区域内的图像块叠加。Similarly, please continue to refer to FIG. 15 to perform blurring processing on the first object on the image of the kth frame output by the GPU to obtain a new image of the kth frame. In order to avoid the loss of the details of the first object, superimpose the kth frame of the new image with the pth frame of the original image output by the GPU to obtain the pth frame of the new image, or the image lost when the kth original image is blurred The information is superimposed with the original image of frame p to obtain a new image of frame p. Optionally, in order to improve efficiency, when superimposing the new image of the kth frame with the original image of the pth frame, it is only necessary to combine the image block in the area where the first object is located on the new image of the kth frame with the original image of the pth frame The image blocks in the region where the first object is located are superimposed.
继续以图15为例，且以将第i帧新图上模糊区域(即第一对象所在区域)与第j帧原图上第一对象所在区域叠加为例，可以理解的是，存在一种情况，第i帧新图上模糊区域与第j帧原图上第一对象所在区域的位置可能发生变化。这种情况下，可以根据两帧图像间的光流(optical flow)确定像素点之间的对应关系，根据第i帧新图上模糊区域以及所述对应关系，确定第j帧原图上对应的区域，将所述第i帧新图上模糊区域与所述确定出的第j帧原图上的区域叠加，得到第j帧新图。Continuing to take Figure 15 as an example, and taking the example of superimposing the blurred area (that is, the area where the first object is located) on the new image of frame i with the area where the first object is located on the original image of frame j, it can be understood that there is a In this case, the positions of the blurred area on the new image of frame i and the area where the first object is located on the original image of frame j may change. In this case, the corresponding relationship between pixels can be determined according to the optical flow between the two frames of images, and the corresponding relationship between the pixels on the original image of the jth frame can be determined according to the blurred area on the new image of the i frame and the corresponding relationship. , and superimpose the blurred area on the i-th frame of the new image with the determined area on the j-th frame of the original image to obtain the j-th frame of the new image.
实施例二Embodiment two
VR眼镜向用户展示图像，图像上不同对象的清晰度不同。比如，图像上景深较大的对象(为了方便描述称为第一对象)模糊，景深较小的对象(为了方便描述称为第二对象)清晰。本实施例二可以适用于前文图8A和图8B所示的应用场景。VR glasses display images to users, and different objects on the images have different clarity. For example, an object with a larger depth of field (called a first object for convenience of description) on an image is blurred, and an object with a smaller depth of field (called a second object for convenience of description) is clear. Embodiment 2 may be applicable to the application scenarios shown in FIG. 8A and FIG. 8B above.
示例性的，请参见图16所示，山所在的第二景深大于树所在的第一景深，所以山模糊，树清晰，这样用户看到的虚拟环境中，山是模糊的，树是清楚的。For example, please refer to Figure 16. The second depth of field where the mountain is located is greater than the first depth of field where the tree is located, so the mountain is blurred and the tree is clear. In this way, in the virtual environment that the user sees, the mountain is blurred and the tree is clear. .
示例性的，请参见图17，为本实施例二提供的图像生成方法的流程示意图，如图17所示，该方法的流程包括：For example, please refer to FIG. 17, which is a schematic flow diagram of the image generation method provided in the second embodiment. As shown in FIG. 17, the flow of the method includes:
S1701，确定预设景深。S1701. Determine a preset depth of field.
预设景深可以是一个具体的景深值，也可以是一个景深范围，本申请实施例不作限定。预设景深用于判断哪些对象是远处对象，哪些对象是近处对象。比如景深大于预设景深的对象是远处对象，景深小于预设景深的对象是近处对象。比如，对于远处对象可以模糊化，对于近景对象可以不作模糊化处理。其中，预设景深有多种方式确定，包括但不限定于如下方式中的至少一种。The preset depth of field may be a specific depth of field value or a depth of field range, which is not limited in this embodiment of the present application. The preset depth of field is used to judge which objects are distant objects and which objects are near objects. For example, an object whose depth of field is greater than the preset depth of field is a distant object, and an object whose depth of field is smaller than the preset depth of field is a near object. For example, distant objects may be blurred, but close-range objects may not be blurred. There are multiple ways to determine the preset depth of field, including but not limited to at least one of the following ways.
方式一，预设景深可以是根据VR场景确定，预设景深随VR场景的不同而不同。其中，所述VR场景包括但不限定于VR游戏、VR观影、VR教学等中的至少一种。In a first manner, the preset depth of field may be determined according to a VR scene, and the preset depth of field varies with different VR scenes. Wherein, the VR scene includes but is not limited to at least one of VR games, VR viewing, VR teaching and the like.
以VR游戏为例，VR游戏中包括游戏人物，可以根据所述游戏人物确定预设景深。比如，如果是第一人称游戏，预设景深可以是游戏场景中用户对应的游戏角色所在景深，或者，用户对应的游戏角色上身体部位所在景深，或者，用户对应的游戏角色当前所持游 戏装备所在景深。以图7A为例，游戏人物的手臂握持***，可以确定手臂或***所在景深为预设景深。再比如，如果是第三人称游戏(比如，观战场景)，可以根据控制游戏的游戏人物(比如被观战的游戏人物)所在景深为预设景深。以VR观影为例，VR观影中包括显示屏，可以确定显示屏所在景深为预设景深。以VR教学为例，VR教学中包括黑板、显示屏、投影等等教学设备，可以确定所述教学设备所在景深为预设景深。Taking a VR game as an example, the VR game includes game characters, and the preset depth of field can be determined according to the game characters. For example, if it is a first-person game, the preset depth of field can be the depth of field of the game character corresponding to the user in the game scene, or the depth of field of the body parts of the game character corresponding to the user, or the depth of field of the game equipment currently held by the game character corresponding to the user . Taking FIG. 7A as an example, the game character's arm is holding a gun, and the depth of field where the arm or gun is located can be determined as the preset depth of field. For another example, if it is a third-person game (for example, a spectator scene), the depth of field of the game character controlling the game (for example, the game character being watched) can be used as the preset depth of field. Taking VR viewing as an example, VR viewing includes a display screen, and the depth of field where the display screen is located can be determined as the preset depth of field. Taking VR teaching as an example, VR teaching includes teaching equipment such as blackboards, display screens, and projections, and the depth of field where the teaching equipment is located can be determined as the preset depth of field.
方式二，预设景深可以是用户设置的，比如用户可以在VR眼镜或与VR眼镜连接的电子设备(比如手机)上设置预设景深。应理解，所述电子设备上包括各种VR应用，不同的VR应用可以设置不同的预设景深。可选的，用户可以批量设置电子设备上的VR应用的预设景深，或者，可以针对每个VR应用进行单独设置。In the second way, the preset depth of field can be set by the user. For example, the user can set the preset depth of field on the VR glasses or an electronic device (such as a mobile phone) connected to the VR glasses. It should be understood that the electronic device includes various VR applications, and different preset depths of field may be set for different VR applications. Optionally, the user can set the preset depth of field of the VR applications on the electronic device in batches, or can set individually for each VR application.
方式三，预设景深还可以是默认景深，所述默认景深可以理解为VR眼镜默认设置的，或与VR眼镜连接的电子设备(比如手机)默认设置的，或与VR眼镜连接的电子设备(比如手机)上当前运行的VR应用默认设置的，等等，本申请实施例不作限定。Mode 3, the preset depth of field can also be the default depth of field, which can be understood as the default setting of the VR glasses, or the default setting of the electronic device (such as a mobile phone) connected to the VR glasses, or the electronic device connected to the VR glasses ( For example, the VR application currently running on the mobile phone) is set by default, etc., which are not limited in this embodiment of the present application.
方式四，预设景深还可以是虚像面所在景深。以图6C为例，虚线面所在景深为深度1，那么预设景深可以是深度1。Mode 4, the preset depth of field may also be the depth of field where the virtual image plane is located. Taking FIG. 6C as an example, the depth of field where the dotted line surface is located is depth 1, so the preset depth of field may be depth 1.
方式五，预设景深还可以根据VR眼镜当前正显示的画面中主体对象所在景深。所述主体对象可以包括在所述画面中占最大面积的对象、或位于所述画面中居中位置的对象、或是所述画面中的虚拟对象(比如UI界面)等等。示例性的，以16为例，图像中包括树、房子和太阳，假设房子的位置居中，那么确定房子为主体对象，预设景深可以是房子所在景深。由于山的景深大于房子所在景深，所以山模糊，树所在景深小于房子所在景深，所以树清晰。In the fifth way, the preset depth of field can also be based on the depth of field of the main object in the picture currently being displayed by the VR glasses. The main object may include an object occupying the largest area in the screen, an object located in the center of the screen, or a virtual object (such as a UI interface) in the screen, and the like. Exemplarily, taking 16 as an example, the image includes a tree, a house, and the sun. Assuming that the house is in the center, then the house is determined to be the main object, and the preset depth of field may be the depth of field where the house is located. Because the depth of field of the mountain is greater than that of the house, the mountain is blurred, and the depth of field of the tree is smaller than that of the house, so the tree is clear.
方式六、预设景深是用户注视点所在景深。VR眼镜中可以包括眼动追踪模块，通过眼动追踪模块可以确定用户注视点，确定用户注视点所在景深为预设景深。示例性的，继续以16为例，图像中包括树、房子和太阳，假设用户注视点在房子，那么预设景深可以是房子所在景深。山的景深大于房子所在景深，所以山模糊，树所在景深小于房子所在景深，所以树清晰。Method 6: The preset depth of field is the depth of field where the user gazes. The VR glasses may include an eye tracking module, through which the user's gaze point can be determined, and the depth of field at which the user's gaze point is determined is the preset depth of field. Exemplarily, continuing to take 16 as an example, the image includes trees, houses and the sun, assuming that the user's focus is on the house, then the preset depth of field may be the depth of field where the house is located. The depth of field of the mountain is greater than that of the house, so the mountain is blurred, and the depth of field of the tree is smaller than that of the house, so the tree is clear.
以上是预设景深的几种确定方式，对于其他的方式也是可行的，本申请实施例不作限定。The above are several methods for determining the preset depth of field, and other methods are also possible, which are not limited in this embodiment of the present application.
S1702，确定待绘制的不同对象的深度。S1702. Determine the depths of different objects to be drawn.
其中，对于各物体的深度，可在渲染管线运行时自动保存，也可依赖双目视觉进行计算，本申请实施例不作限定。Wherein, the depth of each object can be automatically saved when the rendering pipeline is running, or can be calculated by relying on binocular vision, which is not limited in the embodiment of the present application.
S1703，根据所述对象的深度与预设景深之间的距离，确定所述对象的模糊程度。S1703. Determine the blurring degree of the object according to the distance between the depth of the object and a preset depth of field.
示例性的，当对象的深度小于等于预设深度时，可以不对该对象进行模糊化处理；当对象的深度大于预设深度时，则需对该对象进行模糊处理。Exemplarily, when the depth of the object is less than or equal to the preset depth, the object may not be blurred; when the depth of the object is greater than the preset depth, the object needs to be blurred.
或者，图像上不同对象的模糊化程度可以按照景深从小到大依次增加。比如对象1所在的景深1<对象2所在的景深2<对象3所在的景深3，那么，对象1的模糊化程度<对象2的模糊化程度<对象3的模糊化程度。这样，对象1的清晰度>对象2的清晰度>对象3的清晰度。也就是说。用户眼睛所看到的虚拟环境中，近景对象、中景对象、远景对象之间的清晰度依次降低。Alternatively, the blurring degrees of different objects on the image may increase sequentially from small to large depth of field. For example, the depth of field 1 where the object 1 is located < the depth of field 2 where the object 2 is located < the depth of field 3 where the object 3 is located, then the degree of blurring of the object 1 < the degree of blurring of the object 2 < the degree of blurring of the object 3 . In this way, the sharpness of object 1 > the sharpness of object 2 > the sharpness of object 3. That is. In the virtual environment seen by the user's eyes, the sharpness among objects in the foreground, objects in the middle, and objects in the foreground decreases in turn.
S1704，生成图像，并对图像上所述对象做模糊化处理。S1704. Generate an image, and perform blurring processing on the object on the image.
一种可能的实施方式为，VR设备可以先生成图像，然后利用图像模糊化算法对图像 上不同对象做不同程度的模糊化处理。其中，图像模糊化算法包括高斯模糊、图像下采样、基于深度学习的离焦模糊(defocus blur)算法、细节层次(level of detail，LOD)数据结构等中的至少一种，本申请实施例不作限定。One possible implementation is that the VR device can first generate an image, and then use an image blurring algorithm to blur different objects on the image to different degrees. Wherein, the image blurring algorithm includes at least one of Gaussian blur, image down-sampling, defocus blur (defocus blur) algorithm based on deep learning, level of detail (LOD) data structure, etc. limited.
下文以LOD为例进行介绍。The following uses LOD as an example to introduce.
先对LOD作简单介绍。LOD是多层数据结构，数据结构可以理解为图像处理算法，多层数据结构即包括多层图像处理算法。假设包括LOD0至LOD3；其中，LOD0至LOD3中每一层对应一种图像处理算法。而且，LOD0至LOD3中不同层对应的算法不同，具体地，层级越高，对应的图像处理算法越简单。比如，LOD3层级最高，对应的图像处理算法最简单；LOD0层级最低，对应的图像处理算法最复杂。First, a brief introduction to LOD. LOD is a multi-layer data structure. The data structure can be understood as an image processing algorithm, and the multi-layer data structure includes a multi-layer image processing algorithm. It is assumed that LOD0 to LOD3 are included; wherein, each layer in LOD0 to LOD3 corresponds to an image processing algorithm. Moreover, different layers in LOD0 to LOD3 correspond to different algorithms, specifically, the higher the layer, the simpler the corresponding image processing algorithm. For example, LOD3 has the highest level, and the corresponding image processing algorithm is the simplest; LOD0 has the lowest level, and the corresponding image processing algorithm is the most complex.
LOD可以用于生成三维图像。继续以LOD0至LOD3为例；其中，LOD0至LOD3中每一层可用于生成三维图像中的一个图层，然后不同图层用于合成得到三维图像；其中，不同图层对应的深度范围不同。举例来说，可以根据LOD层级数对图像深度进行划分，比如，LOD0至LOD3四个层，将图像深度划分为四个范围。比如，LOD0对应深度范围1，即LOD0对应的图像处理算法用于对深度范围1内的图层作处理；LOD1对应深度范围2，即，LOD1对应的图像处理算法用于对深度范围2内的图层作处理；LOD2对应深度范围3，即，LOD2对应的图像处理算法用于对深度范围3内的图层作处理；LOD3对应深度范围4，即，LOD3对应的图像处理算法用于对深度范围4内的图层作处理。因为本申请考虑深度越大的对象越模糊。所以，深度范围大的图层(远处的图层)需要使用算法简单的LOD层处理，因为算法越简单的LOD层处理后的图层越模糊。所以深度范围较大的图层对应层级较高的LOD层(LOD层级越高算法越简单，参见前文描述)，同理，深度范围越小的图层对应层级越低的LOD层，因为层级越低的LOD层对应的算法越复杂，处理后的图层越清晰。LODs can be used to generate 3D images. Continue to take LOD0 to LOD3 as an example; wherein, each layer in LOD0 to LOD3 can be used to generate a layer in a 3D image, and then different layers are used to synthesize a 3D image; wherein, different layers correspond to different depth ranges. For example, the image depth can be divided according to the number of LOD levels, for example, there are four layers LOD0 to LOD3, and the image depth can be divided into four ranges. For example, LOD0 corresponds to depth range 1, that is, the image processing algorithm corresponding to LOD0 is used to process layers in depth range 1; LOD1 corresponds to depth range 2, that is, the image processing algorithm corresponding to LOD1 is used to process layers in depth range 2 layer for processing; LOD2 corresponds to depth range 3, that is, the image processing algorithm corresponding to LOD2 is used to process layers in depth range 3; LOD3 corresponds to depth range 4, that is, the image processing algorithm corresponding to LOD3 is used to process layers in depth range 3 Layers within range 4 are processed. Because this application considers that objects with greater depth are more blurred. Therefore, layers with a large depth range (distant layers) need to be processed with an LOD layer with a simpler algorithm, because the layer processed by an LOD layer with a simpler algorithm is more blurred. Therefore, a layer with a larger depth range corresponds to a higher-level LOD layer (the higher the LOD level, the simpler the algorithm, see the previous description). Similarly, a layer with a smaller depth range corresponds to a lower-level LOD layer, because the lower the level The more complex the algorithm corresponding to the lower LOD layer, the clearer the processed layer.
示例性的，假设深度范围1是0-0.3m，对应LOD0(因为LOD0生成的图层清晰最高)。深度范围2是0.3-0.5m，对应LOD1(LOD1生成的图层的清晰度比LOD0图层的清晰度低)。深度范围3是0.5-0.8m，对应LOD2(LOD2生成的图层的清晰度比LOD1图层的清晰度低)。再深度范围3是0.8-1m，对应LOD3(LOD3生成的图层的清晰度比LOD2图层的清晰度低)。也就是说，随着深度增加，图层的清晰度越来越低。最后，不同深度范围对应的图层合成图像，将该图像通过VR显示设备显示。Exemplarily, it is assumed that the depth range 1 is 0-0.3m, which corresponds to LOD0 (because the layer generated by LOD0 has the highest clarity). The depth range 2 is 0.3-0.5m, which corresponds to LOD1 (the layer generated by LOD1 has a lower resolution than the layer generated by LOD0). Depth range 3 is 0.5-0.8m, corresponding to LOD2 (the definition of the layer generated by LOD2 is lower than that of LOD1 layer). The depth range 3 is 0.8-1m, corresponding to LOD3 (the definition of the layer generated by LOD3 is lower than that of the LOD2 layer). That is, as the depth increases, the sharpness of the layer becomes lower and lower. Finally, layers corresponding to different depth ranges synthesize an image, and the image is displayed on a VR display device.
图16以VR眼镜显示一帧图像为例进行介绍。可以理解的是，一般情况下，VR眼镜显示图像流，图像流包括多帧图像。Figure 16 uses VR glasses to display a frame of image as an example. It can be understood that, generally, the VR glasses display an image stream, and the image stream includes multiple frames of images.
第一种方式，VR图像生成设备(比如图2中的图像生成设备200)默认使用图17所示的流程生成的图像流(即，每帧图像上的远处对象都模糊)。比如，以VR图像生成设备是手机为例，手机在检测到连接VR眼镜、VR眼镜开机、VR应用(如VR游戏)启动中的至少一种时，默认开始使用图17所示的流程生成图像流，然后通过VR眼镜进行显示。In the first way, the VR image generating device (such as the image generating device 200 in FIG. 2 ) uses the image stream generated by the process shown in FIG. 17 by default (that is, the distant objects on each frame of image are blurred). For example, taking the VR image generation device as a mobile phone as an example, when the mobile phone detects at least one of the connection of the VR glasses, the startup of the VR glasses, and the startup of the VR application (such as a VR game), the mobile phone starts to use the process shown in Figure 17 to generate images by default. stream, and then display it through VR glasses.
第二种方式，VR图像生成设备默认使用现有方式生成图像(即，图像上所有对象清晰度相同)，当检测用于启动第二护眼模式的指示时，使用图17所示的流程生成图像。其中，第二护眼模式请参见前文描述。也就是说，VR眼镜刚开始显示的图像上所有对象的清晰度相同，在检测到用于启动第二护眼模式的指示后，显示的图像上远处对象(如大于预设景深的对象)模糊。示例性的，请参见图18，在第i+1帧之前，图像上所有对象的清 晰度相同。当检测用于启动第二护眼种模式的指示时，第i+1帧至第N帧图像上远处对象(如，山和太阳)模糊化。其中，用于启动第二模式的指示包括但不限定于：检测到用户触发用于启动第二种模式的操作(如VR应用中包括用于启动第二护眼模式的按钮，所述操作可以是点击所述按钮的操作)、用户观看时间大于预设时长、预设时长内用户眼睛眨眼/眯眼次数大于预设次数中的至少一种。可选的，在启动第二护眼模式之前，还可以输出提示信息，该提示信息用于提示用户是否切换到第二护眼模式，在检测到用户确认切换到第二护眼模式的指示后，切换到第二护眼模式。由于启动第二护眼模式之后，图像上远处对象(如，山)模糊化，所以人脑疲劳感得到缓解，用户体验较好。In the second method, the VR image generation device uses the existing method to generate images by default (that is, all objects on the image have the same clarity), and when detecting the indication for starting the second eye protection mode, use the process shown in Figure 17 to generate image. For the second eye protection mode, please refer to the previous description. That is to say, the resolution of all objects on the image displayed by the VR glasses at the beginning is the same. Vague. For example, please refer to FIG. 18 , before the i+1th frame, all objects on the image have the same definition. When the indication for activating the second eye protection mode is detected, distant objects (such as mountains and the sun) on the images of the i+1th frame to the Nth frame are blurred. Wherein, the indication for starting the second mode includes but is not limited to: detecting that the user triggers an operation for starting the second mode (for example, a VR application includes a button for starting the second eye protection mode, and the operation can be is the operation of clicking the button), the viewing time of the user is greater than the preset duration, and the number of times the user blinks/squints within the preset duration is greater than the preset number of times. Optionally, before starting the second eye protection mode, prompt information may also be output, which is used to prompt the user whether to switch to the second eye protection mode. , switch to the second eye protection mode. Since the second eye protection mode is activated, distant objects (such as mountains) on the image are blurred, so the fatigue of the human brain is relieved, and the user experience is better.
上面的第一种方式和第二种方式中，VR图像生成设备生成的图像流中远处对象作模糊化处理，可以缓解人脑疲劳，但是容易丢失远处对象的细节。比如，第一种方式中，远处对象始终是模糊的，用户无法获得该对象的细节；第二种方式中，在检测到用于启动第二护眼模式的指示后，远处对象将一直是模糊的，用户也无法获得该对象的细节。In the first method and the second method above, the distant objects in the image stream generated by the VR image generation device are blurred, which can relieve the fatigue of the human brain, but it is easy to lose the details of the distant objects. For example, in the first method, the distant object is always blurred, and the user cannot obtain the details of the object; in the second method, after the indication for starting the second eye protection mode is detected, the distant object will always be blurred. is blurred, and the user cannot obtain the details of the object.
为了既能够缓解疲劳感，又能获取远处对象的细节。上述第一种方式或第二种方式中VR图像生成设备生成的图像流中远处对象的清晰度可以有高有低。比如，图像流包括多个周期，每个周期内包括多帧图像，且每个周期内图像上第一对象的清晰度先升后降。In order to not only relieve fatigue, but also obtain the details of distant objects. The clarity of distant objects in the image stream generated by the VR image generating device in the first manner or the second manner may be high or low. For example, the image stream includes multiple periods, and each period includes multiple frames of images, and in each period, the definition of the first object on the image increases first and then decreases.
示例性的，请参见图19，一个周期内，第i帧图像上远处对象(如，山)的清晰度低于第j帧图像上的山的清晰度，第j帧图像上的山的清晰度高于第k帧图像上山的清晰度。即，一个周期内远处对象的清晰度呈“模糊-清晰-模糊”的变化趋势。下一个周期内，第k帧图像上山的清晰度低于第p帧图像上山的清晰度，第p帧图像上山的清晰度高于第q帧图像上山的清晰度。即下一个周期内，远处对象的清晰度也是呈“模糊-清晰-模糊”的变化趋势。图19中的两个周期可以相同或不同，不作限定。这种清晰度变化趋势一方面可以缓解人脑疲劳感，另一方面可以避免用户丢失远处对象的图像细节。For example, please refer to FIG. 19. In one period, the sharpness of distant objects (such as mountains) on the i-th frame image is lower than the sharpness of the j-th frame image on the mountain, and the definition of the j-th frame image on the mountain is The sharpness is higher than that of the kth frame image uphill. That is, the sharpness of distant objects shows a change trend of "fuzzy-clear-fuzzy" within a cycle. In the next period, the sharpness of the k-th frame image uphill is lower than that of the p-th frame image uphill, and the p-th frame image's sharpness of the uphill is higher than the qth frame image's sharpness of the uphill. That is to say, in the next cycle, the sharpness of distant objects also shows a change trend of "fuzzy-clear-fuzzy". The two periods in Fig. 19 may be the same or different, without limitation. On the one hand, this sharpness change trend can alleviate the fatigue of the human brain, and on the other hand, it can prevent users from losing image details of distant objects.
示例性的，i、j、k、p满足：j＝i+n，k＝j+m，p＝k+w，q＝p+s，第j帧是第i帧之后的n帧，第k帧是第j帧之后的m帧，第p帧是第k帧之后的w帧，第q帧是第p帧之后的s帧。其中，n、m、q、s为大于或等于1的整数。如，n、m、p、s均为1，即，第j帧图像是第i帧图像的下一帧，第k帧图像是第j帧图像的下一帧，第p帧是第k帧的下一帧，第q帧是第p帧的下一帧。或者，n、m、p、s可以根据用户视觉停留时间、图像刷新帧率确定，与实施例一的实现原理相同，不重复赘述。Exemplarily, i, j, k, p satisfy: j=i+n, k=j+m, p=k+w, q=p+s, the jth frame is the n frame after the ith frame, the The k-th frame is the m-frame after the j-th frame, the p-th frame is the w-frame after the k-th frame, and the q-th frame is the s-frame after the p-th frame. Wherein, n, m, q, s are integers greater than or equal to 1. For example, n, m, p, and s are all 1, that is, the j-th frame image is the next frame of the i-th frame image, the k-th frame image is the next frame of the j-th frame image, and the p-th frame is the k-th frame The next frame of , the qth frame is the next frame of the pth frame. Alternatively, n, m, p, and s can be determined according to the user's visual dwell time and image refresh frame rate, which are the same as the implementation principle of Embodiment 1, and will not be repeated.
可选的，继续以图19为例，图像流中近景对象的清晰度可以不变。比如，图19中，树的清晰度可以不变。Optionally, continuing to take FIG. 19 as an example, the definition of objects in the foreground in the image stream may not change. For example, in Figure 19, the clarity of the tree may not change.
实施例一和实施例二可以单独实施，或者结合实施。比如，VR图像生成设备可以默认使用实施例一的技术方案(比如实施例一中的第一种方式或第二种方式)，或者默认使用实施例二的技术方案(比如实施例二中的第一种方式或第二种方式)，或者，VR图像生成设备上包括切换按钮，通过该切换按钮可以设置VR图像生成设备使用实施例一的技术方案还是实施例二的技术方案；或者，VR应用中包括按钮，通过该按钮用户可以设置VR应用使用实施例一的技术方案还是实施例二的技术方案。 Embodiment 1 and Embodiment 2 can be implemented independently or in combination. For example, the VR image generation device may use the technical solution of Embodiment 1 by default (such as the first method or the second method in Embodiment 1), or use the technical solution of Embodiment 2 by default (such as the first method in Embodiment 2). One method or the second method), or, the VR image generating device includes a switching button, through which the VR image generating device can be set to use the technical solution of the first embodiment or the technical solution of the second embodiment; or, the VR application includes a button, through which the user can set whether the VR application uses the technical solution of Embodiment 1 or the technical solution of Embodiment 2.
在一种实施例中，VR眼镜具有两个显示屏，第一显示屏和第二显示屏。其中，第一显示屏用于向用户左眼呈现图像，第二显示屏用于向用户右眼呈现图像。为了方便描述， 将左眼对应的显示屏称为左眼显示屏，右眼对应的显示屏称为右眼显示屏。In one embodiment, the VR glasses have two display screens, a first display screen and a second display screen. Wherein, the first display screen is used to present images to the user's left eye, and the second display screen is used to present images to the user's right eye. For convenience of description, the display screen corresponding to the left eye is referred to as the left-eye display screen, and the display screen corresponding to the right eye is referred to as the right-eye display screen.
左眼显示屏和右眼显示屏分别用于显示图像流。所述图像流可以是使用实施例一的方式生成的图像流(如，图13或图12所示的图像流)，或者，是使用实施例二的方式生成的图像流(如，图18或图19所示的图像流)。The left-eye display and the right-eye display are used to display image streams, respectively. The image stream may be an image stream generated using the method of Embodiment 1 (such as the image stream shown in FIG. 13 or 12 ), or an image stream generated using the method of Embodiment 2 (such as FIG. 18 or image stream shown in Figure 19).
以左眼显示屏显示的图像流和右眼显示屏显示的图像流均是实施例一中图13的图像流为例。As an example, the image stream displayed on the left-eye display screen and the image stream displayed on the right-eye display screen are both the image streams shown in FIG. 13 in the first embodiment.
为了保证左右眼摄取的图像信息相同，左眼显示屏和右眼显示屏显示的图像同步。比如，请参见图20，当左眼显示屏显示第i帧图像时，右眼显示屏也显示第i帧图像。由于第i帧图像上远离用户注视点的对象(如，树)是模糊的，所以，此时，左眼和右眼看到的树均是模糊的。继续参见图20，当左眼显示屏显示第j帧图像时，右眼显示屏也显示第j帧图像。由于第j帧图像上远离用户注视点的对象(如，树)是清晰的，所以，此时，左眼和右眼看到的树均是清晰的。这种方式中，左、右眼显示屏同步显示第i帧图像时，人脑中将左、右眼显示屏的图像合成得到的图像上远离用户注视点的对象模糊，可以缓解疲惫感。当左、右眼显示屏同步显示第j帧图像时，人脑中将左、右眼显示屏的图像合成得到的图像上远离用户注视点的对象清晰，可以获取到远离用户注视点的对象的细节。In order to ensure that the image information captured by the left and right eyes is the same, the images displayed on the left-eye display screen and the right-eye display screen are synchronized. For example, please refer to FIG. 20 , when the i-th frame of image is displayed on the left-eye display screen, the i-th frame of image is also displayed on the right-eye display screen. Since the object (for example, a tree) away from the user's gaze point on the i-th frame image is blurred, at this time, the trees seen by the left eye and the right eye are both blurred. Continuing to refer to FIG. 20 , when the left-eye display screen displays the j-th frame of image, the right-eye display screen also displays the j-th frame of image. Since the object (for example, a tree) away from the user's gaze point on the image frame j is clear, the trees seen by the left eye and the right eye are both clear at this time. In this way, when the left and right eye display screens display the i-th frame of image synchronously, objects far away from the user's gaze point on the image obtained by synthesizing the left and right eye display screen images in the human brain will be blurred, which can relieve fatigue. When the left and right eye display screens synchronously display the j-th frame of image, the image obtained by synthesizing the images of the left and right eye display screens in the human brain is clear, and the objects far away from the user's gaze point can be obtained. detail.
图20中，左眼显示屏显示的图像流和右眼显示屏显示的图像流中远离用户注视点的对象的清晰度变化趋势相同，都是“模糊-清晰-模糊-清晰”的变化趋势。在另一些实施例中，左眼显示屏显示的图像流和右眼显示屏显示的图像流中远离用户注视点的对象的清晰度变化趋势可以相反。比如，左眼显示屏显示的图像流中远离用户注视点的对象的清晰度呈“模糊-清楚-模糊-清楚”的交替变化，右眼显示屏显示的图像流中远离用户注视点的对象呈“清楚-模糊-清楚-模糊”的交替变化。比如，请参见图21，当左眼显示屏显示第i帧图像时，右眼显示屏也显示第i帧图像。左眼显示屏上的第i帧图像上远离用户注视点的对象(如，树)是模糊的，右眼显示屏上第i帧图像中树是清楚的。所以，此时，左眼看到的树是模糊的，右眼看到的树是清楚的。继续参见图20，当左眼显示屏显示第j帧图像时，右眼显示屏也显示第j帧图像。左眼显示屏上的第j帧图像上树是清楚的，右眼显示屏上第j帧图像中树是模糊的。所以，此时，左眼看到的树是清楚的，右眼看到的树是模糊的。这种方式中，当左、右眼显示屏同步显示图像(第i帧图像或第j帧图像)时，左眼图像上远离注视点的对象清晰，右眼图像上远离注视点的对象模糊，一定程度上可以缓解人脑疲劳，而且人脑中将左眼图像和右眼图像叠加得到的图像上远离注视点的对象的不至于太过模糊，避免丢失该对象的太多细节。In FIG. 20 , the sharpness change trend of objects far away from the user's gaze point in the image stream displayed on the left-eye display screen and the image stream displayed on the right-eye display screen is the same, and both are "blurry-clear-blurry-clear" change trends. In some other embodiments, the sharpness change trends of objects far away from the user's gaze point in the image stream displayed on the left-eye display screen and the image stream displayed on the right-eye display screen may be opposite. For example, the sharpness of objects far away from the user's gaze in the image stream displayed on the left-eye display shows an alternating change of "fuzzy-clear-fuzzy-clear", and the sharpness of objects far away from the user's gaze in the image stream displayed on the right-eye display shows "Clear-blur-clear-blur" alternate change. For example, please refer to FIG. 21 , when the i-th frame image is displayed on the left-eye display screen, the i-th frame image is also displayed on the right-eye display screen. Objects far away from the user's gaze point (for example, trees) on the i-th frame image on the left-eye display screen are blurred, and the tree is clear in the i-th frame image on the right-eye display screen. Therefore, at this time, the tree seen by the left eye is blurred, and the tree seen by the right eye is clear. Continuing to refer to FIG. 20 , when the left-eye display screen displays the j-th frame of image, the right-eye display screen also displays the j-th frame of image. The tree is clear in the jth frame image on the left-eye display screen, and the tree is blurred in the j-th frame image on the right-eye display screen. Therefore, at this time, the tree seen by the left eye is clear, and the tree seen by the right eye is blurred. In this way, when the left and right eye display screens display images synchronously (the i-th frame image or the j-th frame image), the objects far away from the fixation point on the left-eye image are clear, and the objects far from the fixation point on the right-eye image are blurred. To a certain extent, it can alleviate the fatigue of the human brain, and the image obtained by superimposing the left-eye image and the right-eye image in the human brain will not be too blurred for the object far away from the gaze point, and avoid losing too much detail of the object.
以左眼显示屏显示的图像流和右眼显示屏显示的图像流均是实施例二中图19所示的图像流为例。As an example, both the image stream displayed on the left-eye display screen and the image stream displayed on the right-eye display screen are the image streams shown in FIG. 19 in the second embodiment.
为了保证左右眼摄取的图像信息相同，左眼显示屏和右眼显示屏显示的图像同步。比如，请参见图22，当左眼显示屏显示第i帧图像时，右眼显示屏也显示第i帧图像。由于第i帧图像上远处对象(如，山和太阳)是模糊的，所以，此时，左眼和右眼看到的远处对象均是模糊的。继续参见图22，当左眼显示屏显示第j帧图像时，右眼显示屏也显示第j帧图像。由于第j帧图像上远处对象(如，山和太阳)是清晰的，所以，此时，左眼和右眼看到的远处对象均是清晰的。这种方式中，左、右眼显示屏同步显示第i帧图像时，人脑中将左、右眼显示屏的图像合成得到的图像上远处对象模糊，可以缓解疲惫感。当左、 右眼显示屏同步显示第j帧图像时，人脑中将左、右眼显示屏的图像合成得到的图像上远处对象清晰，可以获取到远处对象的细节。In order to ensure that the image information captured by the left and right eyes is the same, the images displayed on the left-eye display screen and the right-eye display screen are synchronized. For example, please refer to FIG. 22 , when the i-th frame image is displayed on the left-eye display screen, the i-th frame image is also displayed on the right-eye display screen. Since the distant objects (such as mountains and the sun) are blurred on the i-th frame image, at this time, the distant objects seen by the left eye and the right eye are both blurred. Continuing to refer to FIG. 22 , when the left-eye display screen displays the j-th frame of image, the right-eye display screen also displays the j-th frame of image. Since the distant objects (eg, the mountain and the sun) are clear on the jth frame image, at this time, the distant objects seen by the left eye and the right eye are both clear. In this way, when the left and right eye display screens synchronously display the i-th frame of image, the human brain synthesizes the images of the left and right eye display screens to obtain blurred distant objects on the image, which can relieve fatigue. When the left and right eye display screens synchronously display the j-th frame of image, the image obtained by combining the images of the left and right eye display screens in the human brain is clear, and the details of the distant object can be obtained.
图22中左眼显示屏显示的图像流和右眼显示屏显示的图像流中远处对象的清晰度变化趋势相同，都是“模糊-清晰-模糊-清晰”的变化趋势。在另一些实施例中，左眼显示屏显示的图像流和右眼显示屏显示的图像流中远处对象的清晰度变化趋势可以相反。比如，左眼显示屏显示的图像流中远处对象的清晰度呈“模糊-清楚-模糊-清楚”的交替变化，右眼显示屏显示的图像流中远处对象呈“清楚-模糊-清楚-模糊”的交替变化。比如，请参见图23，当左眼显示屏显示第i帧图像时，右眼显示屏也显示第i帧图像。左眼显示屏上的第i帧图像上远处对象(如，山和太阳)是模糊的，右眼显示屏上第i帧图像中远处对象是清楚的。所以，此时，左眼看到的远处对象模糊，右眼看到的远处对象清楚。继续参见图23，当左眼显示屏显示第j帧图像时，右眼显示屏也显示第j帧图像。左眼显示屏上的第j帧图像上远处对象(如，山和太阳)清楚，右眼显示屏上第j帧图像中远处对象(如，山和太阳)模糊。所以，此时，左眼看到的远处对象是清楚的，右眼看到的远处对象是模糊的。这种方式中，当左、右眼显示屏同步显示图像(第i帧图像或第j帧图像)时，左眼图像上远处对象清晰，右眼图像上远处对象模糊，一定程度上可以缓解人脑疲劳，而且人脑中将左眼图像和右眼图像叠加得到的图像上远处对象的不至于太过模糊，避免丢失该对象的太多细节。In FIG. 22 , the image stream displayed on the left-eye display screen and the image stream displayed on the right-eye display screen have the same changing trend of the sharpness of distant objects, both of which are "fuzzy-clear-fuzzy-clear". In some other embodiments, the sharpness change trend of the distant object in the image stream displayed on the left-eye display screen and the image stream displayed on the right-eye display screen may be opposite. For example, the sharpness of distant objects in the image stream displayed on the left-eye display shows an alternating change of "blur-clear-blur-clear", and the image stream displayed on the right-eye display shows "clear-blur-clear-blur" " Alternate changes. For example, please refer to FIG. 23 , when the i-th frame image is displayed on the left-eye display screen, the i-th frame image is also displayed on the right-eye display screen. The distant objects (such as mountains and the sun) are blurred on the i-th frame image on the left-eye display screen, and the distant objects are clear in the i-th frame image on the right-eye display screen. Therefore, at this time, the distant objects seen by the left eye are blurred, and the distant objects seen by the right eye are clear. Continuing to refer to FIG. 23 , when the left-eye display screen displays the j-th frame of image, the right-eye display screen also displays the j-th frame of image. The distant objects (such as mountains and the sun) are clear on the jth frame image on the left-eye display screen, and the distant objects (such as mountain and sun) are blurred in the j-th frame image on the right-eye display screen. Therefore, at this time, the distant objects seen by the left eye are clear, and the distant objects seen by the right eye are blurred. In this way, when the left-eye and right-eye display screens display images synchronously (i-th frame image or j-th frame image), the distant objects on the left-eye image are clear, and the distant objects on the right-eye image are blurred, which can be achieved to a certain extent. Relieve the fatigue of the human brain, and the image of the distant object on the image obtained by superimposing the left eye image and the right eye image in the human brain will not be too blurred, and avoid losing too much detail of the object.
基于相同的构思，图24所示为本申请提供的一种电子设备2400。该电子设备2400可以是前文中的VR穿戴设备(如，VR眼镜)。如图24所示，电子设备2400可以包括：一个或多个处理器2401；一个或多个存储器2402；通信接口2403，以及一个或多个计算机程序2404，上述各器件可以通过一个或多个通信总线2405连接。其中该一个或多个计算机程序2404被存储在上述存储器2402中并被配置为被该一个或多个处理器2401执行，该一个或多个计算机程序2404包括指令，上述指令可以用于执行如上面相应实施例中手机的相关步骤。通信接口2403用于实现与其他设备的通信，比如通信接口可以是收发器。Based on the same idea, FIG. 24 shows an electronic device 2400 provided by this application. The electronic device 2400 may be the aforementioned VR wearable device (eg, VR glasses). As shown in Figure 24, the electronic device 2400 may include: one or more processors 2401; one or more memories 2402; a communication interface 2403, and one or more computer programs 2404, and each of the above devices may communicate through one or more bus 2405 connection. Wherein the one or more computer programs 2404 are stored in the above-mentioned memory 2402 and configured to be executed by the one or more processors 2401, the one or more computer programs 2404 include instructions, and the above-mentioned instructions can be used to perform the above-mentioned Relevant steps of the mobile phone in the corresponding embodiment. The communication interface 2403 is used to implement communication with other devices, for example, the communication interface may be a transceiver.
上述本申请提供的实施例中，从电子设备(例如手机)作为执行主体的角度对本申请实施例提供的方法进行了介绍。为了实现上述本申请实施例提供的方法中的各功能，电子设备可以包括硬件结构和/或软件模块，以硬件结构、软件模块、或硬件结构加软件模块的形式来实现上述各功能。上述各功能中的某个功能以硬件结构、软件模块、还是硬件结构加软件模块的方式来执行，取决于技术方案的特定应用和设计约束条件。In the above-mentioned embodiments provided in the present application, the methods provided in the embodiments of the present application are introduced from the perspective of an electronic device (such as a mobile phone) as an execution subject. In order to realize the various functions in the method provided by the above embodiments of the present application, the electronic device may include a hardware structure and/or a software module, and realize the above-mentioned functions in the form of a hardware structure, a software module, or a hardware structure plus a software module. Whether one of the above-mentioned functions is executed in the form of a hardware structure, a software module, or a hardware structure plus a software module depends on the specific application and design constraints of the technical solution.
以上实施例中所用，根据上下文，术语“当…时”或“当…后”可以被解释为意思是“如果…”或“在…后”或“响应于确定…”或“响应于检测到…”。类似地，根据上下文，短语“在确定…时”或“如果检测到(所陈述的条件或事件)”可以被解释为意思是“如果确定…”或“响应于确定…”或“在检测到(所陈述的条件或事件)时”或“响应于检测到(所陈述的条件或事件)”。另外，在上述实施例中，使用诸如第一、第二之类的关系术语来区份一个实体和另一个实体，而并不限制这些实体之间的任何实际的关系和顺序。As used in the above embodiments, depending on the context, the terms "when" or "after" may be interpreted to mean "if" or "after" or "in response to determining..." or "in response to detecting ...". Similarly, depending on the context, the phrases "in determining" or "if detected (a stated condition or event)" may be interpreted to mean "if determining..." or "in response to determining..." or "on detecting (a stated condition or event)" or "in response to detecting (a stated condition or event)". In addition, in the above embodiments, relational terms such as first and second are used to distinguish one entity from another, without limiting any actual relationship and order between these entities.
在本说明书中描述的参考“一个实施例”或“一些实施例”等意味着在本申请的一个或多个实施例中包括结合该实施例描述的特定特征、结构或特点。由此，在本说明书中的不同之处出现的语句“在一个实施例中”、“在一些实施例中”、“在其他一些实施例中”、“在另外一些实施例中”等不是必然都参考相同的实施例，而是意味着“一个或多个但不是所有的实 施例”，除非是以其他方式另外特别强调。术语“包括”、“包含”、“具有”及它们的变形都意味着“包括但不限于”，除非是以其他方式另外特别强调。Reference to "one embodiment" or "some embodiments" or the like in this specification means that a particular feature, structure, or characteristic described in connection with the embodiment is included in one or more embodiments of the present application. Thus, appearances of the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in other embodiments," etc. in various places in this specification are not necessarily All refer to the same embodiment, but mean "one or more but not all embodiments" unless specifically stated otherwise. The terms "including", "comprising", "having" and variations thereof mean "including but not limited to", unless specifically stated otherwise.
在上述实施例中，可以全部或部分地通过软件、硬件、固件或者其任意组合来实现。当使用软件实现时，可以全部或部分地以计算机程序产品的形式实现。所述计算机程序产品包括一个或多个计算机指令。在计算机上加载和执行所述计算机程序指令时，全部或部分地产生按照本实施例所述的流程或功能。所述计算机可以是通用计算机、专用计算机、计算机网络、或者其他可编程装置。所述计算机指令可以存储在计算机可读存储介质中，或者从一个计算机可读存储介质向另一个计算机可读存储介质传输，例如，所述计算机指令可以从一个网站站点、计算机、服务器或数据中心通过有线(例如同轴电缆、光纤、数字用户线(DSL))或无线(例如红外、无线、微波等)方式向另一个网站站点、计算机、服务器或数据中心进行传输。所述计算机可读存储介质可以是计算机能够存取的任何可用介质或者是包含一个或多个可用介质集成的服务器、数据中心等数据存储设备。所述可用介质可以是磁性介质，(例如，软盘、硬盘、磁带)、光介质(例如，DVD)、或者半导体介质(例如固态硬盘Solid State Disk(SSD))等。在不冲突的情况下，以上各实施例的方案都可以组合使用。In the above embodiments, all or part of them may be implemented by software, hardware, firmware or any combination thereof. When implemented using software, it may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on the computer, all or part of the processes or functions described in this embodiment will be generated. The computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website, computer, server or data center Transmission to another website site, computer, server, or data center by wired (eg, coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (eg, infrared, wireless, microwave, etc.). The computer-readable storage medium may be any available medium that can be accessed by a computer, or a data storage device such as a server or a data center integrated with one or more available media. The available medium may be a magnetic medium (for example, a floppy disk, a hard disk, or a magnetic tape), an optical medium (for example, DVD), or a semiconductor medium (for example, a Solid State Disk (SSD)). In the case of no conflict, the solutions of the above embodiments can be used in combination.
需要指出的是，本专利申请文件的一部分包含受著作权保护的内容。除了对专利局的专利文件或记录的专利文档内容制作副本以外，著作权人保留著作权。It should be pointed out that a part of the patent application documents contains content protected by copyright. Copyright is reserved by the copyright owner other than to make copies of the contents of the patent file or records of the Patent Office.

Claims (18)

  1. 一种显示方法，其特征在于，包括：A display method, characterized by comprising:
    通过显示设备向用户展示N帧图像；N为正整数；Display N frames of images to the user through the display device; N is a positive integer;
    所述N帧图像中第i帧图像上，处于第一景深处的第一对象的清晰度为第一清晰度；On the i-th frame of images in the N frames of images, the definition of the first object in the first depth of field is the first definition;
    所述N帧图像中第j帧图像上，处于所述第一景深处的所述第一对象的清晰度为第二清晰度；On the jth frame image in the N frames of images, the definition of the first object in the first depth of field is the second definition;
    所述N帧图像中第k帧图像上，处于所述第一景深处的所述第一对象的清晰度为第三清晰度；On the k-th frame of images in the N frames of images, the definition of the first object in the first depth of field is the third definition;
    其中，所述第一清晰度小于所述第二清晰度，所述第二清晰度大于所述第三清晰度，i、j、k均为小于N的正整数，i＜j＜k；Wherein, the first definition is smaller than the second definition, the second definition is greater than the third definition, i, j, k are all positive integers less than N, i<j<k;
    所述第一景深大于第二景深，或者，所述第一景深与所述用户注视点所在景深之间的距离大于第一距离。The first depth of field is greater than the second depth of field, or, the distance between the first depth of field and the depth of field where the gaze point of the user is located is greater than the first distance.
  2. 根据权利要求1所述的方法，其特征在于，在显示所述第i帧图像、所述第j帧图像以及所述第k帧图像期间，所述用户注视点不变。The method according to claim 1, wherein during displaying the i-th frame image, the j-th frame image and the k-th frame image, the gaze point of the user remains unchanged.
  3. 根据权利要求1或2所述的方法，其特征在于，The method according to claim 1 or 2, characterized in that,
    所述第二景深包括：用户指定景深、用户注视点所在景深、***默认景深、虚像面所在景深、与虚拟场景对应的景深和所述第i帧图像上主体对象所在景深中的至少一种。The second depth of field includes: at least one of the depth of field specified by the user, the depth of field of the user's gaze point, the default depth of field of the system, the depth of field of the virtual image plane, the depth of field corresponding to the virtual scene, and the depth of field of the subject object on the i-th frame image.
  4. 根据权利要求1-3任一所述的方法，其特征在于，在显示所述N帧图像之前，所述方法还包括：检测到用户触发用于启动护眼模式的操作、用户观看时间大于第一时长、第二时长内用户眼睛眨眼/眯眼次数大于第一次数中的至少一项。The method according to any one of claims 1-3, characterized in that, before displaying the N frames of images, the method further comprises: detecting that the user triggers the operation for starting the eye protection mode, and the user's viewing time is longer than the first At least one of the number of times the user blinks/squints within the first time period and the second time period is greater than the first time period.
  5. 根据权利要求1-4任一所述的方法，其特征在于，所述N帧图像中处于第三景深处的第二对象的清晰度相同。The method according to any one of claims 1-4, characterized in that the second objects in the third depth of field in the N frames of images have the same definition.
  6. 根据权利要求5所述的方法，其特征在于，所述第三景深小于所述第一景深。The method of claim 5, wherein the third depth of field is smaller than the first depth of field.
  7. 根据权利要求5所述的方法，其特征在于，所述第三景深与所述用户注视点所在景深之间的距离小于所述第一景深与所述用户注视点所在景深之间的距离。The method according to claim 5, wherein the distance between the third depth of field and the depth of field where the user gaze point is smaller than the distance between the first depth of field and the depth of field where the user gaze point is located.
  8. 根据权利要求1-7任一所述的方法，其特征在于，The method according to any one of claims 1-7, characterized in that,
    所述第j帧图像的显示时间与所述第i帧图像的显示时间之间的时间间隔小于或等于所述用户的视觉停留时长；和/或，The time interval between the display time of the j-th frame image and the display time of the i-th frame image is less than or equal to the user's visual dwell time; and/or,
    所述第k帧图像的显示时间与所述第j帧图像的显示时间之间的时间间隔小于或等于所述时间停留时长。A time interval between the display time of the kth frame of image and the display time of the jth frame of image is less than or equal to the time duration.
  9. 根据权利要求1-8任一所述的方法，其特征在于，The method according to any one of claims 1-8, characterized in that,
    j＝i+n，其中，n大于或等于1，或者，n随着所述用户的时间停留时长与所述显示设备的图像刷新帧率的变化而变化；和/或，j=i+n, wherein, n is greater than or equal to 1, or, n changes with the change of the user's time stay and the image refresh frame rate of the display device; and/or,
    k＝j+m，其中，m大于或等于1，或者，m随着所述用户的时间停留时长与所述显示设备的图像刷新帧率的变化而变化。k=j+m, wherein, m is greater than or equal to 1, or, m changes with the change of the user's time stay and the image refresh frame rate of the display device.
  10. 根据权利要求1-9任一所述的方法，其特征在于，The method according to any one of claims 1-9, characterized in that,
    所述显示设备包括第一显示屏和第二显示屏，所述第一显示屏用于向用户左眼呈现图像，所述第二显示屏用于向用户右眼呈现图像；所述第一显示屏和所述第二显示屏上显示的图像同步；The display device includes a first display screen and a second display screen, the first display screen is used to present images to the user's left eye, and the second display screen is used to present images to the user's right eye; the first display screen screen and the images displayed on the second display screen are synchronized;
    所述第一显示屏和所述第二显示屏分别显示所述N帧图像；或者，The first display screen and the second display screen respectively display the N frames of images; or,
    所述第一显示屏显示所述N帧图像；所述第二显示屏显示另外N帧图像；所述另外N帧图像与所述N帧图像的图像内容相同；The first display screen displays the N frames of images; the second display screen displays another N frames of images; the other N frames of images have the same image content as the N frames of images;
    所述另外N帧图像中第i帧图像上，处于第一景深处的所述第一对象的清晰度为第四清晰度；On the i-th frame image in the other N frames of images, the definition of the first object in the first depth of field is the fourth definition;
    所述另外N帧图像中第j帧图像上，处于所述第一景深处的所述第一对象的清晰度为第五清晰度；On the j-th frame image in the other N frames of images, the definition of the first object in the first depth of field is the fifth definition;
    所述另外N帧图像中第k帧图像上，处于所述第一景深处的所述第一对象的清晰度为第六清晰度；On the k-th frame image in the other N frames of images, the definition of the first object in the first depth of field is the sixth definition;
    其中，所述第四清晰度大于所述第五清晰度，所述第四清晰度小于所述第六清晰度。Wherein, the fourth definition is greater than the fifth definition, and the fourth definition is smaller than the sixth definition.
  11. 根据权利要求10所述的方法，其特征在于，The method according to claim 10, characterized in that,
    所述第四清晰度大于所述第一清晰度；和/或，said fourth sharpness is greater than said first sharpness; and/or,
    所述第五清晰度小于所述第二清晰度；和/或，said fifth resolution is less than said second resolution; and/or,
    所述第六清晰度大于所述第三清晰度。The sixth definition is greater than the third definition.
  12. 根据权利要求1-11任一所述的方法，其特征在于，The method according to any one of claims 1-11, characterized in that,
    所述N帧图像是与游戏相关的图像；The N frames of images are images related to games;
    所述第二景深包括：游戏场景中，所述用户对应的游戏角色所在景深，或者，所述用户对应的游戏角色上身体部位所在景深，或者，所述用户对应的游戏角色当前所持游戏装备所在景深；和/或，The second depth of field includes: in the game scene, the depth of field where the game character corresponding to the user is located, or the depth of field where the body parts of the game character corresponding to the user are located, or where the game equipment currently held by the game character corresponding to the user is located. depth of field; and/or,
    所述用户注视点所在景深包括：游戏场景中，游戏对方对应的游戏角色所在景深，或者，建筑物所在景深，或者，所述用户对应的游戏角色上身体部位所在景深，或者，所述用户对应的游戏角色当前所持游戏装备所在景深。The depth of field where the user's gaze point is located includes: in the game scene, the depth of field where the game character corresponding to the game opponent is located, or the depth of field where the building is located, or the depth of field where the body parts of the game character corresponding to the user are located, or the depth of field where the user's corresponding game character is located. The depth of field of the game equipment currently held by the game character.
  13. 根据权利要求1-11任一所述的方法，其特征在于，The method according to any one of claims 1-11, characterized in that,
    所述N帧图像是与车辆驾驶相关的图像；The N frames of images are images related to vehicle driving;
    所述第二景深包括：车辆驾驶场景中，所述用户当前驾驶车辆所在景深，或者，所述用户当前驾驶车辆上方向盘所在景深，或者，所述用户当前驾驶车辆上挡风玻璃所在景深；和/或，The second depth of field includes: in the vehicle driving scene, the depth of field of the vehicle currently driven by the user, or the depth of field of the steering wheel of the vehicle currently driven by the user, or the depth of field of the windshield of the vehicle currently driven by the user; and /or,
    所述用户注视点所在景深，包括：车辆驾驶场景中，行驶道路上其它用户驾驶车辆，或者，行驶道路的路边设置对象。The depth of field where the user's gaze point is located includes: in the vehicle driving scene, other users driving the vehicle on the driving road, or objects set on the roadside of the driving road.
  14. 根据权利要求1-13任一所述的方法，其特征在于，The method according to any one of claims 1-13, characterized in that,
    所述第i帧图像是对第i帧原图上所述第一对象做模糊化处理后的图像；The image of the i-th frame is an image obtained by blurring the first object on the original image of the i-th frame;
    所述第j帧图像是第j帧原图或对第j帧原图上所述第一对象作清晰化处理后的图像；The image in the jth frame is the original image in the jth frame or an image after clearing the first object on the original image in the jth frame;
    所述第k帧图像是对第k帧原图上所述第一对象做模糊化处理后的图像；The image of the kth frame is an image obtained by blurring the first object on the original image of the kth frame;
    其中，所述第i帧原图、第j帧原图、第k帧原图上所有对象的清晰度相同；Wherein, the clarity of all objects on the original image of the i frame, the original image of the j frame, and the original image of the k frame is the same;
    其中，所述第j帧图像是对第j帧原图上所述第一对象作清晰化处理后的图像，包括：Wherein, the image of the jth frame is an image after clearing the first object on the original image of the jth frame, including:
    所述第j帧图像是所述第i帧图像和所述第j帧原图融合得到的图像；或者，The j-th frame image is an image obtained by fusing the i-th frame image and the j-th frame original image; or,
    所述第j帧图像是对所述第i帧图像作模糊化处理时被丢失的图像信息和所述第j帧原图融合得到的图像。The image of the jth frame is an image obtained by fusing the image information lost when the image of the ith frame is blurred and the original image of the jth frame.
  15. 根据权利要求14所述的方法，其特征在于，所述第j帧图像是所述第i帧图像和所述第j帧原图融合得到的图像，包括：The method according to claim 14, wherein the j-th frame image is an image obtained by fusing the i-th frame image and the j-th frame original image, comprising:
    所述第j帧图像上所述第一对象所在区域内的图像块是第一图像块和第二图像块融合得到的；其中，所述第一图像块是所述第i帧图像上所述第一对象所在区域内的图像块，所述第二图像块是所述第j帧原图上所述第一对象所在区域内的图像块。The image block in the area where the first object is located on the jth frame image is obtained by fusing the first image block and the second image block; wherein, the first image block is the An image block in the area where the first object is located, and the second image block is an image block in the area where the first object is located in the original image of the jth frame.
  16. 一种电子设备，其特征在于，包括：An electronic device, characterized in that it comprises:
    处理器，存储器，以及，一个或多个程序；processor, memory, and, one or more programs;
    其中，所述一个或多个程序被存储在所述存储器中，所述一个或多个程序包括指令，当所述指令被所述处理器执行时，使得所述电子设备执行如权利要求1-15任一项所述的方法步骤。Wherein, the one or more programs are stored in the memory, and the one or more programs include instructions, which, when executed by the processor, cause the electronic device to perform the functions described in claim 1- 15 any one of the method steps.
  17. 一种计算机可读存储介质，其特征在于，所述计算机可读存储介质用于存储计算机程序，当所述计算机程序在计算机上运行时，使得所述计算机执行如权利要求1至15中任意一项所述的方法。A computer-readable storage medium, characterized in that the computer-readable storage medium is used to store a computer program, and when the computer program is run on a computer, the computer executes any one of claims 1 to 15. method described in the item.
  18. 一种计算机程序产品，其特征在于，包括计算机程序，当所述计算机程序在计算机上运行时，使得所述计算机执行如上述权利要求1-15中任意一项所述的方法。A computer program product, characterized in that it includes a computer program, and when the computer program is run on a computer, it causes the computer to execute the method according to any one of claims 1-15.
PCT/CN2022/106280 2021-07-21 2022-07-18 Display method and electronic device WO2023001113A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202110824187.7A CN115686181A (en) 2021-07-21 2021-07-21 Display method and electronic equipment
CN202110824187.7 2021-07-21

Publications (1)

Publication Number Publication Date
WO2023001113A1 true WO2023001113A1 (en) 2023-01-26

Family

ID=84980024

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/106280 WO2023001113A1 (en) 2021-07-21 2022-07-18 Display method and electronic device

Country Status (2)

Country Link
CN (1) CN115686181A (en)
WO (1) WO2023001113A1 (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116850012A (en) * 2023-06-30 2023-10-10 广州视景医疗软件有限公司 Visual training method and system based on binocular vision

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105072436A (en) * 2015-08-28 2015-11-18 胡东海 Automatic adjustment method and adjustment device of virtual reality and augmented reality imaging depth-of-field
CN106484116A (en) * 2016-10-19 2017-03-08 腾讯科技（深圳）有限公司 The treating method and apparatus of media file
US20170160798A1 (en) * 2015-12-08 2017-06-08 Oculus Vr, Llc Focus adjustment method for a virtual reality headset
CN110095870A (en) * 2019-05-28 2019-08-06 京东方科技集团股份有限公司 Optical presentation system, display control unit and augmented reality equipment

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105072436A (en) * 2015-08-28 2015-11-18 胡东海 Automatic adjustment method and adjustment device of virtual reality and augmented reality imaging depth-of-field
US20170160798A1 (en) * 2015-12-08 2017-06-08 Oculus Vr, Llc Focus adjustment method for a virtual reality headset
CN106484116A (en) * 2016-10-19 2017-03-08 腾讯科技（深圳）有限公司 The treating method and apparatus of media file
CN110095870A (en) * 2019-05-28 2019-08-06 京东方科技集团股份有限公司 Optical presentation system, display control unit and augmented reality equipment

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116850012A (en) * 2023-06-30 2023-10-10 广州视景医疗软件有限公司 Visual training method and system based on binocular vision
CN116850012B (en) * 2023-06-30 2024-03-12 广州视景医疗软件有限公司 Visual training method and system based on binocular vision

Also Published As

Publication number Publication date
CN115686181A (en) 2023-02-03

Similar Documents

Publication Publication Date Title
US11899212B2 (en) Image display method and device for head mounted display
US10009542B2 (en) Systems and methods for environment content sharing
US11024083B2 (en) Server, user terminal device, and control method therefor
EP3862845B1 (en) Method for controlling display screen according to eyeball focus and head-mounted electronic equipment
JP7408678B2 (en) Image processing method and head mounted display device
JP6094190B2 (en) Information processing apparatus and recording medium
WO2022252924A1 (en) Image transmission and display method and related device and system
WO2021013043A1 (en) Interactive method and apparatus in virtual reality scene
CN111103975B (en) Display method, electronic equipment and system
WO2023001113A1 (en) Display method and electronic device
CN108989784A (en) Image display method, device, equipment and the storage medium of virtual reality device
WO2023082980A1 (en) Display method and electronic device
WO2022233256A1 (en) Display method and electronic device
CN114416237A (en) Display state switching method, device and system, electronic equipment and storage medium
WO2023035911A1 (en) Display method and electronic device
WO2021057420A1 (en) Method for displaying control interface and head-mounted display
CN116934584A (en) Display method and electronic equipment
US12015758B1 (en) Holographic video sessions
EP3961572A1 (en) Image rendering system and method
WO2023116541A1 (en) Eye tracking apparatus, display device, and storage medium
US20230262406A1 (en) Visual content presentation with viewer position-based audio
EP4261768A1 (en) Image processing system and method

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22845278

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE