WO2020192458A1 - 一种图像处理的方法及头戴式显示设备 - Google Patents

一种图像处理的方法及头戴式显示设备 Download PDF

Info

Publication number
WO2020192458A1
WO2020192458A1 PCT/CN2020/079387 CN2020079387W WO2020192458A1 WO 2020192458 A1 WO2020192458 A1 WO 2020192458A1 CN 2020079387 W CN2020079387 W CN 2020079387W WO 2020192458 A1 WO2020192458 A1 WO 2020192458A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
camera
head
mounted display
display device
Prior art date
Application number
PCT/CN2020/079387
Other languages
English (en)
French (fr)
Inventor
梁天鹰
赖武军
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Priority to US17/442,907 priority Critical patent/US20220197033A1/en
Priority to EP20778172.5A priority patent/EP3920524A4/en
Priority to JP2021557122A priority patent/JP7408678B2/ja
Priority to KR1020217030380A priority patent/KR20210130773A/ko
Priority to AU2020250124A priority patent/AU2020250124B2/en
Publication of WO2020192458A1 publication Critical patent/WO2020192458A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/017—Head mounted
    • G02B27/0172—Head mounted characterised by optical features
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/222—Studio circuitry; Studio devices; Studio equipment
    • H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
    • H04N5/272—Means for inserting a foreground image in a background image, i.e. inlay, outlay
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/204—Image signal generators using stereoscopic image cameras
    • H04N13/239—Image signal generators using stereoscopic image cameras using two 2D image sensors having a relative position equal to or related to the interocular distance
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/204—Image signal generators using stereoscopic image cameras
    • H04N13/243—Image signal generators using stereoscopic image cameras using three or more 2D image sensors
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/296—Synchronisation thereof; Control thereof
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/30—Image reproducers
    • H04N13/332—Displays for viewing with the aid of special glasses or head-mounted displays [HMD]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/45—Cameras or camera modules comprising electronic image sensors; Control thereof for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/50—Constructional details
    • H04N23/51—Housings
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/63—Control of cameras or camera modules by using electronic viewfinders
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/80—Camera processing pipelines; Components thereof
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/90—Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/95—Computational photography systems, e.g. light-field imaging systems
    • H04N23/951—Computational photography systems, e.g. light-field imaging systems by using two or more images to influence resolution, frame rate or aspect ratio
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/222—Studio circuitry; Studio devices; Studio equipment
    • H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
    • H04N5/265—Mixing
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/0101—Head-up displays characterised by optical features
    • G02B2027/0138—Head-up displays characterised by optical features comprising image capture systems, e.g. camera
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/0101—Head-up displays characterised by optical features
    • G02B2027/014—Head-up displays characterised by optical features comprising information/image processing systems
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/0101—Head-up displays characterised by optical features
    • G02B2027/0147—Head-up displays characterised by optical features comprising a device modifying the resolution of the displayed image
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/017—Head mounted
    • G02B2027/0178—Eyeglass type
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/57—Mechanical or electrical details of cameras or camera modules specially adapted for being embedded in other devices
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/222—Studio circuitry; Studio devices; Studio equipment
    • H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
    • H04N5/2628—Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation

Definitions

  • This application relates to the field of communication technology, and in particular to an image processing method and head-mounted display device.
  • Augmented Reality is a technology that calculates the position and angle of the camera image in real time and adds corresponding images, videos, and 3D (Three Dimensional) models.
  • a typical head-mounted display device or head-mounted display (Head-mounted display, HMD)
  • HMD head-mounted display
  • the angular resolution of the image can reach 60 PPD (Pixels Per Degree) when the human eye perceives the best definition.
  • PPD Pixel Per Degree
  • none of the current head-mounted display devices can meet this resolution requirement. If the resolution of the image of the real scene captured by the camera is increased, the amount of data for image processing will be greatly increased, the complexity of image processing will be increased, and the display delay will be increased, and the user will be dizzy.
  • the image processing method and head-mounted display device provided by the present application can not only meet the definition requirements of users using the head-mounted display device to view images, but also help reduce the amount of data and reduce display delay.
  • this application provides an image processing method applied to a head-mounted display device.
  • Two modules are provided on the head-mounted display device.
  • Each module includes a camera module and a camera module corresponding to the camera module.
  • the camera module includes a first camera and a second camera. The field of view of the first camera is greater than the field of view of the second camera.
  • Each module performs the following method: in response to the detected opening operation, The first camera captures a first image with a first resolution, and the second camera captures a second image with a second resolution, where the first resolution is smaller than the second resolution; the processing module generates a fourth image based on the first image and the second image.
  • the fourth image is an image of the real scene, including the first resolution image and the second resolution image; the processing module generates a fifth image based on the fourth image and the third image containing the virtual object, and the fifth image includes the real Image of the scene and virtual objects; the display screen of the module displays the fifth image.
  • the field of view of the background area in the fifth image is large, which can provide users with a wider and wider field of view to satisfy the user’s field of view. Wider demand.
  • the image resolution of the central area of the fifth image is high, which can provide the user with a high-definition image within the focus range of the user's line of sight. It can be seen that the technical solution provided by this application is beneficial to enhance the user's immersion and realism, and enhance the user's visual experience.
  • the resolution of the background area in the fifth image is low, which can reduce the amount of data during image processing, reduce the complexity of calculation, and help reduce display Time delay.
  • the distance from the optical center of the first camera of one module to the optical center of the first camera of another module is 61 mm to 65 mm.
  • the distance between the first camera in the two camera modules is consistent with the pupil distance between the eyes of the user, which can simulate the real scene seen by the eyes of the user.
  • the distance between the optical center of the first camera and the optical center of the second camera in each module is less than 20 mm.
  • the second camera in the two camera modules is used to collect images of the user's visual focus area, so the second camera in the same camera module is as close as possible to the first camera.
  • each module further includes an eyepiece, and the display screen is located between the eyepiece and the camera module.
  • the optical center of the eyepiece of each module, the optical center of the display screen, and the optical center of the first camera are located on a straight line.
  • the large field of view camera in the left camera group and the large field of view camera in the right camera group are used to simulate the eyes of the user and shoot images of real scenes. Therefore, the optical centers of the two cameras are respectively aligned with the centers of the user's left and right eyes. And since the images of the real scene are finally presented through the first display screen and the second display screen, in order to make the user feel immersed and real, the optical centers of the first display screen and the second display screen should also be aligned with the left and right of the user. Align the centers of the eyes. Furthermore, the user sees the image on the display through the first eyepiece and the second eyepiece respectively. Therefore, the first eyepiece and the second eyepiece should also be aligned with the center of the user's left and right eyes respectively.
  • the processing module generating the fourth image according to the first image and the second image includes: the processing module adjusts the viewing angle of the second image to be the same as the viewing angle of the first image; according to the adjusted second image and The first image generates the fourth image.
  • the two images Align the angle of view in.
  • the processing module adjusting the viewing angle of the second image to be the same as the viewing angle of the first image includes: the processing module performs rotation, translation, and non-coplanar line calibration processing on the second image, so that the The viewing angle is the same as that of the first image.
  • the processing module generating the fourth image according to the adjusted second image and the first image includes: the processing module determines the area where the adjusted second image and the first image have the same angle of view as the target area; The image of the target area in the first image is replaced with the adjusted second image.
  • the image of the target area in the first image can be removed and then filled with the image of the target area in the second image after the angle of view is aligned.
  • a weighted fusion method can be used to fuse the two image synthesis edge positions (that is, the edge position of the target area), so that the final synthesized image (that is, the first Four images) are more natural and real.
  • the above-mentioned image synthesis method can use any one or several of direct cutting technology (cut-paste), alpha blending (alpha blending), multiband blending (multiband blending), Poisson blending (poisson blending), etc.
  • This application The specific method of image synthesis is not limited.
  • the method before the processing module generates the fifth image according to the fourth image and the third image containing the virtual object, the method further includes: the processing module according to the posture information of the head-mounted display device and the first image A third image containing the virtual object is generated.
  • generating the fifth image according to the fourth image and the third image containing the virtual object includes: adjusting the resolution of the third image according to the first resolution of the first image, and/or, according to the first resolution
  • the light information in the image adjusts the shadow of the virtual object in the third image; a fifth image is generated according to the fourth image and the adjusted third image.
  • this application provides a head-mounted display device, including: two modules and a memory, where each module includes a camera module, a display screen corresponding to the camera module, and a processing module.
  • the camera module It includes a first camera and a second camera. The field of view of the first camera is greater than the field of view of the second camera.
  • the memory is used to store computer program codes.
  • the computer program codes include computer instructions.
  • each module in the head-mounted display device performs the following operations: in response to detecting the opening operation, the first camera captures a first image with a first resolution, and the second camera captures a second image with a second resolution , Where the first resolution is smaller than the second resolution; the processing module generates a fourth image based on the first image and the second image, the fourth image is an image of the real scene, including the image of the first resolution and the image of the second resolution The processing module generates a fifth image based on the fourth image and the third image containing the virtual object, the fifth image includes the image of the real scene and the virtual object; the display screen of the module displays the fifth image.
  • the distance from the optical center of the first camera of one module to the optical center of the first camera of another module is 61 mm to 65 mm.
  • the distance between the optical center of the first camera and the optical center of the second camera in each module is less than 20 mm.
  • each module further includes an eyepiece, and the display screen is located between the eyepiece and the camera module.
  • the line connecting the optical center of the first camera in each module and the optical center of the second camera is parallel to the horizontal plane.
  • the optical center of the eyepiece of each module, the optical center of the display screen, and the optical center of the first camera are located on a straight line.
  • each module in the head-mounted display device when the processing module reads computer instructions from the memory, each module in the head-mounted display device is also caused to perform the following operations: the processing module adjusts the viewing angle of the second image to be the same as that of the first image. The viewing angle is the same; the fourth image is generated according to the adjusted second image and the first image.
  • each module in the head-mounted display device when the processing module reads computer instructions from the memory, each module in the head-mounted display device is also caused to perform the following operations: the processing module rotates, translates, and performs non-coplanar operations on the second image The calibration process makes the viewing angle of the second image the same as the viewing angle of the first image.
  • each module in the head-mounted display device when the processing module reads computer instructions from the memory, each module in the head-mounted display device is also caused to perform the following operations: the processing module determines the adjusted second image and the viewing angle in the first image The overlapping area is the target area; the image of the target area in the first image is replaced with the adjusted second image.
  • each module in the head-mounted display device is also made to perform the following operations: the processing module according to the posture information of the head-mounted display device, and the first The image generates a third image containing the virtual object.
  • each module in the head-mounted display device is also caused to perform the following operations: the processing module adjusts the third image according to the first resolution of the first image And/or adjust the shadow of the virtual object in the third image according to the light information in the first image; generate a fifth image according to the fourth image and the adjusted third image.
  • a computer storage medium includes computer instructions, which when the computer instructions run on a terminal, cause the terminal to execute the method described in the first aspect and any one of its possible implementation manners.
  • the fourth aspect is a computer program product.
  • the computer program product runs on a computer, the computer executes the method described in the first aspect and any one of the possible implementation manners.
  • 1A is a schematic diagram of field angles of different regions in an image provided by an embodiment of the application.
  • FIG. 1B is a schematic structural diagram of a head-mounted display device provided by an embodiment of the application.
  • FIG. 2A is a schematic structural diagram of a camera module in some head-mounted display devices provided by an embodiment of the application;
  • FIG. 2B is a schematic structural diagram of another head-mounted display device provided by an embodiment of the application.
  • FIG. 3 is a schematic structural diagram of another head-mounted display device provided by an embodiment of the application.
  • FIG. 4 is a schematic structural diagram of another head-mounted display device provided by an embodiment of the application.
  • FIG. 5 is a schematic structural diagram of another head-mounted display device provided by an embodiment of the application.
  • FIG. 6 is a schematic flowchart of an image processing method provided by an embodiment of this application.
  • FIG. 7 is a schematic diagram of an image processing process provided by an embodiment of the application.
  • first and second are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined with “first” and “second” may explicitly or implicitly include one or more of these features. In the description of the embodiments of the present application, unless otherwise specified, “plurality” means two or more.
  • words such as “exemplary” or “for example” are used as examples, illustrations, or illustrations. Any embodiment or design solution described as “exemplary” or “for example” in the embodiments of the present application should not be construed as being more preferable or advantageous than other embodiments or design solutions. To be precise, words such as “exemplary” or “for example” are used to present related concepts in a specific manner.
  • the image resolution of current head-mounted display devices generally cannot meet the user's sharpness requirements when viewing at close range, and simply increasing the image resolution will cause problems such as increased data volume and complexity of image processing.
  • the human eye has such visual characteristics: when the user observes, the area directly in front of the eye is usually the visual concentrated area. This part of the area has the most sensitive perception and requires high image resolution of the head-mounted display device. As for other areas visible to the eyes, the user's perception is not sharp, and the image resolution requirements of the head-mounted display device are not high. That is, the image displayed by the head-mounted display device can be divided into a central area and a background area. As shown in FIG. 1A, the image displayed on the head-mounted display device includes a central area and a background area.
  • the central area is the user's visual concentration area, which can provide a higher resolution image to meet the sharpness requirements of the user's eyes when viewing at a close distance.
  • the background area is a part of the image that is not in the central area, and only a lower resolution image can be provided to reduce the amount of data and complexity of image processing by the head-mounted display device.
  • the resolution of the central area is greater than the resolution of the background area.
  • the rectangle in Figure 1A is used to indicate the visual focus area (ie the central area) of the user’s eyes, which is mainly used to indicate the approximate position of the central area in the image displayed by the head-mounted display device, and does not constitute a
  • the shape of the visual focus area of the human eye is limited.
  • the traditional head-mounted display device has a small field of view, which will cause more blind spots on both sides of the user when using the head-mounted display device to observe, resulting in insufficient user immersion.
  • This application proposes to use a camera with a large field of view to capture an image with a larger field of view (an image including a background area), and the resolution of an image with a large field of view is generally lower.
  • a camera with a small field of view is used to capture high-definition images of the central area.
  • the field angle ⁇ corresponding to the background area is greater than the field angle ⁇ corresponding to the central area.
  • the field of view is used to indicate the maximum angle range that the camera can capture during the image capture process. If the object to be photographed is within this angle range, the object to be photographed will be captured by the camera and then presented in the preview image. If the object to be photographed is outside this angle range, the photographed device will not be captured by the image capturing device, that is, it will not appear in the preview image.
  • the larger the field of view of the camera the larger the shooting range and the shorter the focal length. The smaller the angle of view of the camera, the smaller the shooting range and the longer the focal length.
  • the angle of view includes the horizontal and vertical angles.
  • two viewing angles can be used to represent the horizontal and vertical angles respectively, or one viewing angle can also be used to represent the horizontal and vertical angles at the same time.
  • the following description of the present application is given by taking an angle of view indicating both the horizontal direction and the vertical direction.
  • the field of view ⁇ is the angle corresponding to the diagonals of the two edges of the image (ie, the image of the background area) captured by a camera with a large field of view, which can indicate both the horizontal direction and the vertical direction.
  • the angle of direction. ⁇ is the angle corresponding to the diagonals of the two edges of the image captured by the camera with a small angle of view (ie, the image in the central area), and can represent the angular width in the horizontal and vertical directions at the same time.
  • field of view may also be referred to as “field of view”, “field of view” and other terms. This article does not limit the name of "field of view”, as long as it expresses the above concept.
  • the image of the large field of view and the high-definition image of the central area are synthesized to obtain the image of the real scene.
  • the image containing the virtual object is generated according to the acquired posture information of the head-mounted display device and the image with the large field of view.
  • the image containing the virtual object and the image of the real scene obtained by the first synthesis are re-synthesized to obtain Images of real scenes and virtual objects are displayed on the head-mounted display device.
  • the image displayed at this time has a larger field of view angle, which can be as close as possible to the field of view of the human eye, and the image resolution in the concentrated area of the human eye is higher, which meets the sharpness requirements of the user when viewing images at close range. Increase the user’s immersion and enhance the visual experience.
  • the technical solutions provided in the embodiments of this application can be applied to head-mounted display devices in virtual reality (Virtual Reality, VR)/augmented reality (Augmented Reality, AR)/Mixed Reality (MR) scenarios.
  • VR Virtual Reality
  • AR Augmented Reality
  • MR Mated Reality
  • the specific form of the head-mounted display device implementing this technical solution is not particularly limited.
  • FIG. 1B it is a schematic structural diagram of a head-mounted display device 100 provided by an embodiment of this application.
  • the head-mounted display device 100 includes a helmet body 110 and a belt 140.
  • the strap 140 is used to wear the head-mounted display device 100 on the head of the user.
  • the helmet body 110 has a front panel 120 and a rear panel 130 opposite to the front panel 120.
  • the front panel 120 is a surface away from the user when the user wears the head-mounted display device 100.
  • the rear panel 130 is a surface close to the user when the user wears the head-mounted display device 100.
  • Two sets of camera modules are provided on the front panel 120, namely the left camera module 150 and the right camera module 160, for capturing images of the real scene in front of the user.
  • the left camera module 150 is located in front of the user's left eye
  • the right camera module 160 is located in front of the user's right eye.
  • Each pair of camera modules respectively includes at least two cameras, among the at least two cameras, there is at least one camera with a large field of view, and at least one camera with a small field of view.
  • At least one camera with a large field of view for example: the field of view is between 100 degrees and 200 degrees, and the resolution is between VGA and 720p), for example: camera 151 and camera 161; at least one camera with a small field of view (For example, the field of view is between 20 degrees and 60 degrees, and the resolution is between 720p and 2Kp), for example: camera 152 and camera 162.
  • the image captured by a camera with a large field of view has a larger field of view and a lower resolution.
  • a camera with a small angle of view captures an image with a smaller angle of view and higher resolution.
  • the large field of view camera captures an image with a larger field of view, which can provide users with a wider and wider field of view, and meet the needs of users for a wider field of view.
  • the small field of view camera captures images with higher resolution, which can provide users with images within the focus range of the user's line of sight.
  • the image captured by the camera with a large field of view is synthesized with the image captured by the camera with a small field of view, and applied to scenes such as AR/VR/MR, which is conducive to improving the user's immersion, realism, and improving the user Visual experience.
  • the image resolution of the background area is low, which can also reduce the amount of data during image processing, reduce the complexity of calculation, and help reduce the display delay.
  • the distance between the optical centers of the two large field of view cameras (for example, camera 151 and camera 161)
  • the distance D1 is approximately the same as the distance between the pupils of the user's eyes. Therefore, D1 is usually between 61 mm and 65 mm, for example, it may be 63 mm.
  • a camera with a small field of view is located near the camera with a large field of view.
  • the distance D2 between the optical center of the small field of view camera and the optical center of the large field of view camera is less than 20 mm. Please refer to FIG.
  • the camera module 150 directly in front of the left eye is taken as an example to illustrate the relative positions of the small field of view camera 152 and the large field of view camera 151.
  • the small field of view camera 152 can be located to the left of the large field of view camera 151 (that is, the small field of view camera 152, the larger field of view camera 151 is closer to the central axis, as shown in FIG.
  • the optical center of the small field of view camera 162 is roughly on a horizontal plane.
  • the small field of view camera 152 can also be located above the large field of view camera 151 (that is, the optical center of the small field of view camera 152 to large
  • the direction of the optical center of the field of view camera 151 is the same as the direction of gravity, as shown in (2) in FIG. 2A) or below (that is, the direction from the optical center of the small field of view camera 152 to the optical center of the large field of view camera 151 is the same as The direction of gravity is opposite).
  • the small field of view camera 152 may also be located at any position such as obliquely above or obliquely below the large field of view camera 151 (as shown in (3) in FIG.
  • this application does not limit the relative positional relationship between the small field angle camera and the large field angle camera in a camera module.
  • the small field of view camera can be fixed at a certain position near the large field of view.
  • the small field of view camera can also be located on a rotatable or movable device, that is, the small field of view camera can also move around the large field of view.
  • the above description uses the description of the small field of view camera 152 and the large field of view camera 151 respectively to apply to the small field of view camera 162 and the large field of view camera 161, which will not be repeated in this embodiment of the application.
  • the left and right camera modules respectively include two or more cameras with a large field of view
  • images of a real scene are synthesized or spliced from images captured by multiple cameras.
  • a large field of view camera can be set as the main camera in the left and right camera modules. That is, the angle of view of the image taken by the main camera is used as the reference to synthesize and splice images taken by other cameras with a large field of view to obtain an image of a real scene.
  • the distance between the optical centers of the two main cameras is D1, and it should be consistent with the distance between the pupils of the user's eyes.
  • a camera with a small field of view is located near the main camera in a camera with a large field of view.
  • the distance between the optical center of the small field of view camera and the optical center of the main camera is D2, and is less than 20mm.
  • the left and right camera modules of the head-mounted display device 100 respectively include two large-angle cameras. That is, the left camera module 150 includes two large field of view cameras, namely a camera 1511 and a camera 1512, and the right camera module 160 includes two large field of view cameras, a camera 1611 and a camera 1612, respectively.
  • the distance between the optical center of the camera 1511 and the optical center of the camera 1611 is D1.
  • the small field of view camera is located near the main camera in the large field of view camera. That is, the distance between the optical center of the camera 152 and the optical center of the camera 1511 is D2, and the distance between the optical center of the camera 162 and the optical center of the camera 1611 is D2.
  • the front panel 120 may also be provided with a sensor for sensing the movement of the head-mounted display device 100, such as an inertial measurement unit (IMU) 170 for measuring the three-axis of the head-mounted display device 100.
  • the posture information of the head-mounted display device 100 such as posture angle (or angular rate) and acceleration.
  • the IMU is composed of three accelerometers and three gyroscopes, and the accelerometers and gyroscopes are installed on mutually perpendicular measurement axes.
  • Low-precision IMUs can also be corrected in other ways. For example: GPS is used to correct long-term drift of position, barometer is used to correct altitude, magnetometer is used to correct attitude, etc.
  • the posture information of the head-mounted display device 100 obtained by the IMU 170 and the real scene image obtained by the camera module can be used to locate the real scene of the head-mounted display device 100 And composition.
  • an embodiment of the present application provides a top view of the head-mounted display device 100 when the user wears the head-mounted display device 100 as shown in FIG. 3.
  • Fig. 3 is an example of an example where the left and right camera modules respectively include a large field of view camera and a small field of view camera.
  • a left eyepiece 230 and a right eyepiece 240 respectively corresponding to the left and right eyes are provided on the rear panel 130 of the head-mounted display device 100.
  • a display corresponding to the left and right eyepieces is included inside the head-mounted display device 100. It is understandable that the field of view of the display is not less than the field of view of the large field of view camera in the camera module. In this way, the display can completely present the image taken by the large field of view camera.
  • the left eyepiece 230 corresponds to the left display 210
  • the right eyepiece 240 corresponds to the right display 220.
  • the left and right eyepieces can help human eyes focus on the left and right displays, so that the user's left eye can view the image displayed on the left display 210 through the left eyepiece 230, and the user's right eye can view the display on the right display 220 through the right eyepiece 240 Image.
  • the left display 210 and the right display 220 may be two independent displays, or may be different display areas in one display, which is not limited in the embodiment of the present application.
  • the front panel 120 of the head-mounted display device 100 is provided with a left camera module 150 corresponding to the user's left eye, including a large field of view camera 151 and a small field of view camera 152, as well as the user’s right
  • the right camera module corresponding to the eye includes a large field of view camera 161 and a small field of view camera 162.
  • An IMU 170 may also be provided on the front panel 120.
  • the two images captured by the two cameras of the left camera module 150 are processed by the method provided in this application, and then displayed on the left display 210 in front of the left eye for viewing by the left eye.
  • the two images captured by the two cameras of the right camera module 160 are processed by the method provided in this application and displayed on the right display 220 in front of the right eye for viewing by the right eye.
  • the cameras in the left camera module 150 and the right camera module 160 capture images at the same time, and due to the positional difference between the left camera module 150 and the right camera module 160, the images captured by the two modules are also There is a certain viewing angle difference, and the viewing angle difference is the same as the visual difference of a human face. Therefore, when the user's eyes view the two images from the left display 210 and the right display 220 at the same time, a 3D viewing angle effect will be produced.
  • the large field of view camera in the left camera group and the large field of view camera in the right camera group are used to simulate the eyes of the user and shoot images of real scenes. Therefore, the optical centers of the two cameras are respectively aligned with the centers of the user's left and right eyes. Since the images of the real scene are finally presented through the left display 210 and the right display 220, in order to make the user feel immersed and real, the optical centers of the left display 210 and the right display 220 should also be aligned with the centers of the user's left and right eyes, respectively. . Furthermore, in the head-mounted display device 100, the user sees the image on the display through the left eyepiece 230 and the right eyepiece 240, respectively.
  • the optical centers of the left eyepiece 230 and the right eyepiece 240 should also correspond to the left and right of the user. Align the centers of the eyes. As shown in FIG. 4, it shows a schematic diagram when a user uses a head-mounted display device to watch. Taking the right eye as an example, the center of the user’s right eye is aligned with the optical center point A of the right eyepiece 240, the optical center point B of the right display 220, and the optical center point C of the large field of view 161 in the right camera module. On a straight line, or approximately on a straight line.
  • FIG. 5 it is a schematic structural diagram of yet another head-mounted display device 100 provided by an embodiment of this application.
  • the head-mounted display device 100 may also include a processor 501, an external memory interface 502, an internal memory 503, a universal serial bus (USB) interface 504, a charging management module 505, a power management module 506, a battery 507, a wireless communication module 508, and audio Module 509, speaker 510, receiver 511, microphone 512, earphone interface 513, button 514, motor 515, indicator 516, etc.
  • a processor 501 an external memory interface 502, an internal memory 503, a universal serial bus (USB) interface 504, a charging management module 505, a power management module 506, a battery 507, a wireless communication module 508, and audio Module 509, speaker 510, receiver 511, microphone 512, earphone interface 513, button 514, motor 515, indicator 516, etc.
  • USB universal serial bus
  • the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the head-mounted display device 100.
  • the head-mounted display device 100 may include more or fewer components than shown, or combine some components, or split some components, or arrange different components.
  • the illustrated components can be implemented in hardware, software, or a combination of software and hardware.
  • the processor 501 includes one or more processing units.
  • the processor 501 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), and image signal processing.
  • the different processing units may be independent devices or integrated in one or more processors.
  • the controller can generate operation control signals according to the instruction operation code and timing signals to complete the control of fetching and executing instructions.
  • a memory may also be provided in the processor 501 to store instructions and data.
  • the memory in the processor 501 is a cache memory.
  • the memory can store instructions or data that the processor 501 has just used or recycled. If the processor 501 needs to use the instruction or data again, it can be directly called from the memory. Repeated accesses are avoided, the waiting time of the processor 501 is reduced, and the efficiency of the system is improved.
  • the processor 501 may include one or more interfaces.
  • the interface may include an integrated circuit (inter-integrated circuit, I2C) interface, an integrated circuit built-in audio (inter-integrated circuit sound, I2S) interface, a pulse code modulation (pulse code modulation, PCM) interface, and a universal asynchronous transmitter (universal asynchronous transmitter) interface.
  • I2C integrated circuit
  • I2S integrated circuit built-in audio
  • PCM pulse code modulation
  • PCM pulse code modulation
  • UART universal asynchronous transmitter
  • MIPI mobile industry processor interface
  • GPIO general-purpose input/output
  • SIM subscriber identity module
  • USB Universal Serial Bus
  • the interface connection relationship between the modules illustrated in the embodiment of the present invention is merely illustrative and does not constitute a structural limitation of the head-mounted display device 100.
  • the head-mounted display device 100 may also adopt different interface connection modes in the foregoing embodiments, or a combination of multiple interface connection modes.
  • the charging management module 505 is used to receive charging input from the charger.
  • the charger can be a wireless charger or a wired charger.
  • the charging management module 505 may receive the charging input of the wired charger through the USB interface 504.
  • the charging management module 505 may receive the wireless charging input through the wireless charging coil of the head-mounted display device 100. While charging the battery, the charging management module 505 can also supply power to the head-mounted display device through the power management module.
  • the power management module 506 is used to connect the battery 507, the charging management module 505 and the processor 501.
  • the power management module 506 receives input from the battery 507 and/or the charging management module 505, and supplies power to the processor 501, internal memory, the left display 210 and the right display 220, the camera module 150 and the camera module 160, and the wireless communication module.
  • the power management module 506 can also be used to monitor parameters such as the capacity of the battery 507, the number of cycles of the battery 507, and the health status (leakage, impedance) of the battery 507.
  • the power management module 506 may also be provided in the processor 501.
  • the power management module 506 and the charging management module 505 may also be provided in the same device.
  • the wireless communication module 508 can provide applications on the head-mounted display device 100 including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (bluetooth, BT), Global navigation satellite system (global navigation satellite system, GNSS), frequency modulation (frequency modulation, FM), near field communication technology (near field communication, NFC), infrared technology (infrared, IR) and other wireless communication solutions.
  • WLAN wireless local area networks
  • Bluetooth blue, BT
  • Global navigation satellite system global navigation satellite system
  • GNSS global navigation satellite system
  • frequency modulation frequency modulation, FM
  • near field communication technology near field communication technology
  • NFC near field communication technology
  • infrared technology infrared, IR
  • the wireless communication module 508 may be one or more devices integrating at least one communication processing module.
  • the wireless communication module 508 receives electromagnetic waves via an antenna, modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 501.
  • the wireless communication module 508 can also
  • the head-mounted display device 100 implements a display function through a GPU, a left display 210 and a right display 220, and an application processor.
  • the GPU is a microprocessor for image processing, connected to the left display 210 and the right display 220 and the application processor.
  • the GPU is used to perform mathematical and geometric calculations for graphics rendering.
  • the processor 501 may include one or more GPUs, which execute program instructions to generate or change display information.
  • the head-mounted display device 100 may be an all-in-one HMD, that is, the head-mounted display device 100 can use the CPU, GPU, NPU, application processor, etc. to compare the acquired image with the head-mounted display device.
  • Perform data processing on posture information of 100 such as: preprocessing and synthesizing the low-definition images captured by the large field of view camera and the high-definition images captured by the small field of view camera to obtain images of the real scene; generating images containing virtual objects; Combine the image containing the virtual object with the image of the real scene, etc.
  • the low-definition image captured by the large field of view camera is relative to the image captured by the small field of view camera, and the resolution of the image captured by the large field of view camera is lower than that of the small field of view camera.
  • the low-definition image captured by the large field of view camera meets the user's requirements for the resolution of the background area image when viewing the image using HDM.
  • the high-definition image captured by the small field of view camera is relative to the image captured by the large field of view camera, and the resolution of the image captured by the small field of view camera is higher than that of the small field of view camera.
  • the high-definition image captured by the small field of view camera meets the user's requirements for the resolution of the image in the center area when viewing the image using the HDM.
  • the head-mounted display device 100 may be a split HMD, that is, the head-mounted display device 100 may perform data processing on the acquired image and the posture information of the head-mounted display device 100. Hand over to other electronic equipment or devices in whole or in part.
  • the head-mounted display device 100 may send the acquired high-definition images and low-definition images, as well as the posture of the acquired head-mounted display device 100, to another electronic device (for example, : Mobile phones, personal computers, tablets, etc.).
  • the electronic device performs data processing, and then returns the processing result to the head-mounted display device 100, which is then displayed by the head-mounted display device 100.
  • the head-mounted display device 100 may not include any one or several hardware devices of GPU, NPU, and application processor.
  • the left display 210 is used to display images, videos, etc. corresponding to the left eye
  • the right display 220 is used to display images, videos, etc. corresponding to the right eye.
  • Both the left display 210 and the right display 220 may include a display panel.
  • the display panel can adopt liquid crystal display (LCD), organic light-emitting diode (OLED), active-matrix organic light-emitting diode or active-matrix organic light-emitting diode (active-matrix organic light-emitting diode).
  • LCD liquid crystal display
  • OLED organic light-emitting diode
  • active-matrix organic light-emitting diode active-matrix organic light-emitting diode
  • AMOLED flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, quantum dot light-emitting diode (QLED), etc.
  • the head-mounted display device 100 can implement a shooting function through an ISP, a camera module 150 and a camera module 160, a video codec, a GPU, a left display 210 and a right display 220, and an application processor.
  • the ISP is used to process the data fed back from the camera module 150 and the camera module 160. For example, when a camera module captures an image, light is transmitted to the photosensitive element of the camera through the lens, the light signal is converted into an electrical signal, and the photosensitive element of the camera transfers the electrical signal to the ISP for processing and is converted into an image visible to the naked eye. ISP can also optimize the image noise, brightness, and skin color. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP may be provided in the camera module 150 and the camera module 160.
  • Both the camera module 150 and the camera module 160 can be used to capture still images or videos.
  • the object generates an optical image through the lens and projects it to the photosensitive element.
  • the photosensitive element may be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor.
  • CMOS complementary metal-oxide-semiconductor
  • the photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to the ISP to convert it into a digital image signal.
  • ISP outputs digital image signals to DSP for processing.
  • DSP converts digital image signals into standard RGB, YUV and other formats.
  • the head-mounted display device 100 may include 1 or N camera modules 150, including 1 or M camera modules 160, and N and M are both positive integers greater than 1.
  • Digital signal processors are used to process digital signals. In addition to digital image signals, they can also process other digital signals. For example, when the head-mounted display device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the energy of the frequency point.
  • Video codecs are used to compress or decompress digital video.
  • the head-mounted display device 100 may support one or more video codecs. In this way, the head-mounted display device 100 can play or record videos in a variety of encoding formats, such as: moving picture experts group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.
  • MPEG moving picture experts group
  • MPEG2 MPEG2, MPEG3, MPEG4, etc.
  • NPU is a neural-network (NN) computing processor.
  • NN neural-network
  • applications such as intelligent cognition of the head-mounted display device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.
  • the external memory interface 502 may be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the head-mounted display device 100.
  • the external memory card communicates with the processor 501 through the external memory interface 502 to realize the data storage function. For example, save music, video and other files in an external memory card.
  • the internal memory 503 may be used to store computer executable program code, where the executable program code includes instructions.
  • the internal memory 503 may include a program storage area and a data storage area.
  • the storage program area can store an operating system, at least one application program (such as a sound playback function, an image playback function, etc.) required by at least one function.
  • the data storage area can store data (such as audio data, phone book, etc.) created during the use of the head-mounted display device 100.
  • the internal memory 503 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, a universal flash storage (UFS), and the like.
  • the processor 501 executes various functional applications and data processing of the head-mounted display device 100 by running instructions stored in the internal memory 503 and/or instructions stored in a memory provided in the processor 501.
  • the head-mounted display device 100 can implement audio functions through an audio module 509, a speaker 510, a receiver 511, a microphone 512, a headset interface 513, and an application processor. For example, music playback, recording, etc.
  • the audio module 509 is used to convert digital audio information into analog audio signals for output, and also used to convert analog audio inputs into digital audio signals.
  • the audio module 509 can also be used to encode and decode audio signals.
  • the audio module 509 may be provided in the processor 501, or some functional modules of the audio module 509 may be provided in the processor 501.
  • the speaker 510 also called a "speaker" is used to convert audio electrical signals into sound signals.
  • the head-mounted display device 100 can listen to music through the speaker 510 or listen to a hands-free call.
  • the receiver 511 also called “earpiece” is used to convert audio electrical signals into sound signals.
  • the head-mounted display device 100 answers a call or voice message, the voice can be heard by bringing the receiver 511 close to the human ear.
  • the microphone 512 also called “microphone” or “microphone”, is used to convert sound signals into electric signals. When collecting voice information, the user can approach the microphone 512 through the mouth to make a sound, and input the sound signal into the microphone 512.
  • the head-mounted display device 100 may be provided with at least one microphone 512. In other embodiments, the head-mounted display device 100 may be provided with two microphones 512, which can implement noise reduction functions in addition to collecting sound signals. In other embodiments, the head-mounted display device 100 may also be provided with three, four or more microphones 512 to collect sound signals, reduce noise, identify sound sources, and realize directional recording functions.
  • the earphone interface 513 is used to connect wired earphones.
  • the earphone interface 513 may be a USB interface, or a 3.5mm open mobile terminal platform (OMTP) standard interface, and a cellular telecommunications industry association (cellular telecommunications industry association of the USA, CTIA) standard interface.
  • OMTP open mobile terminal platform
  • CTIA cellular telecommunications industry association of the USA, CTIA
  • the button 514 includes a power button, a volume button, and so on.
  • the button 514 may be a mechanical button 514. It may also be a touch button 514.
  • the head-mounted display device 100 can receive key 514 input, and generate key signal input related to user settings and function control of the head-mounted display device 100.
  • the motor 515 can generate vibration prompts.
  • the motor 515 can be used for incoming call vibration notification, and can also be used for touch vibration feedback.
  • touch operations applied to different applications can correspond to different vibration feedback effects.
  • Acting on touch operations in different areas of the left display 210 and the right display 220, the motor 515 can also correspond to different vibration feedback effects.
  • Different application scenarios for example: time reminding, receiving information, alarm clock, games, etc.
  • the touch vibration feedback effect can also support customization.
  • the indicator 516 can be an indicator light, which can be used to indicate the charging status, power change, and can also be used to indicate messages, missed calls, notifications, and so on.
  • FIG. 6 a flowchart of an image processing method provided by an embodiment of this application specifically includes:
  • two first cameras can be used to simulate the eyes of the user (the first camera on the left corresponds to the left eye, and the first camera on the right corresponds to the right eye).
  • the optical centers of the two first cameras are aligned with the centers of the user's eyes, and the field of view of each first camera should be close to that of the human eye. Field angle.
  • the two first images simultaneously collected by the two first cameras can be considered to be the two images within the maximum range that the user's eyes can see, and can be used as images of the background area in the real scene that the user can see.
  • the two second cameras are used to capture images of the user’s visual focus area (the second camera on the left corresponds to the left eye, and the second camera on the right corresponds to the right eye), which is the center The image of the area.
  • the range of the user's visual concentration area is smaller than the maximum range seen by the user's eyes, and the definition of the visual concentration area is higher. Therefore, the field angle of the second camera is smaller than the field angle of the first camera, and the resolution of the second camera (ie, the second resolution) is greater than the resolution of the first camera (ie, the first resolution).
  • the field of view of the first camera can be between 100 degrees and 200 degrees, and the first camera
  • the resolution ie the first resolution
  • the angle of view of the second camera may be between 20 degrees and 60 degrees, and the resolution of the second camera (ie, the second resolution) is between 720p and 2Kp.
  • the above-mentioned user's opening operation can be the user opening an AR application, the user opening a certain service, or the camera opening operation, for example, the user clicking a certain control or pressing a certain
  • This is not limited in the embodiment of the present application, for example, a physical button, or input of a specific voice or predefined gesture, etc.
  • the head-mounted display device controls the two first cameras and the two second cameras to collect images at the same time to obtain two first images with a large field of view. Since the two first cameras only have a certain position difference in the horizontal direction, the two obtained first images only have a small viewing angle difference in the horizontal direction.
  • the shooting object in the first image includes a tree, a dog, and a ball, and is an image with the first resolution.
  • the second image collected by the second camera is part of the content of the first image, and it is the content in the central area of the first image, but the second image distinguishes The rate is higher than the first image.
  • the shooting object in the second image is a tree and is an image of the second resolution. It can be noticed that the image of the second resolution is marked with shading in FIG. 7 to distinguish the image of the first resolution, which will not be described below.
  • the head-mounted display device has two sets of camera modules (including a first camera and a second camera, respectively) corresponding to the two eyes of the user, and two displays. That is, the first camera and the second camera on the left collect images, and after image processing, they are displayed on the display on the left for the user's left eye to watch. The first camera and the second camera on the right collect images, and after image processing, they are displayed on the right display for the user's right eye to watch. Since the image processing on the left and right sides are the same, the image processing process on one side is used as an example for description, and the description is not repeated below.
  • the processing of the video can also be understood as the processing of multiple frames of images, so this application will not describe the method of video processing.
  • S102 Perform preprocessing on the first image and the second image respectively.
  • preprocessing includes processing such as distortion removal and white balance. Since the camera is composed of a set of lenses, the inherent characteristics of the lens will cause the captured image to be distorted. Therefore, it is necessary to perform de-distortion processing on the captured image according to the physical characteristics of each camera. In addition, due to the influence of ambient light and other factors, the original image captured by the camera will be distorted. Therefore, it is also necessary to perform processing such as white balance on the captured image. That is, the first image is preprocessed according to the physical characteristics of the first camera, and the second image is preprocessed according to the physical characteristics of the second camera. It should be noted that the preprocessing does not change the angle of view, resolution, and shooting objects in the first image and the second image.
  • both the image de-distortion processing and the white balance processing can adopt related technologies in the art, and details are not described herein again.
  • the preprocessed second image has the same subject as the preprocessed first image, but due to the different positions of the first camera and the second camera, there is a certain perspective difference between the images of the same subject in the two images , So you need to align the viewing angles in the two images.
  • the positional relationship between the first camera and the second camera in a camera module has been described in detail above. Therefore, the viewing angle difference of the two images can be determined according to the positional relationship, and the viewing angle of the second image after preprocessing can be determined according to the viewing angle difference. Adjust to be consistent with the viewing angle of the first image after preprocessing.
  • the process of adjusting the angle of view includes: rotation, translation, and non-coplanar calibration of the preprocessed second image. For the related method of adjusting the viewing angle, refer to the prior art, which will not be repeated here.
  • the head-mounted display device performs single-channel data extraction and down-sampling on the preprocessed first image, and combines the posture information obtained from the motion sensor (for example: IMU) (that is, the head-mounted display device collects the first image).
  • the posture information of an image using real-time positioning and mapping to construct a SLAM (Simultaneously Localization and Mapping) algorithm to calculate the pose information of the head-mounted display device, the depth information of the real environment, and the result of 3D reconstruction.
  • SLAM Simultaneously Localization and Mapping
  • a third image containing the virtual objects is generated.
  • the virtual object generated by the head-mounted display device is a robot, and an image containing the robot, that is, the third image, is generated according to the calculated pose information, environment depth information, and the result of three-dimensional reconstruction.
  • the virtual objects can also be adjusted according to the images of the real scene. For example: adapt the resolution of the virtual object to the resolution of the real scene image, adjust the position of the virtual object, and calculate the shadow of the virtual object according to the light conditions in the real scene.
  • the head-mounted display device determines an area where the angle of view overlaps in the preprocessed first image and the second image after the angle of view is aligned, as the target area. That is to say, the images in the target area in the two images are images of the same subject taken at the same time and angle. Image synthesis is performed on the preprocessed first image and the second image after the viewing angle is aligned to obtain a fourth image.
  • the image of the target area in the fourth image is the same as the image of the target area in the second image after the angle of view is aligned, which is the second resolution with higher resolution; the image of other areas in the fourth image except the target area It is the same as the image in the preprocessed first image, and is the first resolution with a lower resolution.
  • the resolution of the tree is the second resolution, except for the tree, for example, the resolution of the dog and the ball is the first resolution.
  • the target area in the fourth image is the user's visual concentration area, that is, the central area, and the resolution requirement is high.
  • the areas other than the target area in the fourth image are the aforementioned background areas, and the resolution requirements are lower, but the field of view angle is larger.
  • the image of the target area in the preprocessed first image may be removed, and then the image of the target area in the second image after the angle of view is aligned for filling.
  • a weighted fusion method can be used to fuse the two image synthesis edge positions (that is, the edge position of the target area), so that the final synthesized image (that is, the first Four images) are more natural and real.
  • the above-mentioned image synthesis method can use any one or several of direct cutting technology (cut-paste), alpha blending (alpha blending), multiband blending (multiband blending), Poisson blending (poisson blending), etc.
  • This application The specific method of image synthesis is not limited.
  • a feature point matching method may be used for determination. It should be noted that although the images of the target area in these two images were obtained by shooting the same subject at the same time, and the viewing angles are the same, because these two photos were taken by two cameras at different positions, factors such as light may cause The pixel values of the images in the target area in the two images are not completely the same, so the target area needs to be determined based on the two images.
  • the feature points in the two images are identified separately, and then the target area is determined by comparing the similarity of each feature point in the two images. That is, the area with similar feature points in the two images is determined to be the area with the same viewing angle, that is, the target area.
  • the feature point is one or more pixel points
  • the similarity of the feature points in the two images may mean that the pixel value of the feature point and the pixel values of the surrounding pixels of the feature point have a similar relationship. This is because although the pixel value of each pixel in the target area of the two images is different due to factors such as light, because the same subject is displayed in the target area, a certain pixel in the target area has a difference with the pixels around the pixel. Fixed relationship.
  • the light in image 1 is strong, and the whole leaf is brighter.
  • the light in image 2 is dark, and the whole leaf is dark. That is, the pixel value of the leaf in image 1 is different from the pixel value of the leaf in image 2.
  • the difference between the pixel values of different parts of the leaf in image 1 is the same as the difference between the pixel values of different parts of the leaf in image 2.
  • the method of identifying feature points can use any one or more of SIFT algorithm, SURF (Speeded Up Robust Features), FAST algorithm, ORB (ORiented Brief) algorithm, etc.
  • SIFT algorithm Scalable Up Robust Features
  • FAST algorithm FAST algorithm
  • ORB ORiented Brief
  • the positional relationship between the first camera and the second camera is fixed, the position of the target area in the first image after preprocessing is also fixed, and the target area in the second image after the angle of view is aligned is also fixed. Therefore, the position of the target area in the pre-processed first image and the position in the second image after the angle of view are aligned can be determined in advance, and then the image synthesis can be performed directly according to the position.
  • the embodiment of the present application does not specifically limit the method for determining the target area.
  • step S105 may be performed first, and then step S104; or, steps S104 and S105 may be performed at the same time, which is not limited in this application.
  • S106 Synthesize the fourth image and the third image to generate and display a fifth image, where the fifth image includes a real scene and a virtual object.
  • the fourth image is combined with the third image, so that the combined image (that is, the fifth image) contains the virtual object in the third image and also contains the real scene in the fourth image, that is, real and virtual Combine.
  • the position of the virtual object has been calculated when the third image is generated in step S104. Therefore, the image at the same position in the fourth image can be removed and filled with the virtual object in the third image.
  • the edge of the filling position is fused using a weighted fusion method to obtain the fifth image. That is, the processing is performed using the method of image synthesis introduced in step S105.
  • the map or texture of the virtual object may be pasted on the same position on the fourth image according to the position of the virtual object calculated in the foregoing steps.
  • a shader can be used to realize image synthesis. That is, the third image and the fourth image are used to jointly participate in rendering.
  • the specific rendering method can refer to the related prior art, which will not be repeated here.
  • the head-mounted display device can respectively display the combined real and virtual images obtained according to the above method on the left and right displays, and the images in the left and right displays have a certain viewing angle difference, so that the user has a 3D effect when viewing.
  • the image resolution of the background area is low, which can reduce the amount of data during image processing, reduce the complexity of calculation, and help reduce the display delay.
  • the above-mentioned terminal and the like include hardware structures and/or software modules corresponding to each function.
  • the embodiments of the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed by hardware or computer software-driven hardware depends on the specific application and design constraint conditions of the technical solution. Professionals and technicians can use different methods for each specific application to implement the described functions, but such implementation should not be considered as going beyond the scope of the embodiments of the present invention.
  • the embodiment of the present application may divide the above-mentioned terminal and the like into functional modules according to the above method examples.
  • each functional module may be divided corresponding to each function, or two or more functions may be integrated into one processing module.
  • the above-mentioned integrated modules can be implemented in the form of hardware or software functional modules. It should be noted that the division of modules in the embodiment of the present invention is illustrative, and is only a logical function division, and there may be other division methods in actual implementation.
  • the functional units in the various embodiments of the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units may be integrated into one unit.
  • the above-mentioned integrated unit can be implemented in the form of hardware or software functional unit.
  • the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium.
  • a computer readable storage medium includes a number of instructions to make a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor 501 execute all or part of the steps of the methods described in the various embodiments of the present application.
  • the aforementioned storage media include: flash memory, mobile hard disk, read-only memory, random access memory, magnetic disk or optical disk and other media that can store program codes.

Landscapes

  • Engineering & Computer Science (AREA)
  • Signal Processing (AREA)
  • Multimedia (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Optics & Photonics (AREA)
  • Human Computer Interaction (AREA)
  • Computing Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Controls And Circuits For Display Device (AREA)
  • Studio Devices (AREA)
  • User Interface Of Digital Computer (AREA)
  • Processing Or Creating Images (AREA)

Abstract

一种图像处理的方法及头戴式显示设备，涉及通信技术领域，通过为用户中心区域提供高分辨率图像，为背景区域提供低分辨率图像，满足头戴式显示设备的图像清晰度要求，且能减少图像的数据量，该方法可运用于HMD，该HMD包括两个模组，每个模组包括摄像头模组、与摄像头模组对应的显示屏以及处理模块，摄像头模组包括第一摄像头和第二摄像头，每个模组执行如下方法：响应于检测到的开启操作，第一摄像头采集低分辨率的第一图像，第二摄像头采集高分辨率的第二图像；处理模块根据第一图像和第二图像生成第四图像，所述第四图像包括低分辨率的图像和高分辨率的图像；处理模块根据第四图像和包含虚拟对象的第三图像生成第五图像，显示屏显示第五图像。

Description

一种图像处理的方法及头戴式显示设备
本申请要求在2019年3月25日提交中国国家知识产权局、申请号为201910228846.3、发明名称为“一种图像处理的方法及头戴式显示设备”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及通信技术领域，尤其涉及一种图像处理的方法及头戴式显示设备。
背景技术
增强现实(Augmented Reality，AR)是一种实时地计算摄影机影像的位置及角度并加上相应图像、视频、3D(Three Dimensional)模型的技术。其中，一种典型的基于视频透视的头戴式显示设备(或称为头戴式显示器(Head-mounted display，HMD)，可以通过摄像头捕获真实场景的图像，然后将捕获的图像与虚拟对象进行合成，并显示在头戴式显示设备的屏幕上。
由于用户佩戴头戴式显示设备时，头戴式显示设备的屏幕距离人眼很近，人眼感受到最佳清晰度时，图像的角度分辨率可以达到60 PPD(Pixels Per Degree)。然而，目前的头戴式显示设备均不能满足该分辨率的要求。若提升摄像头捕获的真实场景的图像的分辨率会大大增加图像处理的数据量、增大图像处理的复杂度，造成增大显示时延，以及用户眩晕等问题。
发明内容
本申请提供的一种图像处理的方法及头戴式显示设备，既能满足用户使用头戴式显示设备观看图像的清晰度要求，又有利于减少数据量，减少显示时延。
第一方面、本申请提供一种图像处理的方法，应用于头戴式显示设备，头戴式显示设备上设置有两个模组，每个模组包括摄像头模组、与摄像头模组对应的显示屏以及处理模块，摄像头模组包括第一摄像头和第二摄像头，第一摄像头的视场角大于第二摄像头的视场角，每个模组执行如下方法：响应于检测到的开启操作，第一摄像头采集第一分辨率的第一图像，第二摄像头采集第二分辨率的第二图像，其中第一分辨率小于第二分辨率；处理模块根据第一图像和第二图像生成第四图像，第四图像为真实场景的图像，包括第一分辨率的图像和第二分辨率的图像；处理模块根据第四图像和包含虚拟对象的第三图像生成第五图像，第五图像包括真实场景的图像和虚拟对象；模组的显示屏显示第五图像。
由此可见，由于第一摄像头采集的第一图像的视场角大，使得第五图像中背景区域的视场角大，可以为用户提供更宽更广的视角，满足用户人眼视场角较宽的需求。并且，由于第二摄像头采集的第二图像的第二分辨率大，使得第五图像的中心区域的图像分辨率高，可以为用户提供用户视线焦点范围内的高清晰度的图像。可见，本申请提供的技术方案有利于提升用户的沉浸感，真实感，提升用户视觉体验。
再有，由于第一摄像头采集的第一图像的第一分辨率低，使得第五图像中背景区域图像分辨率低，可以降低图像处理时的数据量，减少计算的复杂度，有利于减少显示时的时延。
一种可能的实现方式中，一个模组的第一摄像头的光学中心，到另一个模组的第一摄像头的光学中心的距离为61毫米至65毫米。
可见，用两个摄像模组中的第一摄像头间距与用户双眼瞳距一致，可以模拟用户双眼看到的真实场景。
一种可能的实现方式中，每个模组中第一摄像头的光学中心，到第二摄像头的光学中心的距离小于20毫米。
可见，用两个摄像模组中的第二摄像头采集用户视觉集中区域的图像，故同一个摄像模组中的第二摄像头尽量靠近第一摄像头。
一种可能的实现方式中，每个模组还包括一个目镜，且显示屏位于目镜和摄像头模组之间。
一种可能的实现方式中，每个模组的目镜的光学中心、显示屏的光学中心和第一摄像头的光学中心位于一条直线上。
本申请是用左摄像组中的大视场角摄像头与右摄像组中的大视场角摄像头用于模拟用户的双眼，拍摄真实场景的图像。因此，这两个摄像头的光学中心分别与用户的左右眼的中心对齐。又由于真实场景的图像最后通过第一显示屏和第二显示屏进行呈现的，为了使得用户产生沉浸感以及真实感，第一显示屏和第二显示屏的光学中心也应分别与用户的左右眼的中心对齐。再有，用户是分别通过第一目镜和第二目镜观看到显示器上的图像的，因此，第一目镜和第二目镜也应分别与用户的左右眼的中心对齐。
一种可能的实现方式中，处理模块根据第一图像和第二图像生成第四图像包括：处理模块将第二图像的视角调整为与第一图像的视角相同；根据调整后的第二图像和第一图像生成第四图像。
由于第二图像中和第一图像中具有相同的拍摄对象，但由于第一摄像头和第二摄像头的位置不同，两张图像中同一拍摄对象的图像存在一定的视角差，因此需要将两张图像中的视角对齐。
一种可能的实现方式中，处理模块将第二图像的视角调整为与第一图像的视角相同包括：处理模块将第二图像进行旋转、平移和非共面行校准处理，使得第二图像的视角与第一图像的视角相同。
一种可能的实现方式中，处理模块根据调整后的第二图像和第一图像生成第四图像包括：处理模块确定调整后的第二图像与第一图像中视角重合的区域为目标区域；将第一图像中的目标区域的图像更换为调整后的第二图像。
可以将第一图像中目标区域的图像去除，然后用对齐视角后的第二图像中目标区域的图像进行填充。在填充后，为了避免在图像合成过程中带来的锯齿感等问题，可以在两张图像合成边缘位置(即目标区域的边缘位置)使用加权融合的方法进行融合，使得最后合成图像(即第四图像)更加自然，真实。
上述图像合成的方法可以采用直接剪切技术(cut-paste)、Alpha融合(alpha blending)、多频段融合(multiband blending)、泊松融合(poisson blending)等任一种或任几种，本申请对图像合成的具体方法不做限定。
一种可能的实现方式中，在处理模块根据第四图像和包含虚拟对象的第三图像生成第五图像之前，该方法还包括：处理模块根据头戴式显示设备的姿态信息，以及第一图像生成包含虚拟对象的第三图像。
一种可能的实现方式中，根据第四图像和包含虚拟对象的第三图像生成第五图像包括：根据第一图像的第一分辨率调整第三图像的分辨率，和/或，根据第一图像中的光线信息调整第三图像中虚拟对象的阴影；根据第四图像，以及调整后的第三图像生成第五图像。
第二方面、本申请提供一种头戴式显示设备，包括：两个模组和存储器，其中，每个模组包括摄像头模组、与摄像头模组对应的显示屏以及处理模块，摄像头模组包括第一摄像头和第二摄像头，第一摄像头的视场角大于第二摄像头的视场角，存储器用于存储计算机程序代码，计算机程序代码包括计算机指令，当处理模块从存储器中读取计算机指令，以使得头戴式显示设备中的每个模组执行如下操作：响应于检测到开启操作，第一摄像头采集第一分辨率的第一图像，第二摄像头采集第二分辨率的第二图像，其中第一分辨率小于第二分辨率；处理模块根据第一图像和第二图像生成第四图像，第四图像为真实场景的图像，包括第一分辨率的图像和第二分辨率的图像；处理模块根据第四图像和包含虚拟对象的第三图像生成第五图像，第五图像包括真实场景的图像和虚拟对象；模组的显示屏显示第五图像。
一种可能的实现方式中，一个模组的第一摄像头的光学中心，到另一个模组的第一摄像头的光学中心的距离为61毫米至65毫米。
一种可能的实现方式中，每个模组中第一摄像头的光学中心，到第二摄像头的光学中心的距离小于20毫米。
一种可能的实现方式中，每个模组还包括一个目镜，且显示屏位于目镜和摄像头模组之间。
一种可能的实现方式中，头戴式显示设备水平放置时，每个模组中的第一摄像头的光学中心，与第二摄像头的光学中心的连线平行于水平面。
一种可能的实现方式中，每个模组的目镜的光学中心、显示屏的光学中心和第一摄像头的光学中心位于一条直线上。
一种可能的实现方式中，当处理模块从存储器中读取计算机指令，还使得头戴式显示设备中每个模组执行如下操作：处理模块将第二图像的视角调整为与第一图像的视角相同；根据调整后的第二图像和第一图像生成第四图像。
一种可能的实现方式中，当处理模块从存储器中读取计算机指令，还使得头戴式显示设备中每个模组执行如下操作：处理模块将第二图像进行旋转、平移和非共面行校准处理，使得第二图像的视角与第一图像的视角相同。
一种可能的实现方式中，当处理模块从存储器中读取计算机指令，还使得头戴式显示设备中每个模组执行如下操作：处理模块确定调整后的第二图像与第一图像中视角重合的区域为目标区域；将第一图像中的目标区域的图像更换为调整后的第二图像。
一种可能的实现方式中，当处理模块从存储器中读取计算机指令，还使得头戴式显示设备中每个模组执行如下操作：处理模块根据头戴式显示设备的姿态信息，以及第一图像生成包含虚拟对象的第三图像。
一种可能的实现方式中，当处理模块从存储器中读取计算机指令，还使得头戴式显示设备中每个模组执行如下操作：处理模块根据第一图像的第一分辨率调整第三图像的分辨率，和/或，根据第一图像中的光线信息调整第三图像中虚拟对象的阴影；根据第四图像，以及调整后的第三图像生成第五图像。
第三方面、一种计算机存储介质，包括计算机指令，当计算机指令在终端上运行时，使得终端执行如第一方面及其中任一种可能的实现方式中所述的方法。
第四方面、一种计算机程序产品，当计算机程序产品在计算机上运行时，使得计算机执行如第一方面中及其中任一种可能的实现方式中所述的方法。
附图说明
图1A为本申请实施例提供的一种图像中不同区域的视场角的示意图；
图1B为本申请实施例提供的一种头戴式显示设备的结构示意图；
图2A为本申请实施例提供的一些头戴式显示设备中摄像模组的结构示意图；
图2B为本申请实施例提供的又一种头戴式显示设备的结构示意图；
图3为本申请实施例提供的又一种头戴式显示设备的结构示意图；
图4为本申请实施例提供的又一种头戴式显示设备的结构示意图；
图5为本申请实施例提供的又一种头戴式显示设备的结构示意图；
图6为本申请实施例提供的一种图像处理的方法的流程示意图；
图7为本申请实施例提供的一种图像处理过程的示意图。
具体实施方式
下面将结合本申请实施例中的附图，对本申请实施例中的技术方案进行描述。其中，在本申请实施例的描述中，除非另有说明，“/”表示或的意思，例如，A/B可以表示A或B；本文中的“和/或”仅仅是一种描述关联对象的关联关系，表示可以存在三种关系，例如，A和/或B，可以表示：单独存在A，同时存在A和B，单独存在B这三种情况。
以下，术语“第一”、“第二”仅用于描述目的，而不能理解为指示或暗示相对重要性或者隐含指明所指示的技术特征的数量。由此，限定有“第一”、“第二”的特征可以明示或者隐含地包括一个或者更多个该特征。在本申请实施例的描述中，除非另有说明，“多个”的含义是两个或两个以上。
在本申请实施例中，“示例性的”或者“例如”等词用于表示作例子、例证或说明。本申请实施例中被描述为“示例性的”或者“例如”的任何实施例或设计方案不应被解释为比其它实施例或设计方案更优选或更具优势。确切而言，使用“示例性的”或者“例如”等词旨在以具体方式呈现相关概念。
目前的头戴式显示设备的图像分辨率通常不能满足用户近距离观看时的清晰度需求，而且单纯提升图像的分辨率会造成图像处理的数据量和复杂度增加等问题。考虑到人眼具有这样的视觉特点：用户在观察时，通常眼睛正前方的区域为视觉集中区域，这部分区域的感知最敏锐，对头戴式显示设备的图像分辨率要求高。而对于眼睛可视的其他区域，用户的感知不敏锐，对头戴式显示设备的图像分辨率要求不高。即，可以将头戴式显示设备显示的图像划分为中心区域和背景区域。如图1A所示，在头戴式显示设备显示的图像包括中心区域和背景区域。其中，中心区域为用户视觉集中区域，可以提供较高分辨率的图像，以达到用户人眼近距离观看时的清晰度要求。背景区域为图像中非中心区域的部分，可以仅提供较低的分辨率的图像，以减少头戴式显示设备处理图像的数据量和复杂度。换言之，中心区域的分辨率大于背景区域的分辨率。需要说明的是，图1A中以矩形来示意用户人眼的视觉集中区域(即中心区域)，主要用于表示中心区域位于头戴式显示设备显示的图像中的大致位置，并不构成对用户人眼的视觉集中区域的形状限制。
再有，考虑到人眼的视场角较大，可达到180度。而传统的头戴式显示设备的视场角较小，会造成用户使用头戴式显示设备观察时，两侧会出现较多的盲区，造成用户沉浸感不足。本申请提出了使用大视场角的摄像头捕获较大视场角的图像(包含背景区域的图像)，通常大视场角的图像的分辨率较低。同时使用小视场角的摄像头捕获中心区域的高清图像。如图1A所示的图像中，背景区域对应的视场角α大于中心区域对应的视场角β。
需要说明的是，视场角用于指示摄像头在拍摄图像的过程中，所能拍摄到的最大的角度 范围。若待拍摄物体处于这个角度范围内，该待拍摄物体便会被摄像头采集到，进而呈现在预览图像中。若待拍摄物体处于这个角度范围之外，该被拍摄设备便不会被图像捕捉装置采集到，即不会呈现在预览图像中。通常，摄像头的视场角越大，则拍摄范围就越大，焦距就越短。而摄像头的视场角越小，则拍摄范围就越小，焦距就越长。
一般而言，视场角包括水平方向和竖直方向的角度。在本申请中可以用两个视场角来分别表示水平方向和竖直方向的角度，也可以用一个视场角同时表示水平方向和竖直方向的角度。为了便于说明，本申请下文均以一个视场角同时表示水平方向和竖直方向的角度进行举例说明。例如：如图1A所示，视场角α为通过大视场角的摄像头捕捉到图像(即背景区域的图像)的两条边缘的对角线对应的角度，可以同时表示水平方向和竖直方向的角度。β为小视场角的摄像头捕捉到的图像(即中心区域的图像)的两条边缘的对角线对应的角度，可以同时表示水平方向和竖直方向的角宽度。
在本申请中，也可以将“视场角”称为“视场范围”、“视野范围”等词汇。本文对于“视场角”的名称不作限制，只要表达的是如上的概念即可。
而后，将大视场角的图像与中心区域的高清图像进行合成得到真实场景的图像。再根据获取的头戴式显示设备的姿态信息和大视场角的图像生成的包含虚拟对象的图像，最后将包含虚拟对象的图像和第一次合成得到的真实场景的图像再合成，得到包含真实场景和虚拟对象的图像，显示在头戴式显示设备中。此时显示的图像的视场角较大，可以尽量接近人眼的视场角，且在人眼视觉集中区域的图像分辨率较高，满足用户近距离观看图像时的清晰度要求，极大的增加了用户的沉浸感，提升视觉体验。
本申请实施例提供的技术方案可运用于虚拟现实(Virtual Reality，VR)/增强现实(Augmented Reality，AR)/混合现实(Mixed Reality，MR)等场景中头戴式显示设备等，本申请对执行该技术方案的头戴式显示设备的具体形式不做特殊限制。
如图1B所示，为本申请实施例提供的一种头戴式显示设备100的结构示意图。头戴式显示设备100包括头盔主体110，以及带具140。其中，带具140用于将头戴式显示设备100佩戴在用户的头部。头盔主体110具有前面板120，以及和前面板120相对的后面板130。其中，前面板120为用户佩戴头戴式显示设备100时，远离用户的表面。后面板130为用户佩戴头戴式显示设备100时，靠近用户的表面。
在前面板120上设置有两组摄像模组，分别为左摄像模组150和右摄像模组160，用于捕获用户前方真实场景的图像。其中，左摄像模组150位于用户左眼的前方，右摄像模组160位于用户右眼的前方。每对摄像模组都分别包含有至少两个摄像头，至少两个摄像头中有至少一个大视场角的摄像头，和至少一个小视场角的摄像头。其中，至少一个大视场角的摄像头(例如：视场角位于100度至200度之间，分辨率位于VGA到720p之间)，例如：摄像头151和摄像头161；至少一个小视场角的摄像头(例如：视场角位于20度至60度之间，分辨率位于720p到2Kp之间)，例如：摄像头152和摄像头162。其中，大视场角的摄像头捕获的图像视场角较大，分辨率较低。小视场角的摄像头捕获的图像视场角较小，分辨率较高。
在本申请中，大视场角摄像头捕获的图像视场角较大，可以为用户提供更宽更广的视角，满足用户人眼视场角较宽的需求。而小视场角摄像头捕获的图像分辨率较高，可以为用户提供用户视线焦点范围内的图像。可见，本申请中将大视场角摄像头捕获的图像，与小视场角摄像头捕获的图像进行合成，运用于AR/VR/MR等场景中，有利于提升用户的沉浸感，真实感，提升用户视觉体验。与此同时，背景区域的图像分辨率低，还可以降低图像处理时的数 据量，减少计算的复杂度，有利于减少显示时的时延。
在一些实施例中，左右摄像模组分别包含一个大视场角摄像头用于拍摄真实场景的图像时，这两个大视场角摄像头(例如：摄像头151和摄像头161)的光学中心之间的距离D1大致与用户两眼瞳距的间距一致，因此，D1通常位于61mm至65mm之间，例如可以为63mm。在一个摄像模组中，小视场角的摄像头位于大视场角摄像头的附近。通常小视场角摄像头的光学中心与大视场角摄像头光学中心的距离D2小于20mm。请参见图1B，图中示出了头戴式显示设备100的中轴线，头戴式设备大致水平的正向放置时。下面参考该中轴线，以左眼正前方的摄像模组150为例，说明小视场角摄像头152与大视场角摄像头151的相对位置。例如：面对前面板120，小视场角摄像头152可以位于大视场角摄像头151的左方(即小视场角摄像头152较大视场角摄像头151更靠近中轴线，如图1B所示)或右方(即小视场角摄像头152较大视场角摄像头151更远离中轴线，如图2A中(1)所示)，大视场角摄像头151，小视场角摄像头152以及大视场角摄像头161，小视场角摄像头162的光学中心大致在一个水平面。
再例如：当头戴式设备如图1B所示的大致水平的正向放置时，小视场角摄像头152也可以位于大视场角摄像头151的上方(即小视场角摄像头152的光学中心到大视场角摄像头151的光学中心的方向与重力方向相同，如图2A中(2)所示)或下方(即小视场角摄像头152的光学中心到大视场角摄像头151的光学中心的方向与重力方向相反)。小视场角摄像头152也可以位于大视场角摄像头151的斜上方(如图2A中(3)所示)或斜下方等任意位置。也就是说，本申请不限定一个摄像模组中小视场角摄像头与大视场角摄像头的相对位置关系。当然，小视场角摄像头可以固定在大视场角的附近的某个位置。小视场角摄像头也可以位于可转动或可移动的装置上，即，小视场角摄像头也可以在大视场角的附近进行移动。上述描述以小视场角摄像头152和大视场角摄像头151的说明分别适用于小视场角摄像头162和大视场角摄像头161，本申请实施例对此不再重复。
在另一些实施例中，左右摄像模组分别包含有两个或两个以上的大视场角摄像头时，即，真实场景的图像由多个摄像头捕获的图像进行合成或拼接得到。那么，一般可以在左右摄像模组中分别设置一个大视场角摄像头为主摄像头。即，用该主摄像头拍摄的图像的视角为基准合成和拼接其他大视场角摄像头拍摄的图像，得到真实场景的图像。那么，这两个主摄像头的光学中心之间的距离为D1，且应与用户两眼瞳距的间距保持一致。在一个摄像模组中，小视场角的摄像头位于大视场角摄像头中主摄像头的附近。通常小视场角摄像头的光学中心与主摄像头光学中心的距离为D2，且小于20mm。例如：如图2B所示，头戴式显示设备100的左右摄像模组中分别包含有两个大视场角摄像头。即，左摄像模组150包含两个大视场角摄像头，分别为摄像头1511和摄像头1512，右摄像模组160包含两个大视场角摄像头，分别为摄像头1611和摄像头1612。假设将靠近中轴线的两个大视场角的摄像头(即摄像头1511和摄像头1611)设置为主摄像头。那么，摄像头1511的光学中心和摄像头1611的光学中心之间的距离为D1。小视场角摄像头位于大视场角摄像头中主摄像头的附近。即摄像头152的光学中心与摄像头1511的光学中心之间的距离为D2，摄像头162的光学中心与摄像头1611的光学中心之间的距离为D2。
在又一些实施例中，前面板120上还可以设置有感知头戴式显示设备100运动的传感器，例如惯性测量单元(Inertial measurement unit，IMU)170，用于测量头戴式显示设备100三轴姿态角(或角速率)以及加速度等头戴式显示设备100的姿态信息。一般，IMU由三个加速度计和三个陀螺仪组成，加速度计和陀螺仪安装在相互垂直的测量轴上。低精度的IMU还可 以通过其他方式进行修正。例如：使用GPS修正位置的长期漂移，气压计用于修正高度，磁力计用于修正姿态等。在VR/AR/MR等场景中，IMU170获得的头戴式显示设备100的姿态信息，以及摄像模组获取的真实场景的图像，可用于对头戴式显示设备100所处的真实场景进行定位和构图。
为了更清楚的说明头戴式显示设备100的部件和各部件之间的位置关系，本申请实施例给出了如图3所示的用户佩戴头戴式显示设备100时的俯视图。图3是以左右摄像模组分别包含有一个大视场角摄像头和一个小视场角摄像头为例进行示例的。
在头戴式显示设备100的后面板130上设置有与左右眼分别对应的左目镜230和右目镜240。在头戴式显示设备100的内部，包含有与左右目镜分别对应的显示器。可理解的是，显示器的视场角为不小于摄像模组中大视场角摄像头的视场角，这样，显示器可以完全呈现大视场角摄像头拍摄的图像。其中，左目镜230对应的左显示器210，右目镜240对应的右显示器220。左右目镜可以帮助人眼对左右显示器进行对焦等，这样，用户的左眼可以通过左目镜230观看到左显示器210上显示的图像，用户的右眼可以通过右目镜240观看到右显示器220上显示的图像。需要说明的是，左显示器210和右显示器220可以是独立的两个显示器，也可以是一个显示器中不同的显示区域，本申请实施例对此均不做限定。
在图3中还可见，头戴式显示设备100的前面板120上设置有与用户左眼对应的左摄像模组150，包含大视场角摄像头151和小视场角摄像头152，以及与用户右眼对应的右摄像模组，包含大视场角摄像头161和小视场角摄像头162。前面板120上还可以设置有IMU170。
这样，左摄像模组150的两个摄像头捕获的两张图像，经过本申请提供的方法进行处理后，显示在左眼前方的左显示器210上，供左眼观看。右摄像模组160的两个摄像头捕获的两张图像，经过本申请提供的方法进行处理后，显示在右眼前方的右显示器220上，供右眼观看。一般而言，左摄像模组150与右摄像模组160中的摄像头同时捕获图像，且由于左摄像模组150和右摄像模组160的位置差，会造成两个模组同时捕获的画面也存在一定的视角差，而该视角差与人脸的视觉差相同。因此，当用户的双眼同时从左显示器210和右显示器220上观看到这两幅图像时，会产生3D的视角效果。
可见，本申请是用左摄像组中的大视场角摄像头与右摄像组中的大视场角摄像头用于模拟用户的双眼，拍摄真实场景的图像。因此，这两个摄像头的光学中心分别与用户的左右眼的中心对齐。又由于真实场景的图像最后通过左显示器210和右显示器220进行呈现的，为了使得用户产生沉浸感以及真实感，左显示器210和右显示器220的光学中心也应分别与用户的左右眼的中心对齐。再有，在头戴式显示设备100中，用户是分别通过左目镜230和右目镜240观看到显示器上的图像的，因此，左目镜230和右目镜240的光学中心也应分别与用户的左右眼的中心对齐。如图4所示，示出了用户使用头戴式显示设备观看时的示意图。以右眼为例，用户的右眼的中心、与右目镜240的光学中心A点、右显示器220的光学中心B点、右摄像模组中的大视场角161的光学中心C点在一条直线上，或者近似在一条直线上。
如图5所示，为本申请实施例提供的又一种头戴式显示设备100的结构示意图。头戴式显示设备100除了包含图1B至图3所示的部件(例如：左摄像模组150、右摄像模组160、左显示器210和右显示器220、IMU170等)外，头戴式显示设备100还可以包括处理器501，外部存储器接口502，内部存储器503，通用串行总线(universal serial bus，USB)接口504，充电管理模块505，电源管理模块506，电池507，无线通信模块508，音频模块509，扬声器510，受话器511，麦克风512，耳机接口513，按键514，马达515，指示器516等。
可以理解的是，本发明实施例示意的结构并不构成对头戴式显示设备100的具体限定。 在本申请另一些实施例中，头戴式显示设备100可以包括比图示更多或更少的部件，或者组合某些部件，或者拆分某些部件，或者不同的部件布置。图示的部件可以以硬件，软件或软件和硬件的组合实现。
其中，处理器501包括一个或多个处理单元，例如：处理器501可以包括应用处理器(application processor，AP)，调制解调处理器，图形处理器(graphics processing unit，GPU)，图像信号处理器(image signal processor，ISP)，控制器，视频编解码器，数字信号处理器(digital signal processor，DSP)，基带处理器，和/或神经网络处理器(neural-network processing unit，NPU)等。其中，不同的处理单元可以是独立的器件，也可以集成在一个或多个处理器中。
控制器可以根据指令操作码和时序信号，产生操作控制信号，完成取指令和执行指令的控制。
处理器501中还可以设置存储器，用于存储指令和数据。在一些实施例中，处理器501中的存储器为高速缓冲存储器。该存储器可以保存处理器501刚用过或循环使用的指令或数据。如果处理器501需要再次使用该指令或数据，可从所述存储器中直接调用。避免了重复存取，减少了处理器501的等待时间，因而提高了***的效率。
在一些实施例中，处理器501可以包括一个或多个接口。接口可以包括集成电路(inter-integrated circuit，I2C)接口，集成电路内置音频(inter-integrated circuit sound，I2S)接口，脉冲编码调制(pulse code modulation，PCM)接口，通用异步收发传输器(universal asynchronous receiver/transmitter，UART)接口，移动产业处理器接口(mobile industry processor interface，MIPI)，通用输入输出(general-purpose input/output，GPIO)接口，用户标识模块(subscriber identity module，SIM)接口，和/或通用串行总线(universal serial bus，USB)接口等。
可以理解的是，本发明实施例示意的各模块间的接口连接关系，只是示意性说明，并不构成对头戴式显示设备100的结构限定。在本申请另一些实施例中，头戴式显示设备100也可以采用上述实施例中不同的接口连接方式，或多种接口连接方式的组合。
充电管理模块505用于从充电器接收充电输入。其中，充电器可以是无线充电器，也可以是有线充电器。在一些有线充电的实施例中，充电管理模块505可以通过USB接口504接收有线充电器的充电输入。在一些无线充电的实施例中，充电管理模块505可以通过头戴式显示设备100的无线充电线圈接收无线充电输入。充电管理模块505为电池充电的同时，还可以通过电源管理模块为头戴式显示设备供电。
电源管理模块506用于连接电池507，充电管理模块505与处理器501。电源管理模块506接收电池507和/或充电管理模块505的输入，为处理器501，内部存储器，左显示器210和右显示器220，摄像模组150和摄像模组160，和无线通信模块等供电。电源管理模块506还可以用于监测电池507容量，电池507循环次数，电池507健康状态(漏电，阻抗)等参数。在其他一些实施例中，电源管理模块506也可以设置于处理器501中。在另一些实施例中，电源管理模块506和充电管理模块505也可以设置于同一个器件中。
无线通信模块508可以提供应用在头戴式显示设备100上的包括无线局域网(wireless local area networks，WLAN)(如无线保真(wireless fidelity，Wi-Fi)网络)，蓝牙(bluetooth，BT)，全球导航卫星***(global navigation satellite system，GNSS)，调频(frequency modulation，FM)，近距离无线通信技术(near field communication，NFC)，红外技术(infrared，IR)等无线通信的解决方案。无线通信模块508可以是集成至少一个通信处理模块的一个或多个器件。无线通信模块508经由天线接收电磁波，将电磁波信号调频以及滤波处理，将处理后的信号发送到处理器501。无线通信模块508还可以从处理器501接收待发送的信号，对其进行调频，放 大，经天线转为电磁波辐射出去。
头戴式显示设备100通过GPU，左显示器210和右显示器220，以及应用处理器等实现显示功能。GPU为图像处理的微处理器，连接左显示器210和右显示器220和应用处理器。GPU用于执行数学和几何计算，用于图形渲染。处理器501可包括一个或多个GPU，其执行程序指令以生成或改变显示信息。
在本申请的一些实施例中，头戴式显示设备100可以为一体式HMD，即头戴式显示设备100可以通过CPU、GPU、NPU、应用处理器等对获取的图像和头戴式显示设备100的姿态信息等进行数据处理，例如：对大视场角摄像头捕获的低清图像、小视场角摄像头捕获的高清图像进行预处理、图像合成得到真实场景的图像；生成包含虚拟对象的图像；将包含有虚拟对象的图像与真实场景的图像进行合成等。
其中，大视场角摄像头捕获的低清图像，是相对于小视场角摄像头捕获的图像而言，大视场角摄像头捕获图像的分辨率低于小视场角摄像头捕获图像的分辨率。大视场角摄像头捕获的低清图像满足用户使用HDM观看图像时，对背景区域图像的分辨率的要求。同理，小视场角摄像头捕获的高清图像，是相对于大视场角摄像头捕获的图像而言，小视场角摄像头捕获图像的分辨率高于小视场角摄像头捕获图像的分辨率。小视场角摄像头捕获的高清图像满足用户使用HDM观看图像时，对中心区域图像的分辨率的要求。
在本申请的又一些实施例中，头戴式显示设备100可以为分体式HMD，即头戴式显示设备100可以将对获取的图像和头戴式显示设备100的姿态信息等进行的数据处理全部或部分的交给其他电子设备或装置处理。举例来说，头戴式显示设备100可以将获取的高清图像和低清图像，以及获取头戴式显示设备100的姿态，通过例如USB接口504发送给与之连接的另一台电子设备(例如：手机、个人计算机、平板电脑等)上。由该电子设备进行数据处理，而后将处理结果返回给头戴式显示设备100，再由头戴式显示设备100进行显示。在该示例中，头戴式显示设备100可以不包含有GPU、NPU、应用处理器等中任一个或任几个硬件设备。
左显示器210用于显示对应左眼的图像，视频等，右显示器220用于显示对应右眼的图像，视频等。左显示器210和右显示器220均可以包括显示面板。显示面板可以采用液晶显示屏(liquid crystal display，LCD)，有机发光二极管(organic light-emitting diode，OLED)，有源矩阵有机发光二极体或主动矩阵有机发光二极体(active-matrix organic light emitting diode的，AMOLED)，柔性发光二极管(flex light-emitting diode，FLED)，Miniled，MicroLed，Micro-oLed，量子点发光二极管(quantum dot light emitting diodes，QLED)等。
头戴式显示设备100可以通过ISP，摄像模组150和摄像模组160，视频编解码器，GPU，左显示器210和右显示器220以及应用处理器等实现拍摄功能。
ISP用于处理摄像模组150和摄像模组160反馈的数据。例如，摄像模组在捕获图像时，光线通过镜头被传递到摄像头感光元件上，光信号转换为电信号，摄像头感光元件将所述电信号传递给ISP处理，转化为肉眼可见的图像。ISP还可以对图像的噪点，亮度，肤色进行算法优化。ISP还可以对拍摄场景的曝光，色温等参数优化。在一些实施例中，ISP可以设置在摄像模组150和摄像模组160中。
摄像模组150和摄像模组160均可用于捕获静态图像或视频。物体通过镜头生成光学图像投射到感光元件。感光元件可以是电荷耦合器件(charge coupled device，CCD)或互补金属氧化物半导体(complementary metal-oxide-semiconductor，CMOS)光电晶体管。感光元件把光信号转换成电信号，之后将电信号传递给ISP转换成数字图像信号。ISP将数字图像信号输出到 DSP加工处理。DSP将数字图像信号转换成标准的RGB，YUV等格式的图像信号。在一些实施例中，头戴式显示设备100可以包括1个或N个摄像模组150，包括1个或M个摄像模组160，N和M均为大于1的正整数。
数字信号处理器用于处理数字信号，除了可以处理数字图像信号，还可以处理其他数字信号。例如，当头戴式显示设备100在频点选择时，数字信号处理器用于对频点能量进行傅里叶变换等。
视频编解码器用于对数字视频压缩或解压缩。头戴式显示设备100可以支持一种或多种视频编解码器。这样，头戴式显示设备100可以播放或录制多种编码格式的视频，例如：动态图像专家组(moving picture experts group，MPEG)1，MPEG2，MPEG3，MPEG4等。
NPU为神经网络(neural-network，NN)计算处理器，通过借鉴生物神经网络结构，例如借鉴人脑神经元之间传递模式，对输入信息快速处理，还可以不断的自学习。通过NPU可以实现头戴式显示设备100的智能认知等应用，例如：图像识别，人脸识别，语音识别，文本理解等。
外部存储器接口502可以用于连接外部存储卡，例如Micro SD卡，实现扩展头戴式显示设备100的存储能力。外部存储卡通过外部存储器接口502与处理器501通信，实现数据存储功能。例如将音乐，视频等文件保存在外部存储卡中。
内部存储器503可以用于存储计算机可执行程序代码，所述可执行程序代码包括指令。内部存储器503可以包括存储程序区和存储数据区。其中，存储程序区可存储操作***，至少一个功能所需的应用程序(比如声音播放功能，图像播放功能等)等。存储数据区可存储头戴式显示设备100使用过程中所创建的数据(比如音频数据，电话本等)等。此外，内部存储器503可以包括高速随机存取存储器，还可以包括非易失性存储器，例如至少一个磁盘存储器件，闪存器件，通用闪存存储器(universal flash storage，UFS)等。处理器501通过运行存储在内部存储器503的指令，和/或存储在设置于处理器501中的存储器的指令，执行头戴式显示设备100的各种功能应用以及数据处理。
头戴式显示设备100可以通过音频模块509，扬声器510，受话器511，麦克风512，耳机接口513，以及应用处理器等实现音频功能。例如音乐播放，录音等。
音频模块509用于将数字音频信息转换成模拟音频信号输出，也用于将模拟音频输入转换为数字音频信号。音频模块509还可以用于对音频信号编码和解码。在一些实施例中，音频模块509可以设置于处理器501中，或将音频模块509的部分功能模块设置于处理器501中。
扬声器510，也称“喇叭”，用于将音频电信号转换为声音信号。头戴式显示设备100可以通过扬声器510收听音乐，或收听免提通话。
受话器511，也称“听筒”，用于将音频电信号转换成声音信号。当头戴式显示设备100接听电话或语音信息时，可以通过将受话器511靠近人耳接听语音。
麦克风512，也称“话筒”，“传声器”，用于将声音信号转换为电信号。当采集语音信息时，用户可以通过人嘴靠近麦克风512发声，将声音信号输入到麦克风512。头戴式显示设备100可以设置至少一个麦克风512。在另一些实施例中，头戴式显示设备100可以设置两个麦克风512，除了采集声音信号，还可以实现降噪功能。在另一些实施例中，头戴式显示设备100还可以设置三个，四个或更多麦克风512，实现采集声音信号，降噪，还可以识别声音来源，实现定向录音功能等。
耳机接口513用于连接有线耳机。耳机接口513可以是USB接口，也可以是3.5mm的 开放移动电子设备平台(open mobile terminal platform，OMTP)标准接口，美国蜂窝电信工业协会(cellular telecommunications industry association of the USA，CTIA)标准接口。
按键514包括开机键，音量键等。按键514可以是机械按键514。也可以是触摸式按键514。头戴式显示设备100可以接收按键514输入，产生与头戴式显示设备100的用户设置以及功能控制有关的键信号输入。
马达515可以产生振动提示。马达515可以用于来电振动提示，也可以用于触摸振动反馈。例如，作用于不同应用(例如拍照，音频播放等)的触摸操作，可以对应不同的振动反馈效果。作用于左显示器210和右显示器220不同区域的触摸操作，马达515也可对应不同的振动反馈效果。不同的应用场景(例如：时间提醒，接收信息，闹钟，游戏等)也可以对应不同的振动反馈效果。触摸振动反馈效果还可以支持自定义。
指示器516可以是指示灯，可以用于指示充电状态，电量变化，也可以用于指示消息，未接来电，通知等。
以下实施例中所涉及的技术方案均可以在图1B至图5所示的头戴式显示设备100中实现。
如图6所示，为本申请实施例提供的一种图像处理的方法的流程图，具体包括：
S101、响应于检测用户的开启操作，通过第一摄像头采集第一分辨率的第一图像，同时通过第二摄像头采集第二分辨率的第二图像。
在本申请实施例中，可以用两个第一摄像头模拟用户双眼(左侧的第一摄像头对应左眼，右侧的第一摄像头对应右眼)。为了采集用户双眼能够看到的最大范围内的图像或视频，这两个第一摄像头的光学中心分别与用户的双眼的中心对齐，并且每一个第一摄像头的视场角应接近人眼的视场角。这样，两个第一摄像头同时采集到的两幅第一图像可认为是用户双眼能够看到的最大范围内的两幅画面，可以作为用户能够看到的真实场景中背景区域的图像。在两个第一摄像头采集第一图像的同时，使用两个第二摄像头采集用户视觉集中区域的图像(左侧的第二摄像头对应左眼，右侧的第二摄像头对应右眼)，即中心区域的图像。上文已提到，用户视觉集中区域的范围小于用户双眼看到的最大范围，且视觉集中区域的清晰度要求较高。因此，第二摄像头的视场角小于第一摄像头的视场角，且第二摄像头的分辨率(即第二分辨率)大于第一摄像头的分辨率(即第一分辨率)。
根据人眼的视觉特点，确定背景区域和中心区域的视场角以及用户近距离观看时的清晰度要求，例如：第一摄像头的视场角可以位于100度到200度之间，第一摄像头的分辨率(即第一分辨率)位于VGA到720p之间。第二摄像头的视场角可以位于20度到60度之间，第二摄像头的分辨率(即第二分辨率)位于720p到2Kp之间。
上述用户的开启操作，例如可以是用户开启某个AR应用的操作，也可以是用户开启某个服务的操作，还可以是开启摄像头的操作等，例如可以是用户点击某个控件，或者按压某个物理按键，或者输入特定的语音或预定义的手势等，本申请实施例对此不做限定。
在检测到用户的开启操作后，头戴式显示设备控制两个第一摄像头和两个第二摄像头同时采集图像，得到两张大视场角的第一图像。由于两个第一摄像头仅在水平方向上有一定的位置差，所以得到的两张第一图像仅在水平方向上存在较小的视角差。例如：如图7所示，第一图像中的拍摄对象包含有树、狗、球，且为第一分辨率的图像。
由于第二摄像头的视场角小于第一摄像头，故第二摄像头采集得到的第二图像为第一图像中的一部分内容，且是位于第一图像中的中心区域的内容，但第二图像分辨率高于第一图像。例如：如图7所示，第二图像中的拍摄对象为树，且为第二分辨率的图像。可以注意到， 图7中用阴影来标记第二分辨率的图像，以区分第一分辨率的图像，以下不再说明。
需要说明的是，本申请实施例中，头戴式显示设备具有分别对应用户两眼的两组摄像模组(分别包含第一摄像头和第二摄像头)，以及两个显示器。即，左侧的第一摄像头和第二摄像头采集图像，并经过图像处理后，显示在左侧的显示器，供用户左眼观看。右侧的第一摄像头和第二摄像头采集图像，并经过图像处理后，显示在右侧的显示器，供用户右眼观看。由于左右两侧的图像处理是相同的，故下文以其中一侧的图像处理过程为例进行说明的，以下不再说明。
再有，由于摄像头拍摄的视频也是由连续多帧图像组成，因此对视频的处理也可理解为对多帧图像的处理，故本申请不再对视频处理的方法进行说明。
S102、分别对第一图像和第二图像进行预处理。
其中，预处理包括去畸变、白平衡等处理。由于摄像头是由一组透镜组合成的，透镜固有特性使得拍摄的图像会发生畸变，因此，需要根据各个摄像头本身的物理特性，对拍摄的得到的图像进行去畸变处理。又由于周围环境光等因素影响，使得摄像头拍摄的原始图像会出现失真，因此，还需要对拍摄的图像进行白平衡等处理。即，根据第一摄像头的物理特性，对第一图像进行预处理，根据第二摄像头的物理特性，对第二图像进行预处理。需要说明的是，预处理并不改变第一图像和第二图像中的视场角、分辨率以及拍摄对象等。
其中，对图像的去畸变处理和白平衡处理均可采用本领域相关技术，这里不再赘述。
S103、根据第二摄像头与第一摄像头的相对位置，将预处理后的第二图像的视角与预处理后的第一图像的视角对齐。
由于预处理后第二图像中和预处理后的第一图像中具有相同的拍摄对象，但由于第一摄像头和第二摄像头的位置不同，两张图像中同一拍摄对象的图像存在一定的视角差，因此需要将两张图像中的视角对齐。上文已详细阐述一个摄像模组内的第一摄像头和第二摄像头的位置关系，因此可以根据该位置关系确定两张图像的视角差，再根据视角差将预处理后的第二图像的视角调整为与预处理后的第一图像的视角一致。根据计算机图形学可知，调整视角的过程包括：对预处理后的第二图像的旋转、平移和非共面行校准等过程。相关调整视角的方法可参考现有技术，这里不再赘述。
需要说明的是，也可以先将第二图像与同一侧的第一图像的视角对齐，然后将对齐视角的第二图像进行预处理。本申请实施例不限定对齐视角的步骤和预处理步骤的前后顺序。
S104、获取头戴式显示设备的姿态信息，并根据预处理后的第一图像，生成包含虚拟对象的第三图像。
示例性的，头戴式显示设备对预处理后的第一图像进行单通道数据提取、下采样，并结合从运动传感器(例如：IMU)出获取的姿态信息(即头戴式显示设备采集第一图像时的姿态信息)，采用即时定位与地图构建SLAM(Simultaneously Localization and Mapping)算法，计算出头戴式显示设备的位姿信息、真实环境的深度信息以及三维重建的结果。再结合头戴式显示设备中相关应用(例如：AR应用)生成的虚拟对象，生成包含虚拟对象的第三图像。如图7所示，头戴式显示设备生成的虚拟对象为机器人，根据计算得到位姿信息、环境深度信息以及三维重建的结果生成包含机器人的图像，即第三图像。
另外，为了使得相关应用单独生成的虚拟对象与真实场景能够很自然的融合，避免使得用户产生突兀感和不真实感，还可以根据真实场景的图像对虚拟对象进行调整。例如：将虚拟对象的分辨率与真实场景图像的分辨率进行适配，对虚拟物体的位置进行调整、根据真实场景中光线情况计算虚拟对象的阴影等。
S105、将预处理后的第一图像和对齐视角后的第二图像合成为真实场景的第四图像，其中，第四图像包括第一分辨率图像和第二分辨率的图像。
示例性的，头戴式显示设备确定预处理后的第一图像和对齐视角后的第二图像中视角重合的区域，为目标区域。也就是说，两张图像中目标区域内的图像为同一时间同一视角拍摄相同拍摄对象的图像。将预处理后的第一图像和视角对齐后的第二图像进行图像合成，得到第四图像。其中，第四图像中目标区域的图像与视角对齐后的第二图像中目标区域的图像相同，为分辨率较高的第二分辨率；第四图像中除目标区域之外的其他区域的图像与预处理后的第一图像中的图像相同，为分辨率较低的第一分辨率。例如：如图7所示，在第四图像中，树的分辨率为第二分辨率，除树之外，例如狗和球的分辨率为第一分辨率。
可以理解的是，第四图像中的目标区域即为用户视觉集中区域，即中心区域，分辨率要求较高。第四图像中除目标区域之外的其他区域为上述背景区域，分辨率要求较低，但视场角较大。
示例性的，上述图像合成的过程，可以将预处理后的第一图像中目标区域的图像去除，然后用对齐视角后的第二图像中目标区域的图像进行填充。在填充后，为了避免在图像合成过程中带来的锯齿感等问题，可以在两张图像合成边缘位置(即目标区域的边缘位置)使用加权融合的方法进行融合，使得最后合成图像(即第四图像)更加自然，真实。
上述图像合成的方法可以采用直接剪切技术(cut-paste)、Alpha融合(alpha blending)、多频段融合(multiband blending)、泊松融合(poisson blending)等任一种或任几种，本申请对图像合成的具体方法不做限定。
示例性的，在确定目标区域时，可以采用特征点匹配的方法进行确定。需要说明的是，虽然这两张图像中目标区域的图像为同一时间拍摄同一拍摄对象得到的，且视角相同，但由于这两张照片是两个不同位置的摄像头拍摄的，光线等因素会造成两张图像中目标区域内的图像的像素值不完全一致，因此需要根据这两张图像确定出目标区域。
简单来说，分别对两张图像中进行特征点识别，然后通过比对两张图像中各个特征点的相似性来确定目标区域。即，确定两张图像中特征点相似的区域为视角相同的区域，即目标区域。其中，特征点为一个或多个像素点，两张图像中特征点相似可以是指特征点的像素值以及特征点周围像素的像素值具有相似的关系。这是因为，虽然由于光线等因素使得两张图像目标区域中各个像素的像素值不同，但由于目标区域内显示的是同一拍摄对象，所以目标区域内某个像素，与该像素周围的像素具有固定的关系。例如：同一时间拍摄的同一片树叶，图像1中光线较强，整片叶子都偏亮。图像2中光线较暗，整片叶子都偏暗。即，图像1中叶子的像素值与图像2中叶子的像素值不同。但，在图像1中叶子不同部位的像素值之间的差值与图像2中叶子不同部位的像素值之的差值相同。
其中，特征点识别的方法可以采用SIFT算法、SURF(Speeded Up Robust Features)、FAST算法、ORB(ORiented Brief)算法等的任一种或任几种，本申请实施例对识别特征点的具体方法不做限定。
当然，也可以采用其他方法来确定目标区域。例如：由于第一摄像头和第二摄像头的位置关系固定，那么预处理后第一图像中该目标区域的位置也是固定的，对齐视角后的第二图像中的目标区域也是固定的。因此，可以预先确定出该目标区域在预处理后的第一图像中的位置，以及在对齐视角后的第二图像中的位置，而后直接根据位置进行图像合成。本申请实施例对确定目标区域的方法也不做具体限定。
需要说明的是，本申请实施例也可以先执行步骤S105，再执行步骤S104；或者，同时执 行步骤S104和S105，本申请不做限定。
S106、将第四图像与第三图像合成生成并显示第五图像，第五图像包含真实场景和虚拟对象。
具体的，将第四图像与第三图像进行图像合成，使得合成后的图像(即第五图像)包含第三图像中的虚拟对象，也包含第四图像中真实场景，即实现现实和虚拟的结合。
示例性的，在第三图像和第四图像进行图像合成的过程中，由于在步骤S104中生成第三图像时，已计算出虚拟对象的位置。因此可以将第四图像中相同位置的图像去除，用第三图像中的虚拟对象进行填充。在填充位置的边缘使用加权融合的方法进行融合，得到第五图像。即，采用步骤S105中介绍的图像合成的方法进行处理。
或者，在第三图像和第四图像进行图像合成的过程中，也可以根据前述步骤计算出的虚拟对象的位置，在第四图像上相同位置上贴上虚拟对象的贴图或纹理。例如可以采用着色程序(shader)实现图像的合成。也即，使用第三图像和第四图像共同参与渲染。具体的渲染方式可参考相关的现有技术，这里不再赘述。
需要说明的是，本申请实施例不限定具体的图像合成的具体方法。
由此可见，头戴式显示设备可以在左右显示器中分别显示根据上述方法得到的现实和虚拟结合的图像，左右显示器中的图像具有一定视角差，使得用户观看时有3D的效果。
又由于显示的图像中背景区域的视场角大，可以为用户提供更宽更广的视角，满足用户人眼视场角较宽的需求。并且，中心区域的图像分辨率高，可以为用户提供用户视线焦点范围内的高清晰度的图像。可见，本申请提供的技术方案有利于提升用户的沉浸感，真实感，提升用户视觉体验。
与此同时，背景区域的图像分辨率低，可以降低图像处理时的数据量，减少计算的复杂度，有利于减少显示时的时延。
可以理解的是，上述终端等为了实现上述功能，其包含了执行各个功能相应的硬件结构和/或软件模块。本领域技术人员应该很容易意识到，结合本文中所公开的实施例描述的各示例的单元及算法步骤，本申请实施例能够以硬件或硬件和计算机软件的结合形式来实现。某个功能究竟以硬件还是计算机软件驱动硬件的方式来执行，取决于技术方案的特定应用和设计约束条件。专业技术人员可以对每个特定的应用来使用不同方法来实现所描述的功能，但是这种实现不应认为超出本发明实施例的范围。
本申请实施例可以根据上述方法示例对上述终端等进行功能模块的划分，例如，可以对应各个功能划分各个功能模块，也可以将两个或两个以上的功能集成在一个处理模块中。上述集成的模块既可以采用硬件的形式实现，也可以采用软件功能模块的形式实现。需要说明的是，本发明实施例中对模块的划分是示意性的，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式。
通过以上的实施方式的描述，所属领域的技术人员可以清楚地了解到，为描述的方便和简洁，仅以上述各功能模块的划分进行举例说明，实际应用中，可以根据需要而将上述功能分配由不同的功能模块完成，即将装置的内部结构划分成不同的功能模块，以完成以上描述的全部或者部分功能。上述描述的***，装置和单元的具体工作过程，可以参考前述方法实施例中的对应过程，在此不再赘述。
在本申请实施例各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。上述集成的单元既可以采用硬件的形式实现，也可以采用软件功能单元的形式实现。
所述集成的单元如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个计算机可读取存储介质中。基于这样的理解，本申请实施例的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的全部或部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)或处理器501执行本申请各个实施例所述方法的全部或部分步骤。而前述的存储介质包括：快闪存储器、移动硬盘、只读存储器、随机存取存储器、磁碟或者光盘等各种可以存储程序代码的介质。
以上所述，仅为本申请的具体实施方式，但本申请的保护范围并不局限于此，任何在本申请揭露的技术范围内的变化或替换，都应涵盖在本申请的保护范围之内。因此，本申请的保护范围应以所述权利要求的保护范围为准。

Claims (23)

  1. 一种图像处理的方法，应用于头戴式显示设备，其特征在于，所述头戴式显示设备上设置有两个模组，每个所述模组包括摄像头模组、与所述摄像头模组对应的显示屏以及处理模块，所述摄像头模组包括第一摄像头和第二摄像头，所述第一摄像头的视场角大于所述第二摄像头的视场角，每个所述模组执行如下方法：
    响应于检测到的开启操作，所述第一摄像头采集第一分辨率的第一图像，所述第二摄像头采集第二分辨率的第二图像，其中所述第一分辨率小于所述第二分辨率；
    所述处理模块根据所述第一图像和所述第二图像生成第四图像，所述第四图像为真实场景的图像，包括所述第一分辨率的图像和所述第二分辨率的图像；
    所述处理模块根据所述第四图像和包含虚拟对象的第三图像生成第五图像，所述第五图像包括所述真实场景的图像和所述虚拟对象；
    所述模组的显示屏显示所述第五图像。
  2. 根据权利要求1所述的方法，其特征在于，一个所述模组的第一摄像头的光学中心，到另一个所述模组的第一摄像头的光学中心的距离为61毫米至65毫米。
  3. 根据权利要求2所述的方法，其特征在于，每个所述模组中第一摄像头的光学中心，到第二摄像头的光学中心的距离小于20毫米。
  4. 根据权利要求1-3任一项所述的方法，其特征在于，每个所述模组还包括一个目镜，且所述显示屏位于所述目镜和所述摄像头模组之间。
  5. 根据权利要求4所述的方法，其特征在于，每个所述模组的所述目镜的光学中心、所述显示屏的光学中心和所述第一摄像头的光学中心位于一条直线上。
  6. 根据权利要求1-5任一项所述的方法，其特征在于，所述处理模块根据所述第一图像和所述第二图像生成第四图像包括：
    所述处理模块将所述第二图像的视角调整为与所述第一图像的视角相同；
    根据调整后的所述第二图像和所述第一图像生成所述第四图像。
  7. 根据权利要求6所述的方法，其特征在于，所述处理模块将所述第二图像的视角调整为与所述第一图像的视角相同包括：
    所述处理模块将所述第二图像进行旋转、平移和非共面行校准处理，使得所述第二图像的视角与所述第一图像的视角相同。
  8. 根据权利要求6或7所述的方法，其特征在于，所述处理模块根据调整后的所述第二图像和所述第一图像生成所述第四图像包括：
    所述处理模块确定调整后的所述第二图像与所述第一图像中视角重合的区域为目标区域；
    将所述第一图像中的所述目标区域的图像更换为调整后的所述第二图像。
  9. 根据权利要求1-8任一项所述的方法，其特征在于，在所述处理模块根据所述第四图像和包含虚拟对象的第三图像生成第五图像之前，所述方法还包括：
    所述处理模块根据所述头戴式显示设备的姿态信息，以及所述第一图像生成包含虚拟对象的所述第三图像。
  10. 根据权利要求1-9任一项所述的方法，其特征在于，所述处理模块根据所述第四图像和包含虚拟对象的第三图像生成第五图像包括：
    所述处理模块根据所述第一图像的所述第一分辨率调整所述第三图像的分辨率，和/或，根据所述第一图像中的光线信息调整所述第三图像中所述虚拟对象的阴影；
    根据所述第四图像，以及调整后的所述第三图像生成所述第五图像。
  11. 一种头戴式显示设备，其特征在于，包括：两个模组和存储器，其中，每个所述模组包括摄像头模组、与所述摄像头模组对应的显示屏以及处理模块，所述摄像头模组包括第一摄像头和第二摄像头，所述第一摄像头的视场角大于所述第二摄像头的视场角，所述存储器用于存储计算机程序代码，所述计算机程序代码包括计算机指令，当所述处理模块从所述存储器中读取所述计算机指令，以使得所述头戴式显示设备中每个所述模组执行如下操作：
    响应于检测到开启操作，所述第一摄像头采集第一分辨率的第一图像，所述第二摄像头采集第二分辨率的第二图像，其中所述第一分辨率小于所述第二分辨率；
    所述处理模块根据所述第一图像和所述第二图像生成第四图像，所述第四图像为真实场景的图像，包括所述第一分辨率的图像和所述第二分辨率的图像；
    所述处理模块根据所述第四图像和包含虚拟对象的第三图像生成第五图像，所述第五图像包括所述真实场景的图像和所述虚拟对象；
    所述模组的显示屏显示所述第五图像。
  12. 根据权利要求11所述的头戴式显示设备，其特征在于，一个所述模组的第一摄像头的光学中心，到另一个所述模组的第一摄像头的光学中心的距离为61毫米至65毫米。
  13. 根据权利要求12所述的头戴式显示设备，其特征在于，每个所述模组中第一摄像头的光学中心，到第二摄像头的光学中心的距离小于20毫米。
  14. 根据权利要求11-13任一项所述的头戴式显示设备，其特征在于，每个所述模组还包括一个目镜，且所述显示屏位于所述目镜和所述摄像头模组之间。
  15. 根据权利要求14所述的头戴式显示设备，其特征在于，所述头戴式显示设备水平放置时，每个所述模组中的第一摄像头的光学中心，与所述第二摄像头的光学中心的连线平行于水平面。
  16. 根据权利要求14或15所述的头戴式显示设备，其特征在于，每个所述模组的所述目镜的光学中心、所述显示屏的光学中心和所述第一摄像头的光学中心位于一条直线上。
  17. 根据权利要求11-16任一项所述的头戴式显示设备，其特征在于，当所述处理模块从所述存储器中读取所述计算机指令，还使得所述头戴式显示设备中每个所述模组执行如下操作：
    所述处理模块将所述第二图像的视角调整为与所述第一图像的视角相同；
    根据调整后的所述第二图像和所述第一图像生成所述第四图像。
  18. 根据权利要求17所述的头戴式显示设备，其特征在于，当所述处理模块从所述存储器中读取所述计算机指令，还使得所述头戴式显示设备中每个所述模组执行如下操作：
    所述处理模块将所述第二图像进行旋转、平移和非共面行校准处理，使得所述第二图像的视角与所述第一图像的视角相同。
  19. 根据权利要求17或18所述的头戴式显示设备，其特征在于，当所述处理模块从所述存储器中读取所述计算机指令，还使得所述头戴式显示设备中每个所述模组执行如下操作：
    所述处理模块确定调整后的所述第二图像与所述第一图像中视角重合的区域为目标区域；
    将所述第一图像中的所述目标区域的图像更换为调整后的所述第二图像。
  20. 根据权利要求11-19任一项所述的头戴式显示设备，其特征在于，当所述处理模块从所述存储器中读取所述计算机指令，还使得所述头戴式显示设备中每个所述模组执行如下操作：
    所述处理模块根据所述头戴式显示设备的姿态信息，以及所述第一图像生成包含虚拟对象的所述第三图像。
  21. 根据权利要求11-20任一项所述的头戴式显示设备，其特征在于，当所述处理模块从所述存储器中读取所述计算机指令，还使得所述头戴式显示设备中每个所述模组执行如下操作：
    所述处理模块根据所述第一图像的所述第一分辨率调整所述第三图像的分辨率，和/或，根据所述第一图像中的光线信息调整所述第三图像中所述虚拟对象的阴影；
    根据所述第四图像，以及调整后的所述第三图像生成所述第五图像。
  22. 一种计算机存储介质，其特征在于，包括计算机指令，当所述计算机指令在终端上运行时，使得所述终端执行如权利要求1-10中任一项所述图像处理的方法。
  23. 一种计算机程序产品，其特征在于，当所述计算机程序产品在计算机上运行时，使得所述计算机执行如权利要求1-10中任一项所述图像处理的方法。
PCT/CN2020/079387 2019-03-25 2020-03-14 一种图像处理的方法及头戴式显示设备 WO2020192458A1 (zh)

Priority Applications (5)

Application Number Priority Date Filing Date Title
US17/442,907 US20220197033A1 (en) 2019-03-25 2020-03-14 Image Processing Method and Head Mounted Display Device
EP20778172.5A EP3920524A4 (en) 2019-03-25 2020-03-14 IMAGE DISPLAY METHOD AND VISUAL HEAD DEVICE
JP2021557122A JP7408678B2 (ja) 2019-03-25 2020-03-14 画像処理方法およびヘッドマウントディスプレイデバイス
KR1020217030380A KR20210130773A (ko) 2019-03-25 2020-03-14 이미지 처리 방법 및 머리 장착형 디스플레이 디바이스
AU2020250124A AU2020250124B2 (en) 2019-03-25 2020-03-14 Image processing method and head mounted display device

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201910228846.3A CN110139028B (zh) 2019-03-25 2019-03-25 一种图像处理的方法及头戴式显示设备
CN201910228846.3 2019-03-25

Publications (1)

Publication Number Publication Date
WO2020192458A1 true WO2020192458A1 (zh) 2020-10-01

Family

ID=67568837

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/079387 WO2020192458A1 (zh) 2019-03-25 2020-03-14 一种图像处理的方法及头戴式显示设备

Country Status (7)

Country Link
US (1) US20220197033A1 (zh)
EP (1) EP3920524A4 (zh)
JP (1) JP7408678B2 (zh)
KR (1) KR20210130773A (zh)
CN (1) CN110139028B (zh)
AU (1) AU2020250124B2 (zh)
WO (1) WO2020192458A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113489989A (zh) * 2021-06-30 2021-10-08 宁波星巡智能科技有限公司 电池相机唤醒时的视频数据传输方法、装置、设备及介质

Families Citing this family (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113196139B (zh) 2018-12-20 2023-08-11 美国斯耐普公司 用于生成立体图像的具有双摄像头的柔性眼戴设备
CN110139028B (zh) * 2019-03-25 2020-07-07 华为技术有限公司 一种图像处理的方法及头戴式显示设备
JPWO2020246292A1 (zh) * 2019-06-07 2020-12-10
CN111091506A (zh) * 2019-12-02 2020-05-01 RealMe重庆移动通信有限公司 图像处理方法及装置、存储介质、电子设备
US10965931B1 (en) * 2019-12-06 2021-03-30 Snap Inc. Sensor misalignment compensation
CN112991242A (zh) * 2019-12-13 2021-06-18 RealMe重庆移动通信有限公司 图像处理方法、图像处理装置、存储介质与终端设备
CN111885310A (zh) * 2020-08-31 2020-11-03 深圳市圆周率软件科技有限责任公司 一种全景数据处理方法、处理设备和播放设备
US11729373B1 (en) * 2020-09-15 2023-08-15 Apple Inc. Calibration for head-mountable devices
CN112911268B (zh) * 2021-01-05 2022-11-18 展讯通信（上海）有限公司 一种图像的显示方法及电子设备
CN114268786A (zh) * 2021-12-22 2022-04-01 小派科技（上海）有限责任公司 用于虚拟现实装置的光学***和虚拟现实装置
CN114567764A (zh) * 2022-03-11 2022-05-31 联想(北京)有限公司 穿戴设备及信息处理方法
CN116994294B (zh) * 2023-09-26 2023-12-01 山东理工大学 基于神经网络的虚拟现实设备使用者识别***

Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102568026A (zh) * 2011-12-12 2012-07-11 浙江大学 一种多视点自由立体显示的三维增强现实方法
CN104364754A (zh) * 2012-06-12 2015-02-18 索尼公司 信息处理设备、信息处理方法及程序
WO2017173735A1 (zh) * 2016-04-07 2017-10-12 深圳市易瞳科技有限公司 一种基于视频透视的智能眼镜***及其透视方法
CN107462994A (zh) * 2017-09-04 2017-12-12 浙江大学 沉浸式虚拟现实头戴显示装置和沉浸式虚拟现实显示方法
CN108351691A (zh) * 2015-10-26 2018-07-31 微软技术许可有限责任公司 用于虚拟图像的远程渲染
CN108513057A (zh) * 2017-02-28 2018-09-07 深圳市掌网科技股份有限公司 图像处理方法及装置
WO2018175625A1 (en) * 2017-03-22 2018-09-27 Magic Leap, Inc. Depth based foveated rendering for display systems
CN109040738A (zh) * 2014-07-10 2018-12-18 精工爱普生株式会社 利用直接几何建模的头戴式显示器校准
CN110139028A (zh) * 2019-03-25 2019-08-16 华为技术有限公司 一种图像处理的方法及头戴式显示设备

Family Cites Families (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2000310747A (ja) 1999-02-26 2000-11-07 Mr System Kenkyusho:Kk 画像観察装置
US8884984B2 (en) * 2010-10-15 2014-11-11 Microsoft Corporation Fusing virtual content into real content
US9699379B1 (en) * 2012-09-17 2017-07-04 Amazon Technologies, Inc. Camera arrangements for wide-angle imaging
JP2016046699A (ja) 2014-08-25 2016-04-04 株式会社 ゼネテック 画像音声入出力システム
US10122980B2 (en) * 2015-03-02 2018-11-06 Canon Kabushiki Kaisha Image processing apparatus, image processing method, and non-transitory computer-readable storage medium
US10277813B1 (en) * 2015-06-25 2019-04-30 Amazon Technologies, Inc. Remote immersive user experience from panoramic video
US10127725B2 (en) * 2015-09-02 2018-11-13 Microsoft Technology Licensing, Llc Augmented-reality imaging
KR102658303B1 (ko) * 2016-02-18 2024-04-18 애플 인크. 인사이드-아웃 위치, 사용자 신체 및 환경 추적을 갖는 가상 및 혼합 현실을 위한 머리 장착 디스플레이
JP2017204674A (ja) * 2016-05-09 2017-11-16 株式会社ソニー・インタラクティブエンタテインメント 撮像装置、ヘッドマウントディスプレイ、情報処理システム、および情報処理方法
US10330935B2 (en) * 2016-09-22 2019-06-25 Apple Inc. Predictive, foveated virtual reality system
JP6541701B2 (ja) 2017-03-03 2019-07-10 Kddi株式会社 端末装置およびその制御方法、並びにプログラム
JP7098601B2 (ja) 2017-03-31 2022-07-11 ソニーセミコンダクタソリューションズ株式会社 画像処理装置、撮像装置、画像処理方法、およびプログラム
US10401954B2 (en) * 2017-04-17 2019-09-03 Intel Corporation Sensory enhanced augmented reality and virtual reality device
WO2020069420A2 (en) * 2018-09-28 2020-04-02 Ocelot Laboratories Llc Camera system
US11233954B1 (en) * 2019-01-24 2022-01-25 Rockwell Collins, Inc. Stereo infrared imaging for head mounted devices

Patent Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102568026A (zh) * 2011-12-12 2012-07-11 浙江大学 一种多视点自由立体显示的三维增强现实方法
CN104364754A (zh) * 2012-06-12 2015-02-18 索尼公司 信息处理设备、信息处理方法及程序
CN109040738A (zh) * 2014-07-10 2018-12-18 精工爱普生株式会社 利用直接几何建模的头戴式显示器校准
CN108351691A (zh) * 2015-10-26 2018-07-31 微软技术许可有限责任公司 用于虚拟图像的远程渲染
WO2017173735A1 (zh) * 2016-04-07 2017-10-12 深圳市易瞳科技有限公司 一种基于视频透视的智能眼镜***及其透视方法
CN108513057A (zh) * 2017-02-28 2018-09-07 深圳市掌网科技股份有限公司 图像处理方法及装置
WO2018175625A1 (en) * 2017-03-22 2018-09-27 Magic Leap, Inc. Depth based foveated rendering for display systems
CN107462994A (zh) * 2017-09-04 2017-12-12 浙江大学 沉浸式虚拟现实头戴显示装置和沉浸式虚拟现实显示方法
CN110139028A (zh) * 2019-03-25 2019-08-16 华为技术有限公司 一种图像处理的方法及头戴式显示设备

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP3920524A4

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113489989A (zh) * 2021-06-30 2021-10-08 宁波星巡智能科技有限公司 电池相机唤醒时的视频数据传输方法、装置、设备及介质
CN113489989B (zh) * 2021-06-30 2023-08-11 宁波星巡智能科技有限公司 电池相机唤醒时的视频数据传输方法、装置、设备及介质

Also Published As

Publication number Publication date
EP3920524A4 (en) 2022-03-16
KR20210130773A (ko) 2021-11-01
EP3920524A1 (en) 2021-12-08
US20220197033A1 (en) 2022-06-23
JP7408678B2 (ja) 2024-01-05
CN110139028A (zh) 2019-08-16
JP2022527708A (ja) 2022-06-03
AU2020250124A1 (en) 2021-09-16
CN110139028B (zh) 2020-07-07
AU2020250124B2 (en) 2023-02-02

Similar Documents

Publication Publication Date Title
WO2020192458A1 (zh) 一种图像处理的方法及头戴式显示设备
CN110502954B (zh) 视频分析的方法和装置
CN110488977B (zh) 虚拟现实显示方法、装置、***及存储介质
CN113422903B (zh) 拍摄模式切换方法、设备、存储介质
WO2020238741A1 (zh) 图像处理方法、相关设备及计算机存储介质
CN111476911A (zh) 虚拟影像实现方法、装置、存储介质与终端设备
US20180276800A1 (en) Apparatus and methods for source dynamic range processing of panoramic content
US11750926B2 (en) Video image stabilization processing method and electronic device
CN111741303B (zh) 深度视频处理方法、装置、存储介质与电子设备
WO2020249076A1 (zh) 一种人脸校正方法及电子设备
EP4044000A1 (en) Display method, electronic device, and system
CN113741681A (zh) 一种图像校正方法与电子设备
CN110807769B (zh) 图像显示控制方法及装置
CN113573120B (zh) 音频的处理方法及电子设备、芯片***及存储介质
CN115150542B (zh) 一种视频防抖方法及相关设备
CN113487500A (zh) 图像畸变校正方法与装置、电子设备和存储介质
CN113850709A (zh) 图像变换方法和装置
CN113781548A (zh) 多设备的位姿测量方法、电子设备及***
CN112967261B (zh) 图像融合方法、装置、设备及存储介质
WO2022033344A1 (zh) 视频防抖方法、终端设备和计算机可读存储介质
CN112565735B (zh) 一种虚拟现实的测量和显示方法、装置、以及***
RU2782312C1 (ru) Способ обработки изображения и устройство отображения, устанавливаемое на голове
CN110443841B (zh) 地面深度的测量方法、装置及***
CN111982293A (zh) 体温测量方法、装置、电子设备及存储介质
WO2022127612A1 (zh) 图像校准方法和设备

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20778172

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2020778172

Country of ref document: EP

Effective date: 20210903

ENP Entry into the national phase

Ref document number: 2020250124

Country of ref document: AU

Date of ref document: 20200314

Kind code of ref document: A

ENP Entry into the national phase

Ref document number: 20217030380

Country of ref document: KR

Kind code of ref document: A

ENP Entry into the national phase

Ref document number: 2021557122

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE