WO2023071834A1 - 用于显示设备的对齐方法及对齐装置、车载显示*** - Google Patents

用于显示设备的对齐方法及对齐装置、车载显示*** Download PDF

Info

Publication number
WO2023071834A1
WO2023071834A1 PCT/CN2022/125475 CN2022125475W WO2023071834A1 WO 2023071834 A1 WO2023071834 A1 WO 2023071834A1 CN 2022125475 W CN2022125475 W CN 2022125475W WO 2023071834 A1 WO2023071834 A1 WO 2023071834A1
Authority
WO
WIPO (PCT)
Prior art keywords
calibration
virtual
image
display device
human eye
Prior art date
Application number
PCT/CN2022/125475
Other languages
English (en)
French (fr)
Inventor
郁理
徐雪炀
王进
Original Assignee
虹软科技股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 虹软科技股份有限公司 filed Critical 虹软科技股份有限公司
Publication of WO2023071834A1 publication Critical patent/WO2023071834A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
    • G06T7/85—Stereo camera calibration
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G06T19/006—Mixed reality
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras

Definitions

  • the present invention relates to the technical field of image processing, in particular to an alignment method and alignment device for a display device, and a vehicle-mounted display system.
  • HUD head-up display
  • HUD augmented reality
  • immersive AR-HUD requires that the virtual image in the user's eyes be related to the real scene covered by the virtual image in the user's eyes. Such as highlighting the boundary of the lane line, prompting the planning route, prompting the safe driving distance, etc., which greatly improves the user experience.
  • immersive AR-HUD has correspondingly increased requirements for virtual-real alignment and calibration.
  • the main defects of virtual-real alignment and calibration technology of immersive AR-HUD include: (1) In vehicle-mounted applications, the driver’s head movement cannot be constrained, so the virtual image projected by AR-HUD is different from the user’s perspective.
  • common virtual-real alignment technologies include two types: existing technology 1, through 3D calibration and re-projection technology, it is possible to re-project the 3D exterior scene of the car to the human eye and realize it by truncating on the AR-HUD virtual image surface, but due to This technology has very high requirements for sensors and computing power, so it cannot be commercialized on a large scale.
  • the present disclosure provides an alignment method and alignment device for a display device, and a vehicle-mounted display system, so as to at least solve the "immersive" augmented reality that cannot enable drivers and passengers to obtain a good virtual-real alignment while moving their eyesight in the related art
  • the experience reduces the technical problems of the driver and passengers' interest in using it.
  • a method for aligning a display device including: determining a sight focus area of a user in a target vehicle, wherein at least one physical reference object is included in the sight focus area; acquiring the second A camera collects the captured image of the original scene, and extracts the entity identification points in the captured image; through the pre-obtained calibration parameters, determines the two-dimensional image between the captured image and the virtual image in the user's current field of view A mapping relationship, wherein the virtual image is an image to be projected by the display device; according to the two-dimensional image mapping relationship, the entity identification point is projected into the virtual image, wherein the virtual identification in the virtual image The object is aligned with the physical reference object in the focus area of the line of sight.
  • the step of determining the focus area of the user's line of sight in the target vehicle includes: determining the user's focus line of gaze on the front of the target vehicle based on the coordinates of the user's line of sight projection subject; The focused gaze line is merged with the plane of the ground in front of the target vehicle to obtain the user's gaze point; based on the gaze point, the user's line of sight focus area in the target vehicle is determined.
  • the calibration parameters are determined through a virtual-real alignment calibration method in combination with a human eye simulation auxiliary camera.
  • the alignment method further includes: when the positions of the display device and the user's eyes are kept at a fixed position, combining the human eye simulation auxiliary camera at the fixed position to determine the or, when the position of the display device and/or the user's eyes changes, repeat the virtual-real switch calibration method by adjusting the position-changed device to a plurality of different calibration positions and determine the normalization parameter calibration The corresponding relationship, the calibration parameter is determined according to the actual location of the device and the normalized parameter calibration corresponding relationship.
  • the actual positions of the display device and the user's eyes are acquired through a driver monitoring system.
  • the calibration parameters include: a first two-dimensional image mapping relationship between the first image captured by the first camera and the second image captured by the human eye simulation auxiliary camera, and the human eye simulation auxiliary camera captures The second two-dimensional image mapping relationship between the third image and the virtual calibration pattern.
  • determining the calibration parameters through a virtual-real alignment calibration method combined with human eye simulation assistance camera includes: controlling the opening and closing of the display device, combining physical calibration patterns and virtual calibration patterns, according to the human eye simulation assistance Images captured by the camera and the first camera determine the calibration parameters.
  • the calibration parameters are determined according to the images captured by the human eye simulation auxiliary camera and the first camera, including: After the above display device, combined with the physical calibration pattern, determine the third two-dimensional image mapping relationship between the first image collected by the first camera and the second image collected by the human eye simulation auxiliary camera; After the display device is described, in combination with the virtual calibration pattern, the fourth two-dimensional image mapping relationship between the third image collected by the human eye simulation auxiliary camera and the virtual calibration pattern is determined; the third two-dimensional image is The mapping relationship and the fourth two-dimensional image mapping relationship are characterized as the calibration parameters.
  • the physical calibration pattern is placed in front of the target vehicle, and the virtual calibration pattern is displayed on a virtual image plane projected by the display device, wherein the physical calibration pattern includes at least one physical feature point, so The virtual calibration pattern includes at least one virtual feature point.
  • the alignment method further includes: replacing the physical calibration pattern with an equivalent physical calibration pattern that is higher than and parallel to the physical calibration pattern in the field of view of the human eye simulation auxiliary camera.
  • the display device by controlling the display device to be turned on and off, combined with the equivalent physical calibration pattern and the virtual calibration pattern, according to the images captured by the human eye simulation auxiliary camera and the first camera Calibration parameters, including: after turning off the display device, restore the equivalent virtual calibration pattern on the ground through the equivalent physical calibration pattern and project it to the fourth image generated by the first camera, and the human eye simulation auxiliary camera Collecting a fifth image of the equivalent physical calibration pattern, determining a fifth two-dimensional image mapping relationship between the fourth image and the fifth image; after turning on the display device, combining the virtual calibration pattern, determining the sixth two-dimensional image mapping relationship between the sixth image captured by the human eye simulation auxiliary camera and the virtual calibration pattern; combining the fifth two-dimensional image mapping relationship with the sixth two-dimensional image
  • the mapping relationship is characterized as the calibration parameter.
  • the equivalent physical calibration pattern is used to restore the equivalent virtual calibration pattern on the ground and project it to the fourth image generated by the first camera, including: the first camera obtained according to the calibration, the human eye simulation Auxiliary camera and the first space pose of the equivalent physical calibration pattern, determine the height above the ground and the preset zoom ratio of the equivalent physical calibration pattern; according to the height above the ground and the preset zoom ratio, combined
  • the principle of light propagation restores the equivalent physical calibration pattern to the equivalent virtual calibration pattern located on the ground and calculates the second space pose of the virtual calibration pattern; combines the second space pose to convert the virtual calibration pattern
  • the fourth image is generated by projecting onto the imaging plane of the first camera.
  • the normalized parameter calibration correspondence is a mapping relationship between any normalized parameter and a one-to-one corresponding calibration parameter, wherein the normalized parameter is the display device and/or the person
  • the amount of positional movement of the eye simulation auxiliary camera accounts for the ratio of the variable stroke.
  • the device whose position changes includes: the display device and/or the human eye simulation auxiliary camera.
  • the virtual-real opening and closing calibration method by adjusting the position-changed device to a plurality of different calibration positions and determine the normalized parameter calibration correspondence , determining the calibration parameter according to the actual position of the device and the normalized parameter calibration correspondence, including: adjusting the device whose position has changed to at least 2 different calibration positions in the variable space and repeating the virtual and actual Open and close the calibration method and determine the normalized parameter calibration correspondence; determine the normalized position parameter on the variable space based on the actual position of the device, according to the normalized position parameter and the normalized
  • the calibration parameter is determined by a normalization parameter calibration correspondence.
  • the alignment method includes: adjusting the device to at least two different calibration positions within the variable linear stroke, repeating the virtual-real opening and closing calibration method and determining The normalized parameter calibration correspondence relationship, wherein the at least two different calibration positions include: the highest display reference position and the lowest display reference position of the device in the variable linear stroke.
  • the alignment method includes: adjusting the device to at least 3 different calibration positions within the variable plane stroke, repeating the virtual-real opening and closing calibration method and determining The normalized parameter calibration correspondence relationship, wherein the at least three different calibration positions include: three limit reference positions of the device that are not collinear at the same time.
  • the alignment method includes: adjusting the device to at least 4 different calibration positions within the variable three-dimensional space stroke and repeating the virtual-real opening and closing calibration method And determine the normalized parameter calibration correspondence, wherein the at least four different calibration positions include: four limit reference positions of the device that are not coplanar at the same time.
  • adjusting the device whose position changes to at least two different calibration positions in the variable space repeats the virtual-real opening and closing calibration method and determines the normalized parameter calibration correspondence, including: adjusting the position to change to at least 2 different calibration positions in the variable space, and repeat the virtual-real open-close calibration method at the different calibration positions to obtain a calibration parameter set, wherein each calibration parameter set is included in the calibration parameter set One-to-one correspondence with each calibration position; determine the normalization parameter set according to all the calibration positions and the variable space; combine the calibration parameter set and the normalization parameter set to fit and determine the normalization parameter Calibrate the corresponding relationship.
  • the alignment method further includes: adjusting the position of the human eye simulation auxiliary camera by mounting the human eye simulation auxiliary camera on a mechanical arm, and providing corresponding three-dimensional normalized coordinates.
  • the alignment method further includes: combining the two-dimensional image mapping relationship and the pose of the target vehicle, projecting the entity identification point into the virtual image.
  • an alignment device for a display device, including: a first determination unit configured to determine the sight focus area of the user in the target vehicle, wherein the line of sight focus area At least one entity reference object is included; the first acquisition unit is configured to acquire the photographed image captured by the first camera on the original scene, and extracts the entity identification points in the photographed image; the first determination unit is configured to use the pre-obtained Calibration parameters, determining the two-dimensional image mapping relationship between the captured image and the virtual image in the user's current field of view, wherein the virtual image is an image to be projected by the display device; the alignment unit is configured to The image mapping relationship is to project the entity identification point into the virtual image, wherein the virtual identification object in the virtual image is aligned with the entity reference object in the sight focus area.
  • the first determination unit includes: a first determination module, configured to determine the user's focused line of gaze on the front of the target vehicle based on the coordinates of the user's line of sight projection subject; a gaze point determination module , set to intersect the focused gaze line with the plane where the ground in front of the target vehicle is located to obtain the gaze position of the user; the second determination module is configured to determine the gaze position based on the gaze position The focus area of the user's line of sight in the target vehicle.
  • the calibration parameters are determined through a virtual-real alignment calibration method in combination with a human eye simulation auxiliary camera.
  • the alignment device further includes: a third determining unit, configured to, when the positions of the display device and the user's eyes remain at a fixed position, combine the human eye simulation auxiliary camera at the fixed position to pass
  • the virtual-real opening and closing calibration method determines the calibration parameters; or, the fourth determination unit is configured to repeat by adjusting the device whose position changes to a plurality of different calibration positions when the position of the display device and/or the user's eyes changes
  • the virtual-real opening and closing calibration method also determines the normalized parameter calibration correspondence, and determines the calibration parameter according to the actual location of the device and the normalized parameter calibration correspondence.
  • the actual positions of the display device and the user's eyes are acquired through a driver monitoring system.
  • the calibration parameters include: a first two-dimensional image mapping relationship between the first image captured by the first camera and the second image captured by the human eye simulation auxiliary camera, and the human eye simulation auxiliary camera captures The second two-dimensional image mapping relationship between the third image and the virtual calibration pattern.
  • the third determination unit includes: a third determination module, configured to combine the physical calibration pattern and the virtual calibration pattern by controlling the opening and closing of the display device, according to the human eye simulation auxiliary camera and the first Images captured by the camera determine the calibration parameters.
  • a third determination module configured to combine the physical calibration pattern and the virtual calibration pattern by controlling the opening and closing of the display device, according to the human eye simulation auxiliary camera and the first Images captured by the camera determine the calibration parameters.
  • the third determination module includes: a first determination sub-module, configured to determine the difference between the first image captured by the first camera and the simulated human eye in combination with the physical calibration pattern after the display device is turned off.
  • a third two-dimensional image mapping relationship between the second images captured by the auxiliary camera; the second determining submodule is configured to determine the human eye simulation auxiliary camera acquisition in combination with the virtual calibration pattern after the display device is turned on.
  • a fourth two-dimensional image mapping relationship between the third image of the virtual calibration pattern and the third determination submodule, configured to characterize the third two-dimensional image mapping relationship and the fourth two-dimensional image mapping relationship is the calibration parameter.
  • the alignment device further includes: a replacement unit, configured such that the physical calibration pattern is replaced by an equivalent physical calibration pattern that is higher than and parallel to the physical calibration pattern in the field of view of the human eye simulation auxiliary camera.
  • a replacement unit configured such that the physical calibration pattern is replaced by an equivalent physical calibration pattern that is higher than and parallel to the physical calibration pattern in the field of view of the human eye simulation auxiliary camera.
  • the third determination module further includes: a fourth determination submodule, configured to restore the equivalent virtual calibration pattern on the ground through the equivalent physical calibration pattern and project it to the first camera after the display device is turned off
  • the generated fourth image, and the fifth image collected by the human eye simulation auxiliary camera about the equivalent physical calibration pattern determine the fifth two-dimensional image mapping relationship between the fourth image and the fifth image
  • the fifth determining submodule is configured to determine the sixth two-dimensional distance between the sixth image captured by the human eye simulation auxiliary camera and the virtual calibration pattern in combination with the virtual calibration pattern after the display device is turned on.
  • Image mapping relationship a sixth determining submodule, configured to represent the fifth two-dimensional image mapping relationship and the sixth two-dimensional image mapping relationship as the calibration parameters.
  • a vehicle-mounted display system including: a driver monitoring system configured to track the focus area of sight of the user in the target vehicle, wherein the focus area of sight includes at least one entity A reference object; a driving recorder, which is set to shoot the focus area of the line of sight to obtain a photographed image; a vehicle-mounted controller, which is connected with the driver monitoring system and the driving recorder respectively, and performs any of the above described
  • An alignment method for a display device a head-up display, configured to project a virtual image to a preset position directly in front of the user, wherein the virtual marker in the virtual image is referenced to the entity in the focus area of the line of sight object alignment.
  • a vehicle-mounted control device including: a processor; and a memory configured to store executable instructions of the processor; wherein the processor is configured to Executing instructions to execute the alignment method for a display device described in any one of the above.
  • a computer-readable storage medium including a stored computer program, wherein when the computer program is running, the computer-readable storage medium is controlled
  • the device executes the alignment method for a display device described in any one of the above.
  • the focus area of the user's line of sight in the target vehicle is determined first, wherein at least one physical reference object is included in the focus area of the line of sight, the captured image of the original scene is acquired by the first camera, and the captured image is extracted.
  • the physical identification point of through the pre-obtained calibration parameters, determine the two-dimensional image mapping relationship between the captured image and the virtual image in the user's current field of view, where the virtual image is the image to be projected by the display device, according to the two-dimensional image mapping relationship, and project the entity identification point into the virtual image, wherein the virtual identification object in the virtual image is aligned with the entity reference object in the sight focus area.
  • the real scene of the focus area outside the vehicle is captured by the imaging device, and the entity identification points of the captured screen content are projected onto the virtual image, so that the user's field of vision at the current moment
  • the virtual markers in the virtual image in the virtual image are aligned with the physical reference objects in the focus area of sight, and the virtual image in the user's eyes is associated with the real scene covered by the virtual image in the user's eyes, so as to achieve the "immersive" augmented reality experience effect of virtual and real alignment, and improve It improves the user's experience and interest in use, and then solves the "immersive" augmented reality experience that cannot enable drivers and passengers to get a good virtual-real alignment while moving their eyes, and reduces the interest of drivers and passengers. question.
  • the two-dimensional image pixel correspondence between the image collected by the driving recorder in the user's field of view and the virtual image displayed by the display device can be established without three-dimensional reconstruction.
  • the display device can be realized, which is not only easy to operate, but also improves the calibration speed and greatly reduces the requirements for hardware computing power.
  • the human eye simulation auxiliary camera is installed at the position of the human eye, which can simulate the human eye, realize the industrialization of the calibration process, and greatly enrich the usage scenarios.
  • FIG. 1 is a block diagram of a hardware structure of a computer terminal (or mobile device) for implementing an alignment method for a display device according to an embodiment of the present invention
  • FIG. 2 is a flow chart of an optional alignment method for a display device according to an embodiment of the present invention
  • FIG. 3 is a first schematic diagram of a scene of an optional calibration method for a display device according to an embodiment of the present invention
  • Fig. 4 is a schematic diagram of an optional basic principle of small hole imaging in three-dimensional vision according to an embodiment of the present invention.
  • FIG. 5 is a second schematic diagram of a scene of an optional calibration method for a display device according to an embodiment of the present invention.
  • FIG. 6 is a schematic diagram of a third scene of an optional calibration method for a display device according to an embodiment of the present invention.
  • Fig. 7 is an optional alignment effect diagram according to an embodiment of the present invention.
  • Fig. 8 is a schematic diagram of an alignment device for a display device according to an embodiment of the present invention.
  • Augmented reality head-up display (AR-HUD): The display is located in front of the dashboard in front of the driver's eyes, or the display is covered on the windshield, and the projection device displays the virtual image content on the augmented reality head-up display
  • the virtual image surface of the display, and the virtual image content is related to the real scene covered by the virtual image, such as prompts, marks, transparency, etc.
  • Driver monitor system (driver monitor system, DMS): Detect driver behavior through visual methods, such as eye closing, blinking, gaze direction, head movement, etc.
  • AR Augmented reality
  • Virtual-real registration In augmented reality, the generated virtual image is aligned with the real scene.
  • the present invention can calculate the position of the AR-HUD virtual image in real time, so that the virtual image can always be aligned with the real scene in the driver's line of sight.
  • the following embodiments of the present invention can be applied to various vehicles equipped with augmented reality head-up displays.
  • the types of vehicles include but are not limited to: bicycles, cars, driving simulators, AR windows, etc., and can also be applied to various AR projection devices where the geometric pose of the user is not fixed, such as naked-eye AR.
  • a virtual reality alignment technology and its calibration algorithm for AR-HUD following the line of sight movement are proposed.
  • the driver monitoring system can calculate the current moment of the driver.
  • the on-board computer calculates the picture of the real scene outside the car seen by the driver in the driving recorder at the current moment through the gaze point, and further projects the picture in the driving recorder to the AR-HUD virtual image according to the alignment method proposed by the embodiment of the present invention
  • the AR-HUD virtual image in the driver's field of vision at the current moment can completely cover the real scene outside the car, achieving an "immersive" augmented reality experience of virtual-real alignment.
  • To balance computing power, cost, and user experience requirements there is no need for a 3D reconstruction process or complex camera internal and external parameter calibration, and the real-time performance on mainstream vehicles can reach the camera's frame rate.
  • an embodiment of an alignment method for a display device is also provided. It should be noted that the steps shown in the flow charts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions , and, although a logical order is shown in the flowcharts, in some cases the steps shown or described may be performed in an order different from that shown or described herein.
  • FIG. 1 shows a block diagram of a hardware structure of a computer terminal (or mobile device) for implementing an alignment method for a display device.
  • the computer terminal 10 may include one or more (shown by 102a, 102b, ..., 102n in the figure) processors 102 (the processors 102 may include but not limited to microprocessor A processing device such as a processor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission module 106 for communication functions.
  • a processing device such as a processor MCU or a programmable logic device FPGA
  • memory 104 for storing data
  • a transmission module 106 for communication functions.
  • FIG. 1 is only a schematic diagram, and it does not limit the structure of the above-mentioned electronic device.
  • computer terminal 10 may also include more or fewer components than shown in FIG. 1 , or have a different configuration than that shown in FIG. 1 .
  • the one or more processors 102 and/or other data processing circuits described above may generally be referred to herein as "data processing circuits".
  • the data processing circuit may be implemented in whole or in part as software, hardware, firmware or other arbitrary combinations.
  • the data processing circuit can be a single independent processing module, or be fully or partially integrated into any of the other elements in the computer terminal 10 (or mobile device).
  • the data processing circuit is used as a processor control (for example, the selection of the terminal path of the variable resistor connected to the interface).
  • the memory 104 can be used to store software programs and modules of application software, such as the program instruction/data storage device corresponding to the alignment method for the display device in the embodiment of the present invention, the processor 102 runs the software program stored in the memory 104 and module, so as to execute various functional applications and data processing, that is, to realize the above-mentioned vulnerability detection method of the application program.
  • the memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory.
  • the memory 104 may further include a memory that is remotely located relative to the processor 102 , and these remote memories may be connected to the computer terminal 10 through a network. Examples of the aforementioned networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.
  • the transmission device 106 is used to receive or transmit data via a network.
  • the specific example of the above-mentioned network may include a wireless network provided by the communication provider of the computer terminal 10 .
  • the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet.
  • the transmission device 106 may be a radio frequency (Radio Frequency, RF) module, which is used to communicate with the Internet in a wireless manner.
  • RF Radio Frequency
  • the display may be, for example, a touchscreen liquid crystal display (LCD), which may enable a user to interact with the user interface of the computer terminal 10 (or mobile device).
  • LCD liquid crystal display
  • an embodiment of an alignment method for a display device is provided. It should be noted that the steps shown in the flow chart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, Also, although a logical order is shown in the flowcharts, in some cases the steps shown or described may be performed in an order different from that shown or described herein.
  • Fig. 2 is a flow chart of an optional alignment method for a display device according to an embodiment of the present invention. As shown in Fig. 2, the method includes the following steps:
  • Step S202 determining the sight focus area of the user in the target vehicle, wherein the sight focus area includes at least one physical reference object;
  • Step S204 acquiring the captured image of the original scene captured by the first camera, and extracting entity identification points in the captured image
  • Step S206 using pre-obtained calibration parameters, determine the two-dimensional image mapping relationship between the captured image and the virtual image in the user's current field of view, where the virtual image is an image to be projected by the display device;
  • Step S208 according to the two-dimensional image mapping relationship, project the entity identification point into the virtual image, wherein the virtual identification object in the virtual image is aligned with the entity reference object in the sight focus area.
  • the focus area of the user's line of sight in the target vehicle can be determined first, wherein at least one physical reference object is included in the focus area of the line of sight, the captured image of the original scene captured by the first camera is obtained, and the captured image is extracted.
  • Entity identification points through pre-obtained calibration parameters, determine the two-dimensional image mapping relationship between the captured image and the virtual image in the user's current field of view, where the virtual image is the image to be projected by the display device, according to the two-dimensional image mapping relationship , project the entity marker point into the virtual image, wherein the virtual marker in the virtual image is aligned with the entity reference object in the focus area of the line of sight.
  • the real scene of the sight focus area outside the vehicle is captured by the photographing device, and the physical identification points of the captured screen content are projected onto the virtual image, so that the user at the current moment
  • the virtual markers in the virtual image in the field of view are aligned with the physical reference objects in the focus area of sight, and the virtual image in the user's eyes is associated with the real scene covered by the virtual image in the user's eyes, so as to achieve the "immersive" augmented reality experience effect of virtual and real alignment.
  • a car vehicle is taken as an example for schematic illustration, wherein the AR-HUD is installed in front of the instrument panel directly in front of the driver behind the steering wheel, and the virtual image screen is projected onto the virtual image surface where the AR-HUD is located.
  • the recorder (recorded as Cam adas ) is installed at the rearview mirror of the car, facing the forward direction of the car.
  • the alignment method of the display device can be applied to a vehicle-mounted augmented reality system, and the display device is an augmented reality head-up display AR-HUD.
  • Step S202 determining the focus area of sight of the user in the target vehicle, wherein at least one physical reference object is included in the focus area of sight.
  • the target vehicle includes but is not limited to: bicycles, cars, driving simulators, AR windows, etc.
  • the sight focus area is the main viewing area of the real scene outside the car that can be observed in the user's field of vision
  • the physical reference object Including but not limited to: lane lines, road signs, pedestrians, vehicles in front and other reference objects that can assist users in driving decision-making in any scene.
  • the step of determining the focus area of the user's line of sight in the target vehicle includes: determining the user's focus line of gaze on the front of the target vehicle based on the coordinates of the user's sight projection subject; The planes where the ground is located are merged to obtain the gaze point of the user; based on the gaze point, the focus area of the user's line of sight in the target vehicle is determined.
  • One option is to use the driver monitoring system (DMS) to track the coordinates of the user's gaze-casting subject, and use the driver-monitoring system to calculate the user's eye coordinates at the current moment (that is, the coordinates of the gaze-casting subject), and then determine the coordinates of the user's eye-casting subject.
  • the focused gaze line from the eyes to the ground or physical reference object in front of the target vehicle is used to obtain the user's gaze point.
  • the embodiment of the present application does not limit the method of obtaining the line of sight and viewpoint from the collected user's line of sight projecting subject.
  • the user's line of sight and viewpoint can be determined based on the corneal pupil model or based on deep learning.
  • the embodiment of the present application does not limit the method based on the gaze position
  • the method of determining the focal area of sight in the user's field of view can be determined by traditional geometric methods or deep learning.
  • Step S204 acquiring a captured image of the original scene captured by the first camera, and extracting entity identification points in the captured image.
  • the entity identification point is a pixel point on the entity reference object.
  • the entity reference object can refer to the entity object on the road, including but not limited to: lane lines, road signs, pedestrians, front
  • a reference object that can assist users in driving decisions in any scene such as a vehicle. Since the actual reference objects contain rich scene information, user decision-making often depends on the actual reference objects in the scene, and actively extracting the scene information contained in them provides the possibility for immersive guidance.
  • Step S206 using pre-obtained calibration parameters, determine the two-dimensional image mapping relationship between the captured image and the virtual image in the user's current field of view, where the virtual image is an image to be projected by the display device.
  • the calibration parameters may be determined by a virtual-real alignment calibration method according to the position and state of the display device and the user's eyes, combined with the human eye simulation auxiliary camera.
  • the human eye simulation auxiliary camera is used to simulate the position of the human eye, which can accurately quantify the coordinates of the user's eyes in space, improve the efficiency of calibration while ensuring the accuracy of calibration, and realize the industrialization process of calibration. If the human eye is directly used for calibration, the user cannot be fixed at the calibration position, and the spatial range of calibration will be limited, which will affect the accuracy of calibration parameters.
  • the height of the display device In the actual use scene, different users have different requirements for the height of the display device, and the height of the display device is not fixed. Similarly, the user's line of sight does not remain unchanged during use, such as changes in sitting posture, head movement, scene bumps, etc. It can cause the user's eye position and line of sight to change.
  • the virtual image of the display device is always aligned with the real scene of the scene by following the movement of the line of sight, regardless of the user's sitting posture change, head movement and display device adjusted and terminated.
  • the alignment method further includes: when the positions of the display device and the user's eyes are kept at a fixed position, at the fixed position, combined with the human eye simulation auxiliary camera, the calibration parameters are determined by a virtual-real open-close calibration method; or, when When the position of the display device and/or the user's eyes changes, repeat the virtual and real opening and closing calibration method by adjusting the device whose position has changed to multiple different calibration positions and determine the normalized parameter calibration correspondence, according to the actual position of the device and the normalized parameters The calibration correspondence determines the calibration parameters.
  • the device whose position changes includes: a display device and/or a human eye simulation auxiliary camera.
  • the actual positions of the display device and the user's eyes are acquired through a driver monitoring system.
  • the calibration parameters include: the first two-dimensional image mapping relationship between the first image captured by the first camera and the second image captured by the human eye simulation auxiliary camera, the third image captured by the human eye simulation auxiliary camera and the virtual A second two-dimensional image mapping relationship between calibration patterns.
  • the present application does not limit the order of determining the calibration parameters, which may be calculated sequentially or in parallel.
  • Fig. 3 is a scene diagram of an optional calibration method for a display device according to an embodiment of the present invention.
  • a physical calibration pattern (1) and its form is not limited (for example , calibration board, calibration cloth, ground printing calibration pattern, etc., the style of the calibration pattern and the content displayed on the pattern are not specifically limited)
  • a virtual calibration pattern (2) and its resolution is the same as that of the AR-HUD virtual image Consistent, can be displayed by AR-HUD
  • an auxiliary calibration camera in this embodiment refers to the human eye simulation auxiliary camera, denoted as Cam eye
  • the style of the calibration pattern includes but is not limited to: checkerboard grid, dot grid , graphic code (one-dimensional code, two-dimensional code), etc.
  • the Cam eye is installed at the position of the human eye to simulate the human eye, and the physical calibration pattern (1) is laid on the ground in front of the car, and the feature points in the physical calibration pattern (1) are displayed on the auxiliary calibration camera Cam eye and the main calibration camera Cam adas (this implementation In this example, it refers to the first camera, which is set to shoot the real scene outside the vehicle, and can be located at the installation position of the driving recorder. Its types include but are not limited to: visible light camera, infrared camera, etc.). The feature points of are distinguishable in Cam eye .
  • the calibration methods in this embodiment include multiple situations, and each calibration method will be described below, and the following calibration methods in this embodiment can be used in combination with each other.
  • combine the human eye simulation auxiliary camera to determine the calibration parameters through the virtual-real alignment calibration method including: by controlling the opening and closing of the display device, combining the physical calibration pattern and the virtual calibration pattern, according to the human eye simulation auxiliary camera and the first camera capture image to determine the calibration parameters.
  • the display device in this embodiment may include but not limited to: AR-HUD device.
  • This embodiment uses the virtual image surface of the AR-HUD device for schematic illustration.
  • the calibration parameters are determined according to the images captured by the human eye simulation auxiliary camera and the first camera, including: after the display device is turned off, combined with the physical calibration pattern Pattern, determine the third two-dimensional image mapping relationship between the first image captured by the first camera and the second image captured by the human eye simulation auxiliary camera; after the display device is turned on, combined with the virtual calibration pattern, determine the human eye simulation auxiliary camera A fourth two-dimensional image mapping relationship between the collected third image and the virtual calibration pattern; characterize the third two-dimensional image mapping relationship and the fourth two-dimensional image mapping relationship as calibration parameters.
  • the physical calibration pattern is placed in front of the target vehicle, and the virtual calibration pattern is displayed on the virtual image surface projected by the display device, wherein the physical calibration pattern includes at least one physical feature point, and the virtual calibration pattern includes at least one virtual feature point.
  • the embodiment of the present application does not limit the extraction of feature points, and the method of determining the transformation relationship of the two-dimensional image based on the feature points.
  • Cam adas collects the scene image, and extracts the pixel point of the reference object in the scene image, such as the pixel point where the lane line is located, denoted as p, through the two-dimensional image transformation relationship H adas2eye and two-dimensional
  • the lit pixels are the virtual lane lines displayed on the display device, that is, under the field of view of the user's current position, the virtual lane lines displayed on the display device are overlaid on the real white lane lines.
  • the two-dimensional image pixel correspondence between the image collected by the driving recorder in the user's field of view and the virtual image displayed by the display device can be established without three-dimensional reconstruction through the virtual calibration pattern and the physical calibration pattern.
  • the display device can be realized, and the operation is simple, while improving the calculation efficiency and calibration speed, and greatly reducing the requirements for hardware computing power.
  • the human eye simulation auxiliary camera is installed at the position of the human eye to simulate the human eye, which helps to realize the industrialization of the calibration process and enriches the usage scenarios.
  • the embodiment of the present invention if the calibration operation uses the real ground as the real scene area covered by the AR-HUD, in actual situations, the covered real ground is about 30-100 meters away from the front of the vehicle.
  • the space requirements and the size requirements of the physical calibration pattern (1) will be extremely high.
  • the embodiment of the present application shortens the space distance required by the physical calibration pattern (1) in the calibration process by using the equivalent physical calibration pattern.
  • the alignment method further includes: replacing the physical calibration pattern with an equivalent physical calibration pattern that is higher than and parallel to the physical calibration pattern in the field of view of the human eye simulation auxiliary camera.
  • Fig. 4 is a schematic diagram of the basic principle of small hole imaging in an optional three-dimensional vision according to an embodiment of the present invention.
  • the images (1) and (2) on the same imaging plane through the small hole projection center are consistent.
  • the ratio of their object heights is consistent with the ratio of object distance, through small hole optics System imaging is consistent. Therefore, according to this principle, according to the actual situation of different calibration sites, the distance between the physical calibration pattern (1) and the calibration vehicle can be shortened during the calibration process of AR-HUD.
  • the calibration parameters are determined according to the images captured by the human eye simulation auxiliary camera and the first camera, including: after the display device is turned off, by The equivalent physical calibration pattern restores the equivalent virtual calibration pattern on the ground and projects it to the fourth image generated by the first camera, and the human eye simulation assists the camera to collect the fifth image about the equivalent physical calibration pattern, and determines the fourth image and the fourth image.
  • the fifth two-dimensional image mapping relationship and the sixth two-dimensional image mapping relationship are represented as calibration parameters.
  • the equivalent physical calibration pattern is used to restore the equivalent virtual calibration pattern on the ground and project it to the fourth image generated by the first camera, including: the first camera obtained according to the calibration, the human eye simulation auxiliary camera and the equivalent physical object
  • the first space pose of the calibration pattern is determined to determine the height above the ground and the preset zoom ratio of the equivalent physical calibration pattern; according to the height above the ground and the preset zoom ratio, combined with the principle of light propagation, the equivalent physical calibration pattern is restored to the ground an equivalent virtual calibration pattern and calculating a second space pose of the virtual calibration pattern; combining the second space pose, projecting the virtual calibration pattern onto the imaging plane of the first camera to generate a fourth image.
  • Fig. 5 is a schematic diagram of a second optional calibration method for display devices according to an embodiment of the present invention. As shown in Fig.
  • an "equivalent physical calibration pattern (3)" is introduced in this embodiment
  • the physical calibration pattern (1) use the equivalent physical calibration pattern (3) higher than and parallel to the physical calibration pattern (1) in the Cam eye field of view instead of the physical calibration pattern (1), the equivalent physical calibration pattern (3)
  • the zoom ratio of the physical calibration pattern (1) is equivalent to the ratio of the height above the ground of the equivalent physical calibration pattern (3) to the height of the Cam eye , and can also be equivalent to the horizontal distance from the physical calibration pattern (1) to the Cam eye
  • the ratio of the horizontal distance from the equivalent physical calibration pattern (3) to the Cam eye the layout and requirements of the rest of the calibration equipment are consistent with those in Figure 3, and will not be repeated here.
  • the calibration working distance can be greatly shortened, and the requirements for the calibration site can be reduced.
  • the size of the equivalent physical calibration pattern (3) can be reduced compared to the physical calibration pattern (1).
  • the equivalent physical calibration pattern (3) only fits the field of view of Cam eye , its projection relationship does not satisfy the field of view of Cam adas , so the poses of Cam eye , Cam adas and physical calibration pattern (3) can be calculated separately relationship, and restore the equivalent virtual calibration pattern (1) according to the height above the ground of the equivalent physical calibration pattern (3), and then re-project to Cam adas , wherein, re-projection refers to the The pixels of the equivalent physical calibration pattern (3) are projected onto the pixels of the Cam adas field of view to generate a new image.
  • the camera internal parameters of the first camera Cam eye and the human eye simulation auxiliary camera Cam adas can be independently calibrated. This application does not limit the camera Calibration method.
  • the spatial pose not only provides the positional relationship between devices, but also provides equivalent physical calibration Pattern (3) height h from the ground and preset zoom ratio information; according to the equivalent physical calibration pattern (3) height h from the ground (this embodiment refers to the height from the ground of the equivalent physical calibration pattern) and the light along the line Propagation principle, calculate the space pose of the virtual calibration pattern (1) located on the ground (this embodiment refers to the second space pose), and finally re-project the feature points of the virtual calibration pattern (1) to Cam adas to obtain the generated A new image (that is, projected onto the imaging surface of the first camera to generate a fourth image).
  • the alignment steps after obtaining two sets of calibration parameters are the same as the steps included in the first calibration method, and will not be repeated here.
  • the virtual ground is calculated and re-projected by introducing an equivalent calibration plane, which greatly shortens the calibration working distance and reduces the requirements for the calibration site.
  • the first calibration method and the second calibration method may be used to acquire calibration parameters at the fixed position.
  • the user's line of sight does not remain unchanged. Changes in sitting posture, head movement, scene bumps, etc. can cause changes in the position and line of sight of the user's eyes.
  • the position of the display device and/or the user's eyes changes, by adjusting the position Change the equipment to multiple different calibration positions, repeat the first calibration method and the second calibration method at multiple different calibration positions to determine the normalized parameter calibration correspondence, according to the actual position of the device and the normalization
  • the calibration parameters are determined according to the calibration parameter calibration correspondence.
  • the normalized parameter calibration correspondence is a mapping relationship between any normalized parameter and a one-to-one corresponding calibration parameter, wherein the normalized parameter is the position movement amount of the display device and/or the human eye simulation auxiliary camera Percentage of variable travel.
  • the normalized parameter calibration corresponding relationship determines the calibration parameters, including: adjusting the equipment whose position changes to at least 2 different calibration positions in the variable space, repeating the virtual and real opening and closing calibration method and determining the normalized parameter calibration corresponding relationship; based on the equipment
  • the actual position of is determined as the normalized position parameter in the variable space, and the calibration parameter is determined according to the calibration correspondence between the normalized position parameter and the normalized parameter.
  • variable spaces include but are not limited to: variable linear strokes, variable plane strokes, and variable three-dimensional space strokes.
  • the changeable position (stroke) is represented by normalized parameters, and through the corresponding relationship of several sets of pre-calibrated "normalized parameters-calibration parameters", the corresponding distance of any normalized parameter can be fitted. Calibration parameters.
  • the embodiment of the present application does not limit the method of obtaining travel information, for example, it may be provided through external sensor detection.
  • the minimum value of the above "several” should be greater than or equal to 2 if the position that can be changed is approximately linear (such as adjusting up and down on the AR-HUD virtual image surface), and these several sets of correspondences are not at the same time; if the position that can be changed is approximately
  • the area array (such as the movement of the human eye in the DMS camera) should be greater than or equal to 3, and these several groups of correspondences are not collinear at the same time; furthermore, if the changeable position is within the approximate three-dimensional range (such as simulating the human eye camera in the The movement in the eye box), should be greater than or equal to 4, and these groups of correspondences are not coplanar at the same time.
  • the alignment method includes: adjusting the device to at least two different calibration positions within the variable linear stroke, repeating the virtual and real opening and closing calibration method and determining the normalized parameter calibration correspondence, Among them, at least two different calibration positions include: the highest display reference position and the lowest display reference position of the device in the variable linear stroke.
  • adjust the position-changed device to at least two different calibration positions in the variable space repeat the virtual-real open-close calibration method and determine the normalized parameter calibration correspondence, including: adjust the position-changed device to the variable space At least 2 different calibration positions in different calibration positions, and repeat the virtual and real opening and closing calibration method on different calibration positions to obtain a calibration parameter set, wherein, the calibration parameter set contains each calibration parameter and each calibration position corresponds one by one; according to all calibration positions and available
  • the normalization parameter set is determined in the variable space; the calibration correspondence relationship of the normalization parameter is determined by combining the calibration parameter set and the normalization parameter set fitting.
  • the moving range is 10mm
  • the moving ratio of moving 1mm is 0.1 (that is, the single independent variable is 0.1))
  • the corresponding calibration parameters when the AR-HUD virtual image surface moves at this position are obtained.
  • the human eye simulation assists the camera to simulate the movement of the human eye and determines the calibration parameters corresponding to any position through the above method.
  • Fig. 6 is a schematic diagram of a third scene of an optional calibration method for a display device according to an embodiment of the present invention.
  • the position of the device can be adjusted in a variable linear stroke. When it is adjusted to the highest position, it is the highest display reference position, and when it is adjusted to the lowest position, it is the lowest display reference position.
  • the AR-HUD virtual image surface is adjusted to the highest, that is, when the normalized stroke value is 1, the AR - After the virtual image surface of the HUD, use Cam adas and Cam eye to shoot the 1# calibration pattern (1), and calculate the two-dimensional image transformation relationship H (1 ) adas2eye , and then open the AR-HUD virtual image surface, use the Cam eye to shoot the 1#HUD virtual image surface (that is, the virtual image surface displayed by the display device, the virtual image surface is in front of the front glass and perpendicular to the ground), according to the 1#HUD
  • the on-board computer extracts the pixels corresponding to the real scene (such as lane lines) in the driving recorder image (Cam adas ) Points, according to the current stroke of the AR-HUD virtual image surface and the corresponding relationship between normalized parameter calibration, calculate the calibration parameters of the AR-HUD virtual image surface at the current stroke, and then map the lane line pixels in the driving recorder to the AR- Pixels on the HUD virtual image, at this time, in the driver's sight, the lane lines displayed by the AR-HUD virtual image will cover the real lane lines in the driver's sight, and can be adjusted up and down according to the habitual sight direction of different drivers
  • the virtual image screen of AR-HUD realizes the alignment of virtual and real.
  • the alignment method includes: adjusting the device to at least 3 different calibration positions within the variable plane stroke, repeating the virtual and real opening and closing calibration method and determining the normalized parameter calibration correspondence, Among them, at least 3 different calibration positions include: 3 limit reference positions of the equipment that are not collinear at the same time.
  • the moving position of the human eye simulation auxiliary camera in the cab is captured by DMS, and the moving position can be approximated in a two-dimensional plane (that is, a preset plane area).
  • the normalized coordinates of at least three vertices of the two-dimensional plane may be set as (0,0), (0,1), (1,1).
  • four vertices are used to schematically illustrate the normalized coordinates of the variable plane.
  • the human eye simulation assists the camera to move to the lower left limit position within the variable plane stroke in the cab, which is the normalized coordinate (0,0) (ie the first display reference position), and the DMS records the position of the human eye in the DMS camera at this time.
  • Coordinates denoted as d (0) (that is, the first coordinate of the first display reference position corresponding to the subject of sight projection), the human eye simulation assists the camera to move to the lower right, upper right, and upper left limit positions of the feasible space in the cab, which is the return Normalized coordinates (0,1), (1,1), (1,0), DMS records the coordinates of the human eyes in the DMS camera at the three positions at this time, which is recorded as d (1) (that is, the subject of sight projection corresponds to The second coordinate of the second display reference position), d (2) (that is, the third coordinate of the line of sight projecting subject corresponding to the third display reference position), d (3) (that is, the line of sight projecting subject corresponding to the fourth display reference position fourth coordinate), according to d (0) , d (1) , d (2) , d (3) and normalized coordinates (0,0), (0,1), (1,1), (1 ,0), the fitting normalization matrix H normal can be obtained, based on the coordinates of the human eye recorded in
  • the limit position of the movement is used as the calibration basis, that is, the first display reference position, the second display reference position, the third display reference position and the fourth display reference position
  • the values of the M two-dimensional normalized positions of the human eye in the cab are taken, combined with the M sets of calibration parameter groups and the corresponding values of the M two-dimensional normalized positions to fit and determine the normalized parameter calibration correspondence relationship.
  • the application does not limit the mathematical method of establishing the correspondence, for example, using interpolation or fitting algorithms. Then, the value of the two-dimensional normalized position of the human eye in the cab captured by DMS is used as a single independent variable to obtain the corresponding calibration parameters when the human eye is at any position in the cab.
  • the on-board computer extracts the corresponding pixels of the real scene (such as lane lines) in the driving recorder image (Cam adas ), according to this
  • the coordinates (two-dimensional coordinates) of the human eye in the DMS camera at that time are combined with the fitting normalization matrix H normal to determine the normalized coordinates of the human eye in the preset plane area.
  • the corresponding relationship between the normalized coordinates and the normalized parameter calibration calculate the calibration parameters of the human eye in the current travel, and then map the lane line pixels in the driving recorder to the pixels on the AR-HUD virtual image, at this time , in the driver's line of sight, the lane lines displayed by the AR-HUD virtual image will cover the real lane lines in the driver's line of sight, and the virtual image screen of the AR-HUD can be adjusted up and down according to the habitual line of sight of different drivers to realize virtual reality align.
  • the alignment method includes: adjusting the device to at least 4 different calibration positions within the variable three-dimensional space stroke, repeating the virtual-real opening and closing calibration method and determining the normalized parameter calibration corresponding relationship, wherein at least 4 different calibration positions include: 4 limit reference positions of the equipment that are not coplanar at the same time.
  • the normalized parameters are calibrated with four limit reference positions that are different in time and coplanar, similar to the variable space as a variable two-dimensional plane, and the human eye simulation assists the camera to move to the variable plane travel in the cab
  • the normalization matrix H normal2 can obtain the normalized coordinates of the human eyes in the preset space area at this time.
  • the limit position of the movement is used as the calibration basis, that is, the first display reference position, the second display reference position, the third display reference position and the fourth display reference position are used as the basis for calibration.
  • the present application does not limit the mathematical method for establishing the correspondence, for example, using interpolation or fitting algorithms. Then, the value of the three-dimensional normalized position of the human eye in the cab captured by DMS is used as a single independent variable, and the corresponding calibration parameters when the human eye is at any position in the cab can be obtained.
  • the on-board computer extracts the corresponding pixels of the real scene (such as lane lines) in the driving recorder image (Cam adas ), according to this
  • the coordinates of the human eye in the DMS camera at that time combined with the fitting normalization matrix H normal2 , determine the normalized coordinates (three-dimensional coordinates) of the human eye in the preset plane area, according to the human eye in the preset plane area at this time
  • the corresponding relationship between the normalized coordinates and the normalized parameter calibration calculate the calibration parameters of the human eye in the current travel, and then map the lane line pixels in the driving recorder to the pixels on the AR-HUD virtual image.
  • the lane lines displayed by the AR-HUD virtual image will cover the real lane lines in the driver's sight, and the virtual image of the AR-HUD can be adjusted up and down according to the habitual sight direction of different drivers to achieve virtual-real alignment .
  • the alignment method further includes: adjusting the position of the human eye simulation auxiliary camera by mounting the human eye simulation auxiliary camera on the mechanical arm, and providing corresponding three-dimensional normalized coordinates.
  • the mechanical arm can be used to mount the auxiliary calibration camera Cam eye , and the mechanical arm can be controlled to simulate the movement of the human eye up, down, left, right, front and back in the feasible area in the three-dimensional space of the cab.
  • the Cam eye that simulates the human eye can be given its three-dimensional normalized coordinates in the feasible area by the mechanical arm or the DMS camera that can detect the depth (that is, the normalized coordinates in the preset plane area, such as , define a certain corner of the feasible area as (0,0,0), then the farthest corner from this corner is (1,1,1)).
  • the robot arm is used to provide three-dimensional normalized coordinates, it is also necessary to calibrate the transformation relationship between the robot arm coordinate system and the DMS camera coordinate system. , there is no need for a transformation relationship between the two.
  • Step S208 according to the two-dimensional image mapping relationship, project the entity identification point into the virtual image, wherein the virtual identification object in the virtual image is aligned with the entity reference object in the sight focus area.
  • FIG. 7 is an optional alignment effect diagram according to an embodiment of the present invention.
  • the frame indicated by 101 in FIG. 7 is the virtual lane line displayed in the AR-HUD virtual image range, and the virtual lane line is covered in the Cam eye field of view on the true white lane line (102).
  • the on-board computer recognizes information such as lane lines, pedestrians, and vehicles in front of the Cam adas image that can assist the driver in driving decisions through semantic recognition algorithms (such as neural networks, etc.), and extracts their corresponding pixels in the Cam adas image, and then The two-dimensional image transformation relationship is mapped to the AR-HUD virtual image.
  • the lane lines displayed on the AR-HUD virtual image will cover the real lane lines in the Cam eye field of view to achieve virtual-real alignment.
  • the input human eye position and the output virtual image can be processed in real time, and the real-time performance depends on the lowest frame rate among the DMS camera or Cam adas camera or the AR-HUD display screen.
  • the alignment method further includes: combining the two-dimensional image mapping relationship and the posture of the target vehicle, projecting the entity identification point into the virtual image.
  • the overall state of the target vehicle has attitude changes. For example, when the vehicle brakes or starts, the front of the vehicle correspondingly has a change of raising or lowering the head. If only the calibration under the normal and stable vehicle The parameters do not consider the attitude of the target vehicle, which will lead to inconsistencies between the scene in the user's line of sight and the scene on the virtual image surface, which will affect the alignment effect.
  • the embodiment of the present application further combines the posture correction alignment of the target vehicle on the basis of the two-dimensional image mapping relationship, which improves the accuracy and robustness of the virtual-real alignment and further expands the scope of application scenarios.
  • the present application does not limit the target vehicle. Acquisition methods with attitude, for example, external sensors can be used to obtain the attitude of the target vehicle in real time.
  • the two-dimensional image pixel correspondence between the image collected by the driving recorder in the user's field of view and the virtual image displayed by the display device can be established without three-dimensional reconstruction through the virtual calibration pattern and the physical calibration pattern.
  • the display device can be realized, and the operation is simple, while improving the calculation efficiency and calibration speed, and greatly reducing the requirements for hardware computing power.
  • the human eye simulation auxiliary camera is installed at the position of the human eye to simulate the human eye, which helps to realize the industrialization of the calibration process and enriches the usage scenarios.
  • An alignment device for a display device provided in this embodiment includes a plurality of implementation units, and each implementation unit corresponds to each implementation step in the first embodiment above.
  • FIG. 8 is a schematic diagram of an alignment device for a display device according to an embodiment of the present invention.
  • the alignment device may include: a first determination unit 81, a first acquisition unit 82, and a first determination unit 83 , alignment unit 84, wherein,
  • the first determination unit 81 is configured to determine the focus area of sight of the user in the target vehicle, wherein at least one physical reference object is included in the focus area of sight;
  • the first acquisition unit 82 is configured to acquire the captured image of the original scene captured by the first camera, and extract the entity identification point in the captured image;
  • the first determination unit 83 is configured to determine the two-dimensional image mapping relationship between the captured image and the virtual image in the user's current field of view through pre-obtained calibration parameters, wherein the virtual image is an image to be projected by the display device;
  • the alignment unit 84 is configured to project the entity identification point into the virtual image according to the two-dimensional image mapping relationship, wherein the virtual identification object in the virtual image is aligned with the entity reference object in the sight focus area.
  • the above-mentioned alignment device for a display device may first determine the sight focus area of the user in the target vehicle through the first determination unit 81, wherein at least one physical reference object is included in the sight focus area, and the second acquisition unit 82 is used to obtain the A camera collects the captured image of the original scene, and extracts the entity identification points in the captured image, and determines the two-dimensional image between the captured image and the virtual image in the user's current field of view through the first determining unit 83 through the pre-obtained calibration parameters.
  • Two-dimensional image mapping relationship wherein, the virtual image is the image to be projected by the display device, and the alignment unit 84 projects the entity identification point into the virtual image according to the two-dimensional image mapping relationship, wherein the virtual marker in the virtual image is focused on the line of sight Entity reference objects in the region are aligned.
  • the real scene of the sight focus area outside the vehicle is photographed by the photographing device, and the physical identification points of the captured picture content are projected onto the virtual image, so that the user at the current moment
  • the virtual markers in the virtual image in the field of view are aligned with the physical reference objects in the focus area of sight, and the virtual image in the user's eyes is associated with the real scene covered by the virtual image in the user's eyes, so as to achieve the "immersive" augmented reality experience effect of virtual and real alignment.
  • the first determination unit includes: a first determination module, configured to determine the user's focused line of gaze on the front of the target vehicle based on the coordinates of the user's line of sight; The planes of the ground in front of the target vehicle are merged to obtain the gaze point of the user; the second determination module is configured to determine the focus area of the user's line of sight in the target vehicle based on the gaze point.
  • the calibration parameters are determined through a virtual-real alignment calibration method in combination with a human eye simulation auxiliary camera.
  • the alignment device further includes: a third determination unit, configured to determine the calibration parameters by combining the human eye simulation auxiliary camera at the fixed position with a virtual-real opening and closing calibration method when the positions of the display device and the user's eyes are kept at a fixed position ; or, the fourth determining unit is configured to repeat the virtual-real opening and closing calibration method by adjusting the device whose position changes to a plurality of different calibration positions when the position of the display device and/or the user's eyes changes and determine the normalization parameter calibration correspondence , and determine the calibration parameters according to the actual position of the device and the corresponding relationship between normalized parameter calibration.
  • the actual positions of the display device and the user's eyes are obtained through a driver monitoring system.
  • the calibration parameters include: the first two-dimensional image mapping relationship between the first image captured by the first camera and the second image captured by the human eye simulation auxiliary camera, the third image captured by the human eye simulation auxiliary camera and the virtual A second two-dimensional image mapping relationship between calibration patterns.
  • the third determining unit includes: a third determining module, configured to control the opening and closing of the display device, combine the physical calibration pattern and the virtual calibration pattern, and determine the calibration according to the images captured by the human eye simulation auxiliary camera and the first camera. parameter.
  • the third determining module includes: a first determining submodule, configured to determine the difference between the first image captured by the first camera and the second image captured by the human eye simulation auxiliary camera in combination with the physical calibration pattern after the display device is turned off.
  • the third two-dimensional image mapping relationship between; the second determining submodule is set to determine the fourth and second between the third image collected by the human eye simulation auxiliary camera and the virtual calibration pattern in combination with the virtual calibration pattern after the display device is turned on.
  • three-dimensional image mapping relationship; the third determining submodule is configured to represent the third two-dimensional image mapping relationship and the fourth two-dimensional image mapping relationship as calibration parameters.
  • the alignment device further includes: a replacement unit, configured such that the physical calibration pattern is replaced by an equivalent physical calibration pattern that is higher than and parallel to the physical calibration pattern in the field of view of the human eye simulation auxiliary camera.
  • a replacement unit configured such that the physical calibration pattern is replaced by an equivalent physical calibration pattern that is higher than and parallel to the physical calibration pattern in the field of view of the human eye simulation auxiliary camera.
  • the third determining module further includes: a fourth determining submodule, configured to restore the equivalent virtual calibration pattern on the ground through the equivalent physical calibration pattern after the display device is turned off, and project it to the fourth image generated by the first camera.
  • the fifth determination submodule is set to display when it is turned on
  • the sixth determination sub-module is set to map the fifth two-dimensional image relationship
  • the mapping relationship with the sixth two-dimensional image is represented as a calibration parameter.
  • the alignment device for the display device is also used for: after the display device is turned off, restore the equivalent virtual calibration pattern located on the ground through the equivalent physical calibration pattern and project it to the fourth image generated by the first camera, and
  • the eye simulation auxiliary camera collects the fifth image about the equivalent physical calibration pattern, and determines the fifth two-dimensional image mapping relationship between the fourth image and the fifth image; after turning on the display device, combined with the virtual calibration pattern, determine the human eye simulation
  • a sixth two-dimensional image mapping relationship between the sixth image captured by the auxiliary camera and the virtual calibration pattern; the fifth two-dimensional image mapping relationship and the sixth two-dimensional image mapping relationship are represented as calibration parameters.
  • the alignment device used for the display device is also used to: determine the ground clearance of the equivalent physical calibration pattern according to the first spatial pose obtained by calibration, the human eye simulation auxiliary camera, and the equivalent physical calibration pattern The height value and the preset zoom ratio; according to the height above the ground and the preset zoom ratio, combined with the principle of light propagation, the equivalent physical calibration pattern is restored to the equivalent virtual calibration pattern on the ground and the second space pose of the virtual calibration pattern is calculated; Combined with the second space pose, the virtual calibration pattern is projected onto the imaging surface of the first camera to generate a fourth image.
  • the normalized parameter calibration correspondence is a mapping relationship between any normalized parameter and a one-to-one corresponding calibration parameter, wherein the normalized parameter is the position movement amount of the display device and/or the human eye simulation auxiliary camera Percentage of variable travel.
  • the device whose position changes includes: a display device and/or a human eye simulation auxiliary camera.
  • the alignment device for the display device is also used to: adjust the device whose position has changed to at least two different calibration positions in the variable space and repeat virtual and real opening and closing
  • the calibration method determines the normalized parameter calibration corresponding relationship; the normalized position parameter in the variable space is determined based on the actual position of the device, and the calibration parameter is determined according to the normalized position parameter and the normalized parameter calibration corresponding relationship.
  • the alignment device for the display device is also used to: adjust the device to at least 2 different calibration positions within the variable linear stroke, repeat the virtual and real opening and closing calibration method and determine the return The corresponding relationship of calibration parameters, wherein at least two different calibration positions include: the highest display reference position and the lowest display reference position of the device in the variable linear stroke.
  • the alignment device for the display device is also used to: adjust the device to at least 3 different calibration positions within the variable plane stroke, repeat the virtual and real opening and closing calibration method and determine the return One chemical parameter calibration correspondence, wherein at least 3 different calibration positions include: 3 limit reference positions of the equipment that are not collinear at the same time.
  • the alignment device used for the display device is also used to: adjust the device to at least 4 different calibration positions within the variable three-dimensional space stroke, repeat the virtual-real opening and closing calibration method and Determine the normalized parameter calibration correspondence, wherein at least 4 different calibration positions include: 4 limit reference positions of the equipment that are not coplanar at the same time.
  • the alignment device for the display device is also used to: adjust the device whose position has changed to at least 2 different calibration positions in the variable space, and repeat the virtual and real opening and closing calibration method at different calibration positions to obtain a calibration parameter set , wherein the calibration parameter group contains each calibration parameter corresponding to each calibration position; the normalization parameter group is determined according to all calibration positions and variable space; the normalization is determined by combining the calibration parameter group and the normalization parameter group fitting Parameter calibration correspondence.
  • the alignment device for the display device is also used for: adjusting the position of the human eye simulation auxiliary camera by mounting the human eye simulation auxiliary camera on the mechanical arm, and providing corresponding three-dimensional normalized coordinates.
  • a vehicle-mounted display system including: a driver monitoring system, configured to track the sight focus area of the user in the target vehicle, wherein the sight focus area includes at least one entity The reference object; the driving recorder, which is set to shoot the focus area of the line of sight to obtain the captured image; the vehicle-mounted controller, which is respectively connected with the driver monitoring system and the driving recorder, and executes any one of the above-mentioned alignment methods for the display device;
  • the head-up display is configured to project a virtual image to a preset position directly in front of the user, wherein the virtual marker in the virtual image is aligned with the physical reference object in the sight focus area.
  • a wearable device including: a gaze tracking device configured to track the sight focus area of the user in the target vehicle, wherein the sight focus area includes at least one entity reference object; the collection device is set to shoot the focus area of the line of sight to obtain the shot image; the terminal controller is connected to the line of sight tracking device and the collection device respectively, and executes any one of the above-mentioned alignment methods for the display device; the head-up display is set In order to project the virtual image to a preset position directly in front of the user, the virtual marker in the virtual image is aligned with the physical reference object in the sight focus area.
  • the present disclosure solves the technical problem in the related art that it is impossible for the user to obtain an "immersive" augmented reality experience with a good virtual-real alignment while moving the line of sight, which reduces the user's interest in using it.
  • a vehicle-mounted control device including: a processor; and a memory configured to store executable instructions of the processor; wherein the processor is configured to execute Any of the above alignment methods for display devices.
  • a computer-readable storage medium includes a stored computer program, wherein, when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute any of the above-mentioned One of the alignment methods used to display the device.
  • the disclosed technical content can be realized in other ways.
  • the device embodiments described above are only illustrative.
  • the division of the units may be a logical function division.
  • multiple units or components may be combined or may be Integrate into another system, or some features may be ignored, or not implemented.
  • the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of units or modules may be in electrical or other forms.
  • the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple units. Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.
  • each functional unit in each embodiment of the present invention may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit.
  • the above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.
  • the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer-readable storage medium.
  • the computer software product is stored in a storage medium , including several instructions to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in various embodiments of the present invention.
  • the aforementioned storage media include: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disc, etc., which can store program codes. .

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Computer Graphics (AREA)
  • Computer Hardware Design (AREA)
  • General Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • User Interface Of Digital Computer (AREA)
  • Controls And Circuits For Display Device (AREA)

Abstract

本发明公开了一种用于显示设备的对齐方法及对齐装置、车载显示***，涉及图像处理领域，该对齐方法包括：确定目标载具中的用户的视线聚焦区域；获取第一相机对原始场景进行采集的拍摄图像，并提取拍摄图像中的实体标识点；通过预先得到的标定参数，确定拍摄图像和在用户当前视场中虚拟图像之间的二维图像映射关系，其中，虚拟图像为显示设备待投影的图像；根据二维图像映射关系，将实体标识点投影至虚拟图像中，其中，虚拟图像中的虚拟标识物与视线聚焦区域中的实体参照物对齐。本发明解决了相关技术中无法使驾乘人员能够在视线移动的同时得到良好的虚实对齐的"沉浸式"增强现实的体验，降低了驾乘人员的使用兴趣的技术问题。

Description

用于显示设备的对齐方法及对齐装置、车载显示***
本申请要求于2021年10月28日提交中国专利局、申请号为202111267017.X、申请名称“用于显示设备的对齐方法及对齐装置、车载显示***”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本发明涉及图像处理技术领域，具体而言，涉及一种用于显示设备的对齐方法及对齐装置、车载显示***。
背景技术
随着“智能化”浪潮的兴起，计算机视觉、人工智能、虚拟\增强现实等技术在车辆的行驶和安全领域得到了广泛的应用。通过平视显示器(head-up display，HUD)使得用户不需要低头查看仪表的信息，从而始终保持抬头状态，降低因为低头与抬头之间忽略外界环境的快速变化而引起的安全隐患，很大程度上提升了车辆的驾驶体验。2012年，研究人员在HUD中加入了导航信息，这样使车辆能够根据自身位置得到下一时刻的行驶方向的提示，里程碑式地在HUD中引入了增强现实(augmented reality，AR)的概念，使得HUD可在虚拟图像中提示真实场景的信息，丰富了HUD的应用场景。
近年来，HUD逐步向沉浸式增强现实平视显示器(augmented reality head-up display,AR-HUD)发展，沉浸式AR-HUD要求用户眼中的虚拟图像与用户眼中虚拟图像所覆盖的真实场景有关联，如高亮车道线边界，提示规划路径，提示安全行车距离等等，极大地提高了用户体验。但同样地，沉浸式AR-HUD对虚实对齐及标定的要求也相应提高。当前，沉浸式AR-HUD的虚实对齐及其标定技术的主要缺陷包括：(1)在车载应用中，无法约束驾驶员的头部运动，因此AR-HUD所投影的虚像与使用者的视角的几何关系无法如AR头盔、AR眼镜等其他AR应用一般固定；(2)车载应用中，车辆高速运动，即便是很小的时延都会因车速放大进而引起对齐精度下降。因此发展跟随视线移动的AR-HUD的虚实对齐技术及其标定算法，并且提高算法性能，是使AR-HUD能够大规模商用的关键。
相关技术中，常见的虚实对齐技术包括两种：现有技术一，通过三维标定重投影技术，能够将三维的车外景物向人眼重投影并在截断在AR-HUD虚像面实现，但由于该技术对传感器和算力的要求非常高，所以无法大规模商用，而且为了降低成本，大多 数将车外景物约定在某一深度上，因此在此深度之外的景物重投影到AR-HUD虚像时，虚像与所覆盖的真实景物尺寸大小不一致，造成驾驶员视觉差异；现有技术二，通过二维对齐技术，直接根据标定时驾驶员视角所看到的固定场景“粘贴”对应的虚像内容，由于该技术标定缺乏定量指标，要求人眼、AR-HUD虚像面的位置和姿态应保证和标定时一致，因此不能兼容“跟随驾驶员视线移动”这一需求，该技术适用场景过于单一，仅适用于标注地面平面的环境，不能适应上下坡，也无法标注行人、车辆、信号***等高于地面平面的景物。
针对上述的问题，目前尚未提出有效的解决方案。
发明内容
本公开提供了一种用于显示设备的对齐方法及对齐装置、车载显示***，以至少解决相关技术中无法使驾乘人员能够在视线移动的同时得到良好的虚实对齐的“沉浸式”增强现实的体验，降低了驾乘人员的使用兴趣的技术问题。
根据本公开的一个方面，提供了一种用于显示设备的对齐方法，包括：确定目标载具中的用户的视线聚焦区域，其中，所述视线聚焦区域内至少包括一个实体参照物；获取第一相机对原始场景进行采集的拍摄图像，并提取所述拍摄图像中的实体标识点；通过预先得到的标定参数，确定所述拍摄图像和在用户当前视场中虚拟图像之间的二维图像映射关系，其中，所述虚拟图像为显示设备待投影的图像；根据所述二维图像映射关系，将所述实体标识点投影至所述虚拟图像中，其中，所述虚拟图像中的虚拟标识物与所述视线聚焦区域中的实体参照物对齐。
可选地，确定目标载具中的用户的视线聚焦区域的步骤，包括：基于所述用户的视线投放主体的坐标，确定所述用户对所述目标载具前方的聚焦注视线；将所述聚焦注视线与所述目标载具前方地面所处的平面进行交汇处理，得到所述用户的注视位置点；基于所述注视位置点，确定所述目标载具中的用户的视线聚焦区域。
可选地，根据所述显示设备和用户眼睛的位置状态，结合人眼模拟辅助相机通过虚实对齐标定方法确定所述标定参数。
可选地，所述对齐方法还包括：当所述显示设备和所述用户眼睛的位置保持在固定位置时，在该固定位置上结合所述人眼模拟辅助相机通过虚实开闭标定方法确定所述标定参数；或者，当所述显示设备和/或所述用户眼睛的位置改变时，通过调整位置发生改变的设备至多个不同标定位置重复所述虚实开闭标定方法并确定归一化参数标定对应关系，根据所述设备的实际位置以及所述归一化参数标定对应关系确定所述标定参数。
可选地，所述显示设备和所述用户眼睛的实际位置通过驾驶员监控***获取。
可选地，所述标定参数包括：第一相机采集的第一图像与所述人眼模拟辅助相机采集的第二图像之间的第一二维图像映射关系，所述人眼模拟辅助相机采集的第三图像与虚拟标定图样之间的第二二维图像映射关系。
可选地，结合人眼模拟辅助相机通过虚实对齐标定方法确定所述标定参数，包括：通过控制所述显示设备的开启与关闭，结合实物标定图样和虚拟标定图样，根据所述人眼模拟辅助相机和所述第一相机捕获的图像确定所述标定参数。
可选地，通过控制所述显示设备的开启与关闭，结合实物标定图样和虚拟标定图样，根据人眼模拟辅助相机和所述第一相机捕获的图像确定所述标定参数，包括：在关闭所述显示设备后，结合所述实物标定图样，确定所述第一相机采集的第一图像与所述人眼模拟辅助相机采集的第二图像之间的第三二维图像映射关系；在开启所述显示设备后，结合所述虚拟标定图样，确定所述人眼模拟辅助相机采集的第三图像与所述虚拟标定图样之间的第四二维图像映射关系；将所述第三二维图像映射关系和所述第四二维图像映射关系表征为所述标定参数。
可选地，所述实物标定图样放置于所述目标载具前方，所述虚拟标定图样显示在所述显示设备投影的虚像面上，其中，所述实物标定图样包括至少一个实物特征点，所述虚拟标定图样中包括至少一个虚拟特征点。
可选地，所述对齐方法还包括：所述实物标定图样由在所述人眼模拟辅助相机视场中高于且平行于所述实物标定图样的等效实物标定图样代替。
可选地，通过控制所述显示设备的开启与关闭，结合所述等效实物标定图样和所述虚拟标定图样，根据所述人眼模拟辅助相机和所述第一相机捕获的图像确定所述标定参数，包括：在关闭所述显示设备后，通过等效实物标定图样还原位于地面等效的虚拟标定图样并投影至所述第一相机生成的第四图像，以及所述人眼模拟辅助相机采集关于所述等效实物标定图样的第五图像，确定所述第四图像和所述第五图像之间的第五二维图像映射关系；在开启所述显示设备后，结合所述虚拟标定图样，确定所述人眼模拟辅助相机采集的第六图像与所述虚拟标定图样之间的第六二维图像映射关系；将所述第五二维图像映射关系和所述第六二维图像映射关系表征为所述标定参数。
可选地，通过等效实物标定图样还原位于地面等效的虚拟标定图样并投影至所述第一相机生成的第四图像，包括：根据标定获得的所述第一相机、所述人眼模拟辅助相机和所述等效实物标定图样的第一空间位姿，确定等效实物标定图样的离地高度值和预设缩放比；根据所述离地高度值和所述预设缩放比，结合光线传播原理将所述等 效实物标定图样还原位于地面等效的所述虚拟标定图样以及计算所述虚拟标定图样的第二空间位姿；结合所述第二空间位姿将所述虚拟标定图样投影至所述第一相机成像面生成所述第四图像。
可选地，所述归一化参数标定对应关系是任意归一化参数与一一对应的标定参数之间映射关系，其中，所述归一化参数为所述显示设备和/或所述人眼模拟辅助相机的位置移动量占可变行程的比例。
可选地，所述位置发生改变的设备包括：所述显示设备和/或所述人眼模拟辅助相机。
可选地，当所述显示设备和/或所述用户眼睛的位置改变时，通过调整位置发生改变的设备至多个不同标定位置重复所述虚实开闭标定方法并确定归一化参数标定对应关系，根据所述设备的实际位置以及所述归一化参数标定对应关系确定所述标定参数，包括：调整所述位置发生改变的设备至可变空间中的至少2个不同标定位置重复所述虚实开闭标定方法并确定所述归一化参数标定对应关系；基于所述设备的实际位置确定在所述可变空间上的归一化位置参数，根据所述归一化位置参数和所述归一化参数标定对应关系确定所述标定参数。
可选地，当所述可变空间为可变直线行程时，所述对齐方法包括：调整所述设备至可变直线行程内的至少2个不同标定位置重复所述虚实开闭标定方法并确定所述归一化参数标定对应关系，其中，所述至少2个不同标定位置包括：所述设备在可变直线行程的最高显示基准位置和最低显示基准位置。
可选地，当所述可变空间为可变平面行程时，所述对齐方法包括：调整所述设备至可变平面行程内的至少3个不同标定位置重复所述虚实开闭标定方法并确定所述归一化参数标定对应关系，其中，所述至少3个不同标定位置包括：所述设备3个不同时共线的极限基准位置。
可选地，当所述可变空间为可变三维空间行程时，所述对齐方法包括：调整所述设备至可变三维空间行程内的至少4个不同标定位置重复所述虚实开闭标定方法并确定所述归一化参数标定对应关系，其中，所述至少4个不同标定位置包括：所述设备4个不同时共面的极限基准位置。
可选地，调整发生位置改变的设备至可变空间中的至少2个不同标定位置重复所述虚实开闭标定方法并确定所述归一化参数标定对应关系，包括：调整所述位置发生改变的设备至所述可变空间中的至少2个不同标定位置，并在所述不同标定位置上重复所述虚实开闭标定方法获得标定参数组，其中，所述标定参数组中包含各标定参数 与各标定位置一一对应；根据全部所述标定位置和所述可变空间确定归一化参数组；结合所述标定参数组以及所述归一化参数组拟合确定所述归一化参数标定对应关系。
可选地，所述对齐方法还包括：通过机械臂挂载所述人眼模拟辅助相机调整所述人眼模拟辅助相机的位置，并提供对应的三维归一化坐标。
可选地，所述对齐方法还包括：结合所述二维图像映射关系和所述目标载具的姿态，将所述实体标识点投影至所述虚拟图像中。
根据本公开的另一方面，还提供了一种用于显示设备的对齐装置，包括：第一确定单元，设置为确定目标载具中的用户的视线聚焦区域，其中，所述视线聚焦区域内至少包括一个实体参照物；第一获取单元，设置为获取第一相机对原始场景进行采集的拍摄图像，并提取所述拍摄图像中的实体标识点；第一确定单元，设置为通过预先得到的标定参数，确定所述拍摄图像和在用户当前视场中虚拟图像之间的二维图像映射关系，其中，所述虚拟图像为显示设备待投影的图像；对齐单元，设置为根据所述二维图像映射关系，将所述实体标识点投影至所述虚拟图像中，其中，所述虚拟图像中的虚拟标识物与所述视线聚焦区域中的实体参照物对齐。
可选地，所述第一确定单元包括：第一确定模块，设置为基于所述用户的视线投放主体的坐标，确定所述用户对所述目标载具前方的聚焦注视线；注视点确定模块，设置为将所述聚焦注视线与所述目标载具前方地面所处的平面进行交汇处理，得到所述用户的注视位置点；第二确定模块，设置为基于所述注视位置点，确定所述目标载具中的用户的视线聚焦区域。
可选地，根据所述显示设备和用户眼睛的位置状态，结合人眼模拟辅助相机通过虚实对齐标定方法确定所述标定参数。
可选地，所述对齐装置还包括：第三确定单元，设置为在所述显示设备和所述用户眼睛的位置保持在固定位置时，在该固定位置上结合所述人眼模拟辅助相机通过虚实开闭标定方法确定所述标定参数；或者，第四确定单元，设置为在所述显示设备和/或所述用户眼睛的位置改变时，通过调整位置发生改变的设备至多个不同标定位置重复所述虚实开闭标定方法并确定归一化参数标定对应关系，根据所述设备的实际位置以及所述归一化参数标定对应关系确定所述标定参数。
可选地，所述显示设备和所述用户眼睛的实际位置通过驾驶员监控***获取。
可选地，所述标定参数包括：第一相机采集的第一图像与所述人眼模拟辅助相机采集的第二图像之间的第一二维图像映射关系，所述人眼模拟辅助相机采集的第三图像与虚拟标定图样之间的第二二维图像映射关系。
可选地，第三确定单元包括：第三确定模块，设置为通过控制所述显示设备的开启与关闭，结合实物标定图样和虚拟标定图样，根据所述人眼模拟辅助相机和所述第一相机捕获的图像确定所述标定参数。
可选地，第三确定模块包括：第一确定子模块，设置为在关闭所述显示设备后，结合所述实物标定图样，确定所述第一相机采集的第一图像与所述人眼模拟辅助相机采集的第二图像之间的第三二维图像映射关系；第二确定子模块，设置为在开启所述显示设备后，结合所述虚拟标定图样，确定所述人眼模拟辅助相机采集的第三图像与所述虚拟标定图样之间的第四二维图像映射关系；第三确定子模块，设置为将所述第三二维图像映射关系和所述第四二维图像映射关系表征为所述标定参数。
可选地，所述对齐装置还包括：代替单元，设置为所述实物标定图样由在所述人眼模拟辅助相机视场中高于且平行于所述实物标定图样的等效实物标定图样代替。
可选地，第三确定模块还包括：第四确定子模块，设置为在关闭所述显示设备后，通过等效实物标定图样还原位于地面等效的虚拟标定图样并投影至所述第一相机生成的第四图像，以及所述人眼模拟辅助相机采集关于所述等效实物标定图样的第五图像，确定所述第四图像和所述第五图像之间的第五二维图像映射关系；第五确定子模块，设置为在开启所述显示设备后，结合所述虚拟标定图样，确定所述人眼模拟辅助相机采集的第六图像与所述虚拟标定图样之间的第六二维图像映射关系；第六确定子模块，设置为将所述第五二维图像映射关系和所述第六二维图像映射关系表征为所述标定参数。
根据本公开的另一方面，还提供了一种车载显示***，包括：驾驶员监控***，设置为追踪目标载具中的用户的视线聚焦区域，其中，所述视线聚焦区域内至少包括一个实体参照物；行车记录仪，设置为对所述视线聚焦区域进行拍摄，得到拍摄图像；车载控制器，分别与所述驾驶员监控***和所述行车记录仪连接，执行上述任意一项所述的用于显示设备的对齐方法；平视显示器，设置为将虚拟图像投影至所述用户的正前方的预设位置，其中，所述虚拟图像中的虚拟标识物与所述视线聚焦区域中的实体参照物对齐。
根据本公开的另一方面，还提供了一种车载控制设备，包括：处理器；以及存储器，设置为存储所述处理器的可执行指令；其中，所述处理器配置为经由执行所述可执行指令来执行上述任意一项所述的用于显示设备的对齐方法。
根据本公开的另一方面，还提供了一种计算机可读存储介质，所述计算机可读存储介质包括存储的计算机程序，其中，在所述计算机程序运行时控制所述计算机可读 存储介质所在设备执行上述任意一项所述的用于显示设备的对齐方法。
在本公开中，采用先确定目标载具中的用户的视线聚焦区域，其中，视线聚焦区域内至少包括一个实体参照物，获取第一相机对原始场景进行采集的拍摄图像，并提取拍摄图像中的实体标识点，通过预先得到的标定参数，确定拍摄图像和在用户当前视场中虚拟图像之间的二维图像映射关系，其中，虚拟图像为显示设备待投影的图像，根据二维图像映射关系，将实体标识点投影至虚拟图像中，其中，虚拟图像中的虚拟标识物与视线聚焦区域中的实体参照物对齐。
在本公开中，预先进行标定得到标定参数后，通过拍摄装置对载具外的视线聚焦区域的实景进行拍摄，将拍摄到的画面内容的实体标识点投影到虚拟图像上，使得当前时刻用户视野中的虚拟图像中虚拟标识物与视线聚焦区域中的实体参照物对齐，用户眼中的虚拟图像与用户眼中虚拟图像所覆盖的真实场景关联，达到虚实对齐的“沉浸式”增强现实体验效果，提高了用户的体验和使用兴趣，进而解决了相关技术中无法使驾乘人员能够在视线移动的同时得到良好的虚实对齐的“沉浸式”增强现实的体验，降低了驾乘人员的使用兴趣的技术问题。
本公开中，通过虚拟标定图样和实物标定图样，无需三维重建即可建立用户视场下行车记录仪采集的图像和显示设备显示的虚像图之间的二维图像像素对应关系，仅需开闭显示设备即可实现，不仅操作简单，而且提高了标定速度，大幅降低了对硬件算力的要求。
本公开中，通过人眼模拟辅助相机安装于人眼位置，能够仿真人眼，实现标定流程工业化，大大丰富了使用场景。
附图说明
此处所说明的附图用来提供对本公开的进一步理解，构成本公开的一部分，本公开的示意性实施例及其说明用于解释本公开，并不构成对本公开的不当限定。在附图中：
图1是根据本发明实施例的一种用于实现用于显示设备的对齐方法的计算机终端(或移动设备)的硬件结构框图；
图2是根据本发明实施例的一种可选的用于显示设备的对齐方法的流程图；
图3是根据本发明实施例的一种可选的用于显示设备的标定方法的场景示意图一；
图4是根据本发明实施例的一种可选的三维视觉中小孔成像基本原理的示意图；
图5是根据本发明实施例的一种可选的用于显示设备的标定方法的场景示意图二；
图6是根据本发明实施例的一种可选的用于显示设备的标定方法的场景示意图三；
图7是根据本发明实施例的一种可选的的对齐效果图；
图8是根据本发明实施例的一种用于显示设备的对齐装置的示意图。
具体实施方式
为了使本技术领域的人员更好地理解本发明方案，下面将结合本公开中的附图，对本公开中的技术方案进行清楚、完整地描述，显然，所描述的实施例仅仅是本公开一部分的实施例，而不是全部的实施例。基于本公开中的实施例，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例，都应当属于本公开保护的范围。
需要说明的是，本公开的说明书和权利要求书及上述附图中的术语“第一”、“第二”等是用于区别类似的对象，而不必用于描述特定的顺序或先后次序。应该理解这样使用的数据在适当情况下可以互换，以便这里描述的本公开中各实施例能够以除了在这里图示或描述的那些以外的顺序实施。此外，术语“包括”和“具有”以及他们的任何变形，意图在于覆盖不排他的包含，例如，包含了一系列步骤或单元的过程、方法、***、产品或设备不必限于清楚地列出的那些步骤或单元，而是可包括没有清楚地列出的或对于这些过程、方法、产品或设备固有的其它步骤或单元。
首先，在对本公开各实施例进行描述的过程中出现的部分名词或术语适用于如下解释：
增强现实平视显示器(augmented reality head-up display，AR-HUD)：显示器位于驾驶员目视前方的仪表盘前方，或者将显示器覆盖于挡风玻璃上，投影设备将虚像图像内容显示于增强现实平视显示器的虚像面上，且虚拟图像内容和虚拟图像所覆盖的真实场景有关联，如提示、标记、透明化等。
驾驶员监控***(driver monitor system，DMS)：通过视觉方法检测驾驶员行为，如闭眼、眨眼、凝视方向、头部运动等。
增强现实(augmented reality，AR)：通过计算机图形技术和可视化技术产生现实环境中不存在的虚拟对象，并通过显示技术“嵌入”到真实环境中与融为一体。
虚实对齐(virtual-real registration)：增强现实中实现生成的虚拟图像与真实场景对齐。
跟随视线移动：指当驾驶员视线移动时，本发明能够实时计算出AR-HUD虚像应该所处的位置，使得在驾驶员视线中，虚像总是能够与实景对齐。
本发明下述各实施例可以应用于各种装载有增强现实平视显示器的载具，载具类型包括但不限于：自行车、汽车、驾驶模拟器、AR窗户等，也可以应用于各种与使用者的几何位姿不固定的AR投影设备中，例如，裸眼AR等。
在本发明中，提出了一种用于AR-HUD的跟随视线移动的虚实对齐技术及其标定算法，在依据本发明中的标定步骤进行标定后，驾驶员监控***能够计算当前时刻驾驶员的注视点，车载计算机经由注视点计算当前时刻驾驶员所见车外实景在行车记录仪中的画面，进一步地将行车记录仪中的画面依据本发明实施例提出的对齐方法投影到AR-HUD虚像上，使得当前时刻驾驶员视野中的AR-HUD虚像画面能够完全覆盖车外实景，达到虚实对齐的“沉浸式”增强现实体验，本发明提出一种跟随驾驶员视线移动的虚实对齐模型，能够平衡算力、成本以及用户体验的需求，不需要三维重建的过程也不需要复杂的相机内外参标定，在主流车机上实时性能够达到相机的帧率。
实施例一
根据本发明实施例，还提供了一种用于显示设备的对齐方法实施例，需要说明的是，在附图的流程图示出的步骤可以在诸如一组计算机可执行指令的计算机***中执行，并且，虽然在流程图中示出了逻辑顺序，但是在某些情况下，可以以不同于此处的顺序执行所示出或描述的步骤。
本申请实施例一所提供的方法实施例可以在移动终端、计算机终端或者类似的运算装置中执行。图1示出了一种用于实现用于显示设备的对齐方法的计算机终端(或移动设备)的硬件结构框图。如图1所示，计算机终端10(或移动设备10)可以包括一个或多个(图中采用102a、102b，……，102n来示出)处理器102(处理器102可以包括但不限于微处理器MCU或可编程逻辑器件FPGA等的处理装置)、用于存储数据的存储器104、以及用于通信功能的传输模块106。除此以外，还可以包括：显示器、输入/输出接口(I/O接口)、通用串行总线(USB)端口(可以作为I/O接口的端口中的一个端口被包括)、网络接口、电源和/或相机。本领域普通技术人员可以理解，图1所示的结构仅为示意，其并不对上述电子装置的结构造成限定。例如，计算机终端10还可包括比图1中所示更多或者更少的组件，或者具有与图1所示不同的配置。
应当注意到的是上述一个或多个处理器102和/或其他数据处理电路在本文中通常可以被称为“数据处理电路”。该数据处理电路可以全部或部分的体现为软件、硬件、固件或其他任意组合。此外，数据处理电路可为单个独立的处理模块，或全部或部分 的结合到计算机终端10(或移动设备)中的其他元件中的任意一个内。如本申请实施例中所涉及到的，该数据处理电路作为一种处理器控制(例如与接口连接的可变电阻终端路径的选择)。
存储器104可用于存储应用软件的软件程序以及模块，如本发明实施例中的用于显示设备的对齐方法对应的程序指令/数据存储装置，处理器102通过运行存储在存储器104内的软件程序以及模块，从而执行各种功能应用以及数据处理，即实现上述的应用程序的漏洞检测方法。存储器104可包括高速随机存储器，还可包括非易失性存储器，如一个或者多个磁性存储装置、闪存、或者其他非易失性固态存储器。在一些实例中，存储器104可进一步包括相对于处理器102远程设置的存储器，这些远程存储器可以通过网络连接至计算机终端10。上述网络的实例包括但不限于互联网、企业内部网、局域网、移动通信网及其组合。
传输装置106用于经由一个网络接收或者发送数据。上述的网络具体实例可包括计算机终端10的通信供应商提供的无线网络。在一个实例中，传输装置106包括一个网络适配器(Network Interface Controller，NIC)，其可通过基站与其他网络设备相连从而可与互联网进行通讯。在一个实例中，传输装置106可以为射频(Radio Frequency，RF)模块，其用于通过无线方式与互联网进行通讯。
显示器可以例如触摸屏式的液晶显示器(LCD)，该液晶显示器可使得用户能够与计算机终端10(或移动设备)的用户界面进行交互。
根据本发明实施例，提供了一种用于显示设备的对齐方法实施例，需要说明的是，在附图的流程图示出的步骤可以在诸如一组计算机可执行指令的计算机***中执行，并且，虽然在流程图中示出了逻辑顺序，但是在某些情况下，可以以不同于此处的顺序执行所示出或描述的步骤。
图2是根据本发明实施例的一种可选的用于显示设备的对齐方法的流程图，如图2所示，该方法包括如下步骤：
步骤S202，确定目标载具中的用户的视线聚焦区域，其中，视线聚焦区域内至少包括一个实体参照物；
步骤S204，获取第一相机对原始场景进行采集的拍摄图像，并提取拍摄图像中的实体标识点；
步骤S206，通过预先得到的标定参数，确定拍摄图像和在用户当前视场中虚拟图像之间的二维图像映射关系，其中，虚拟图像为显示设备待投影的图像；
步骤S208，根据二维图像映射关系，将实体标识点投影至虚拟图像中，其中，虚拟图像中的虚拟标识物与视线聚焦区域中的实体参照物对齐。
通过上述步骤，可以先确定目标载具中的用户的视线聚焦区域，其中，视线聚焦区域内至少包括一个实体参照物，获取第一相机对原始场景进行采集的拍摄图像，并提取拍摄图像中的实体标识点，通过预先得到的标定参数，确定拍摄图像和在用户当前视场中虚拟图像之间的二维图像映射关系，其中，虚拟图像为显示设备待投影的图像，根据二维图像映射关系，将实体标识点投影至虚拟图像中，其中，虚拟图像中的虚拟标识物与视线聚焦区域中的实体参照物对齐。在该实施例中，预先进行标定得到标定参数后，通过拍摄装置对载具外的视线聚焦区域的实景进行拍摄，将拍摄到的画面内容的实体标识点投影到虚拟图像上，使得当前时刻用户视野中的虚拟图像中虚拟标识物与视线聚焦区域中的实体参照物对齐，用户眼中的虚拟图像与用户眼中虚拟图像所覆盖的真实场景关联，达到虚实对齐的“沉浸式”增强现实体验效果，提高了用户的体验和使用兴趣，进而解决了相关技术中无法使驾乘人员能够在视线移动的同时得到良好的虚实对齐的“沉浸式”增强现实的体验，降低了驾乘人员的使用兴趣的技术问题。
在本发明实施例中，以汽车载具为例进行示意性说明，其中，AR-HUD安装于驾驶员正前方方向盘后的仪表台前方，虚像画面投影至AR-HUD所在的虚像面上，行车记录仪(记为Cam adas)安装于车内后视镜处，朝向汽车前进方向。
可选的，显示设备的对齐方法可应用于车载增强现实***中,显示设备为增强现实平视显示器AR-HUD。
下面结合各步骤对本发明实施例进行详细说明。
步骤S202，确定目标载具中的用户的视线聚焦区域，其中，视线聚焦区域内至少包括一个实体参照物。
在本发明实施例中，目标载具包括但不限于：自行车、汽车、驾驶模拟器、AR窗户等，视线聚焦区域为用户视野中能观察到的车外真实场景主要观看的区域，实体参照物包括但不限于：车道线、路标、行人、前方车辆等任意场景中可辅助用户驾驶决策的参照物。
可选的，确定目标载具中的用户的视线聚焦区域的步骤，包括：基于用户的视线投放主体的坐标，确定用户对目标载具前方的聚焦注视线；将聚焦注视线与目标载具前方地面所处的平面进行交汇处理，得到用户的注视位置点；基于注视位置点，确定 目标载具中的用户的视线聚焦区域。
一种可选的，采用驾驶员监控***(DMS)追踪用户的视线投放主体的坐标，使用驾驶员监控***计算当前时刻用户的眼睛坐标(即视线投放主体的坐标)，之后能够确定由用户的眼睛到目标载具前方地面或实体参照物的聚焦注视线，得到用户的注视位置点。本申请实施例并不限制从采集的用户视线投放主体获得视线和视点的方法，可通过基于角膜瞳孔模型或基于深度学习确定用户视线和视点，同理，本申请实施例亦不限制基于注视位置点确定用户视野中的视线聚焦区域的方法，采用传统的几何方法或深度学习均可。
步骤S204，获取第一相机对原始场景进行采集的拍摄图像，并提取拍摄图像中的实体标识点。
在本发明实施例中，实体标识点为实体参照物上的像素点，以驾驶场景为例，实体参照物可以是指道路上的实体物，包括但不限于：车道线、路标、行人、前方车辆等任意场景中可辅助用户驾驶决策的参照物。由于实际参照物包含丰富的场景信息，用户决策往往依赖于场景中实际参照物，积极提取其包含的场景信息为沉浸式引导提供了可能。
步骤S206，通过预先得到的标定参数，确定拍摄图像和在用户当前视场中虚拟图像之间的二维图像映射关系，其中，虚拟图像为显示设备待投影的图像。
可选的，本实施例中可根据显示设备和用户眼睛的位置状态，结合人眼模拟辅助相机通过虚实对齐标定方法确定标定参数。具体的，标定过程中，采用人眼模拟辅助相机模拟人眼的位置，能精确量化用户眼睛在空间中的坐标，提高标定的效率同时可保证标定的精度，实现了标定的工业化流程。若直接采用人眼标定，则用户无法固定在标定位置，且将限制标定的空间范围，影响标定参数的精度。
在实际使用的场景中，不同用户对显示设备的高度需求各异，显示设备的高度并不是固定的，同样，使用过程中用户的视线并不是维持一成不变，坐姿改变、头部移动、场景颠簸等都能造成用户眼睛位置以及视线的改变。本实施例通过确定任意位置下用户眼睛与显示设备标定参数对应关系，从而始终跟随视线的移动保持显示设备的虚像与场景实景的虚实对齐状态，而不因用户坐姿改变、头部移动以及显示设备调整而终止。
另一种可选的，对齐方法还包括：当显示设备和用户眼睛的位置保持在固定位置时，在该固定位置上结合人眼模拟辅助相机通过虚实开闭标定方法确定标定参数；或者，当显示设备和/或用户眼睛的位置改变时，通过调整位置发生改变的设备至多个不 同标定位置重复虚实开闭标定方法并确定归一化参数标定对应关系，根据设备的实际位置以及归一化参数标定对应关系确定标定参数。
可选的，位置发生改变的设备包括：显示设备和/或人眼模拟辅助相机。
可选的，显示设备和用户眼睛的实际位置通过驾驶员监控***获取。
可选的，标定参数包括：第一相机采集的第一图像与人眼模拟辅助相机采集的第二图像之间的第一二维图像映射关系，人眼模拟辅助相机采集的第三图像与虚拟标定图样之间的第二二维图像映射关系。具体的，本申请不限制确定标定参数的顺序，可以先后计算亦可以并行计算。
在本发明实施例中，标定参数为预先通过标定操作得到的。图3是根据本发明实施例的一种可选的用于显示设备的标定方法的场景示意图，如图3所示，标定需要准备：实物标定图样(1)一幅，其形式不限(例如，标定板、标定布、地面喷绘标定图样等，标定图样的样式和图样上所显示的内容并不做具体限定)，虚拟标定图样(2)一幅，其分辨率与AR-HUD虚像分辨率一致，可由AR-HUD显示，以及辅助标定相机一台(本实施例中是指人眼模拟辅助相机，记为Cam eye)，其中，标定图样的样式包括但不限于：棋盘格、圆点格、图形码(一维码、二维码)等。Cam eye安装于人眼位置用于仿真人眼，在车前方地面铺设实物标定图样(1)，实物标定图样(1)中的特征点在辅助标定相机Cam eye和主标定相机Cam adas(本实施例中是指第一相机，设置为拍摄车外真实场景，可位于行车记录仪安装的位置，其类型包括但不限于：可见光相机、红外线相机等)中可分辨，虚拟标定图样(2)中的特征点在Cam eye中可分辨。
本实施例中的标定方式包括多种情况，下面分别对每种标定方式进行说明，本实施例下述各标定方式之间可以相互结合使用。
(1)第一种标定方式，显示设备开闭标定
可选的，结合人眼模拟辅助相机通过虚实对齐标定方法确定标定参数，包括：通过控制显示设备的开启与关闭，结合实物标定图样和虚拟标定图样，根据人眼模拟辅助相机和第一相机捕获的图像确定标定参数。
本实施例中显示设备可以包括但不限于：AR-HUD设备，本实施例以AR-HUD设备的虚像面进行示意说明。
可选的，通过控制显示设备的开启与关闭，结合实物标定图样和虚拟标定图样， 根据人眼模拟辅助相机和第一相机捕获的图像确定标定参数，包括：在关闭显示设备后，结合实物标定图样，确定第一相机采集的第一图像与人眼模拟辅助相机采集的第二图像之间的第三二维图像映射关系；在开启显示设备后，结合虚拟标定图样，确定人眼模拟辅助相机采集的第三图像与虚拟标定图样之间的第四二维图像映射关系；将第三二维图像映射关系和第四二维图像映射关系表征为标定参数。
可选的，实物标定图样放置于目标载具前方，虚拟标定图样显示在显示设备投影的虚像面上，其中，实物标定图样包括至少一个实物特征点，虚拟标定图样中包括至少一个虚拟特征点。
如图3所示，可以先关闭AR-HUD虚像面，使用Cam adas与Cam eye拍摄标定图样(1)(即实物标定图样，位于地面上)，根据所拍摄实物标定图样(1)的特征点，计算两者之间的二维图像变换关系H adas2eye(即第三二维图像映射关系)；之后打开AR-HUD虚像面，使用Cam eye拍摄虚拟标定图样(2)(即显示设备所显示的虚像面，该虚像面在前挡玻璃前，与地面垂直)，根据所拍摄虚拟标定图样(2)的特征点(即第三图像的第四特征点)，计算拍摄虚拟标定图样(2)得到的特征点与最终在AR-HUD虚像面所显示的特征点(即虚像面中的第三特征点)之间的二维图像变换关系H eye2image(即第四二维图像变换关系)，最后得到了两组标定参数：二维图像变换关系H adas2eye和二维图像变换关系H eye2image。本申请实施例并不限制特征点提取，以及基于特征点确定二维图像变换关系的方法，可采用传统方法，例如，提取特征点的描述信息Ransac剔除错误匹配以及方程求解变换关系，亦可采用深度估计变换关系，例如HomographyNet。此外，实物标定图样和虚拟标定图样在获取两组标定参数时分别使用，特征点无需一一对应，故两者的样式亦无需相同。
在获取两组标定参数后的对齐步骤，Cam adas采集场景图像，提取场景图像中参照物的像素点，例如车道线所在的像素点，记为p,通过二维图像变换关系H adas2eye和二维图像变换关系H eye2image，计算AR-HUD中点亮的像素点p’＝H eye2image*H eye2image*p。点亮的像素点即为显示设备中显示的虚拟车道线，即在用户当前位置的视场下，显示设备显示的虚拟车道线覆盖在真实白色车道线上。
本申请实施例通过虚拟标定图样和实物标定图样，无需三维重建即可建立用户视 场下行车记录仪采集的图像和显示设备显示的虚像图之间的二维图像像素对应关系，仅需开闭显示设备即可实现，操作简单的同时提高了计算效率和标定速度，大幅降低了对硬件算力的要求。此外，人眼模拟辅助相机安装于人眼位置用于仿真人眼，有助于实现标定流程工业化，丰富了使用场景。
(2)第二种标定方式，等效页面标定
在本发明实施例中，标定操作若以真实地面作为AR-HUD所覆盖的真实场景区域，在实际情况下，被覆盖的真实地面距离车头约30-100米左右，所以，标定工作对场地的空间要求以及实物标定图样(1)的尺寸要求会极高。为了进一步适应不同标定场景的实际情况，本申请实施例通过等效实物标定图样缩短标定过程中实物标定图样(1)所需要的空间距离。
可选的，对齐方法还包括：实物标定图样由在人眼模拟辅助相机视场中高于且平行于实物标定图样的等效实物标定图样代替。
图4是根据本发明实施例的一种可选的三维视觉中小孔成像基本原理的示意图，如图4所示，当物距更远的尺寸大的物(1)平行于物距更近的尺寸小的物(2)，且它们的物高之比与物距之比一致时，经小孔投影中心，在同一个成像面上成像(1)(2)是一致的。换句话说，依据三维视觉中小孔成像的基本原理，当物距更远的大物平行于物距更近的小物，且它们的物高之比与物距之比一致时，经小孔光学***成像一致。因此，可以依据该原理，根据不同标定场地的实际情况，AR-HUD的标定过程中实物标定图样(1)到标定车辆的距离可缩短。
可选的，通过控制显示设备的开启与关闭，结合等效实物标定图样和虚拟标定图样，根据人眼模拟辅助相机和第一相机捕获的图像确定标定参数，包括：在关闭显示设备后，通过等效实物标定图样还原位于地面等效的虚拟标定图样并投影至第一相机生成的第四图像，以及人眼模拟辅助相机采集关于等效实物标定图样的第五图像，确定第四图像和第五图像之间的第五二维图像映射关系；在开启显示设备后，结合虚拟标定图样，确定人眼模拟辅助相机采集的第六图像与虚拟标定图样之间的第六二维图像映射关系；将第五二维图像映射关系和第六二维图像映射关系表征为标定参数。
可选的，通过等效实物标定图样还原位于地面等效的虚拟标定图样并投影至第一相机生成的第四图像，包括：根据标定获得的第一相机、人眼模拟辅助相机和等效实物标定图样的第一空间位姿，确定等效实物标定图样的离地高度值和预设缩放比；根据离地高度值和预设缩放比，结合光线传播原理将等效实物标定图样还原位于地面等效的虚拟标定图样以及计算虚拟标定图样的第二空间位姿；结合第二空间位姿将虚拟 标定图样投影至第一相机成像面生成第四图像。
具体的，在第一种标定方式的基础上，根据小孔成像原理，实物标定图样由在人眼模拟辅助相机视场中高于且平行于实物标定图样的等效实物标定图样代替。图5是根据本发明实施例的一种可选的用于显示设备的标定方法的场景示意图二，如图5所示，在本发明实施例中引入“等效实物标定图样(3)”代替实物标定图样(1)，在Cam eye视场中使用高于且平行于实物标定图样(1)的等效实物标定图样(3)代替实物标定图样(1)，等效实物标定图样(3)与实物标定图样(1)的缩放比相当于等效实物标定图样(3)的离地高度与Cam eye的离地高度之比，也可以相当于实物标定图样(1)到Cam eye的水平距离与等效实物标定图样(3)到Cam eye的水平距离之比，其余标定设备的布置和要求与图3一致，不再赘述。依据小孔成像的原理，可极大的缩短标定工作距离，降低对标定场地的要求，同时等效实物标定图样(3)相对于实物标定图样(1)的尺寸可减小。
由于等效实物标定图样(3)仅仅适配Cam eye视场，其投影关系并不满足Cam adas的视场，因此可以分别计算Cam eye、Cam adas与实物标定图样(3)三者的位姿关系，并依据等效实物标定图样(3)的离地高度还原出被等效的虚拟标定图样(1)，再重投影到Cam adas，其中，重投影是指通过对Cam eye视场中的等效实物标定图样(3)的像素点投影到Cam adas视场中产生新的图像的像素点。如图5所示，可以独立标定第一相机Cam eye和人眼模拟辅助相机Cam adas的相机内参(即与相机本身特性相关的参数，例如相机的焦距、像素大小等)，本申请不限制相机标定的方法。基于相机内参标定Cam eye、Cam adas和等效实物标定图样(3)的空间位姿(即第一空间位姿)，空间位姿不仅提供设备之间的位置关系，进一步可提供等效实物标定图样(3)距离地面高度h和预设缩放比信息；根据等效实物标定图样(3)距离地面高度h(本实施例是指等效实物标定图样的离地高度值)以及光线沿着线传播原理，计算位于地面上的虚拟标定图样(1)的空间位姿(本实施例是指第二空间位姿)，最后将虚拟标定图样(1)的特征点重投影至Cam adas，得到生成新的图像(即投影至第一相机成像面生成第四图像)。
在获取两组标定参数后的对齐步骤，与第一种标定方法所包含的步骤一致，不再赘述。本实施例中，结合几何关系和小孔成像原理，通过引入等效标定平面推算虚拟地面并进行重投影，降低了可极大的缩短标定工作距离，降低对标定场地的要求。
(3)第三种标定方式，可变空间标定
当所述显示设备和所述用户眼睛的位置保持在固定位置时，在固定位置上可采用第一种标定方式和第二种标定方式获取标定参数。
然而用户的视线并不是维持一成不变，坐姿改变、头部移动、场景颠簸等都能造成用户眼睛位置以及视线的改变，当所述显示设备和/或所述用户眼睛的位置改变时，通过调整位置发生改变的设备至多个不同标定位置，在多个不同标定位置上重复第一种标定方式和第二种标定方式确定归一化参数标定对应关系，根据所述设备的实际位置以及所述归一化参数标定对应关系确定所述标定参数。
可选的，归一化参数标定对应关系是任意归一化参数与一一对应的标定参数之间映射关系，其中，归一化参数为显示设备和/或人眼模拟辅助相机的位置移动量占可变行程的比例。
可选的，当显示设备和/或用户眼睛的位置改变时，通过调整位置发生改变的设备至多个不同标定位置重复虚实开闭标定方法并确定归一化参数标定对应关系，根据设备的实际位置以及归一化参数标定对应关系确定标定参数，包括：调整位置发生改变的设备至可变空间中的至少2个不同标定位置重复虚实开闭标定方法并确定归一化参数标定对应关系；基于设备的实际位置确定在可变空间上的归一化位置参数，根据归一化位置参数和归一化参数标定对应关系确定标定参数。
本实施例中，可变空间包括的类型包括但不限于：可变直线行程、可变平面行程、可变三维空间行程。本实施例中，可改变的位置(行程)使用归一化参数表示，通过预标定的若干组“归一化参数——标定参数”的对应关系，可拟合出任意归一化参数对应的标定参数。对位置发生改变的设备，本申请实施例不限制行程信息获取的方法，例如，可通过外部传感器检测提供。
上述“若干”的最小值，可改变的位置若为近似线性(如AR-HUD虚像面上下调节)，应大于等于2，且这若干组对应关系不同时共点；可改变的位置若为近似面阵(如人眼在DMS相机内的移动)，应大于等于3，且这若干组对应关系不同时共线；更进一步地，可改变的位置若为近似立体范围(如模拟人眼相机在眼盒中的移动)，应大于等于4，且这若干组对应关系不同时共面。
可选的，当可变空间为可变直线行程时，对齐方法包括：调整设备至可变直线行程内的至少2个不同标定位置重复虚实开闭标定方法并确定归一化参数标定对应关系，其中，至少2个不同标定位置包括：设备在可变直线行程的最高显示基准位置和最低显示基准位置。
可选的，调整发生位置改变的设备至可变空间中的至少2个不同标定位置重复虚实开闭标定方法并确定归一化参数标定对应关系，包括：调整位置发生改变的设备至可变空间中的至少2个不同标定位置，并在不同标定位置上重复虚实开闭标定方法获得标定参数组，其中，标定参数组中包含各标定参数与各标定位置一一对应；根据全部标定位置和可变空间确定归一化参数组；结合标定参数组以及归一化参数组拟合确定归一化参数标定对应关系。
以移动AR-HUD为例，AR-HUD可调节虚像面的N个不同的行程(其中，N>＝2，表示至少应具有最高和最低两个行程)，之后进行标定工作，得到N组的标定参数，N组标定参数和分别对应AR-HUD虚像面归一化行程的N个0-1之间的值，可确定出归一化参数标定对应关系。此后，以AR-HUD虚像面归一化行程作为单一自变量(即AR-HUD在一定范围内可以上下移动，移动比例即为单一自变量，例如，如果移动范围为10mm，移动1mm的移动比例为0.1(即单一自变量为0.1))，结合归一化参数标定对应关系便求出AR-HUD虚像面移动于该位置时对应的标定参数。同理，用户眼睛在可变直线行程上移动时，通过人眼模拟辅助相机模拟人眼移动移动并且通过上述方法确定出任意位置对应的标定参数。
图6是根据本发明实施例的一种可选的用于显示设备的标定方法的场景示意图三，如图6所示，以AR-HUD虚像面移动为例，在本发明实施例中，显示设备的位置可以在可变直线行程调整，调整到最高时即为最高显示基准位置，调整到最低时即为最低显示基准位置。AR-HUD虚像面调整上下N＝2个极限行程，即调整到最高显示基准位置与最低显示基准位置，当AR-HUD虚像面调至最高，即归一化行程值为1时，在关闭AR-HUD虚像面后，使用Cam adas与Cam eye拍摄1#标定图样(1)，根据所拍摄实物1#标定图样(1)的特征点，计算两者之间的二维图像变换关系H (1) adas2eye，之后打开AR-HUD虚像面，使用Cam eye拍摄1#HUD虚像面(即显示设备所显示的虚像面，该虚像面在前挡玻璃前，与地面垂直)，根据所拍摄1#HUD虚像面的特征点，计算拍摄1#HUD虚像面得到的特征点与最终在AR-HUD虚像面所显示的特征点之间的二维图像变换关系H (1) eye2image。当AR-HUD虚像面调至最低，即归一化行程值为0时，在关闭AR-HUD虚像面后，使用Cam adas与Cam eye拍摄2#标定图样(1)，根据所拍摄实物2#标定图样(1)的特征点，计算两者之间的二维图像变换关系H (2) adas2eye，之后打开AR-HUD虚像面，使用Cam eye拍摄2#HUD虚像面，根据所拍摄2#HUD虚像面的特征点，计算拍摄2#HUD虚像面得到的特征点与 最终在AR-HUD虚像面所显示的特征点之间的二维图像变换关系H (2) eye2image，最后得到了2个基准位置上两组标定参数：二维图像变换关系H (1) adas2eye、二维图像变换关系H (1) eye2image、二维图像变换关系H (2) adas2eye和二维图像变换关系H (2) eye2image。结合标定参数组以及对应的归一化参数组拟合确定归一化参数标定对应关系，本申请不限制建立对应关系的数学方法，例如，采用插值或拟合算法。
在随后的对齐步骤中，以AR-HUD虚像面的行程移动为例，在本发明实施例中，车载计算机提取真实场景(如车道线)在行车记录仪图像(Cam adas)中所对应的像素点，根据此时AR-HUD虚像面的行程以及归一化参数标定对应关系，计算AR-HUD虚像面在当前行程时的标定参数，再将行车记录仪中的车道线像素点映射到AR-HUD虚像上的像素点，此时，在驾驶员视线中，AR-HUD虚像显示的车道线等将覆盖在驾驶员视线中真实的车道线，且可以根据不同驾驶员的习惯视线方向，上下调整AR-HUD的虚像画面，实现虚实对齐。
可选的，当可变空间为可变平面行程时，对齐方法包括：调整设备至可变平面行程内的至少3个不同标定位置重复虚实开闭标定方法并确定归一化参数标定对应关系，其中，至少3个不同标定位置包括：设备3个不同时共线的极限基准位置。
在本发明实施例中，以移动用户的眼睛为例，通过DMS拍摄人眼模拟辅助相机在驾驶室内的移动位置，该移动位置可以近似在一个二维平面内(即预设平面区域)，该二维平面的至少三个顶点的归一化坐标可以设置为(0,0)，(0,1)，(1,1)。
在本发明实施例中，以四个顶点对可变平面的归一化坐标进行示意说明。人眼模拟辅助相机移动到驾驶室内可变平面行程内的左下极限位置，即为归一化坐标(0,0)(即第一显示基准位置)，DMS记录此时人眼在DMS相机中的坐标，记为d (0)(即视线投放主体对应于第一显示基准位置的第一坐标)，人眼模拟辅助相机移动到驾驶室内可行空间的右下、右上、左上极限位置，即为归一化坐标(0,1)，(1,1)，(1,0)，DMS记录此时三个位置人眼在DMS相机中的坐标，记为d (1)(即视线投放主体对应于第二显示基准位置的第二坐标)、d (2)(即视线投放主体对应于第三显示基准位置的第三坐标)、d (3)(即视线投放主体对应于第四显示基准位置的第四坐标)，根据d (0),d (1)，d (2)，d (3)与归一化坐标(0,0),(0,1),(1,1),(1,0)的对应关系，能够得到拟合归一化矩阵H normal，基于DMS记录人眼在DMS相机中的坐标和拟合归一化矩阵H normal，能够得到此时人眼在预设平面区域内的归一化坐标。
在本发明实施例中，由于人眼的移动难以定量，因此以移动的极限位置作为标定依据，即以第一显示基准位置、第二显示基准位置、第三显示基准位置和第四显示基准位置为四个极限位置，人眼在驾驶室内可移动的M个不同时共线的极限位置(M>＝3)，之后分别进行标定工作，得到M组的标定参数，M组标定参数分别对应DMS拍摄的人眼在驾驶室内的M个二维归一化位置的值，结合M组标定参数组以及对应的M个二维归一化位置的值拟合确定归一化参数标定对应关系，本申请不限制建立对应关系的数学方法，例如，采用插值或拟合算法。随后以DMS拍摄的人眼在驾驶室内二维归一化位置的值作为单一自变量，可以获得人眼在驾驶室内任意位置时对应的标定参数。
在随后的对齐步骤中，以人眼的移动为例，在本发明实施例中，车载计算机提取真实场景(如车道线)在行车记录仪图像(Cam adas)中所对应的像素点，根据此时人眼在DMS相机中的坐标(二维坐标)，结合拟合归一化矩阵H normal，确定人眼在预设平面区域内的归一化坐标，根据此时人眼在预设平面区域内的归一化坐标以及归一化参数标定对应关系，计算人眼在当前行程时的标定参数，再将行车记录仪中的车道线像素点映射到AR-HUD虚像上的像素点，此时，在驾驶员视线中，AR-HUD虚像显示的车道线等将覆盖在驾驶员视线中真实的车道线，且可以根据不同驾驶员的习惯视线方向，上下调整AR-HUD的虚像画面，实现虚实对齐。
可选的，当可变空间为可变三维空间行程时，对齐方法包括：调整设备至可变三维空间行程内的至少4个不同标定位置重复虚实开闭标定方法并确定归一化参数标定对应关系，其中，至少4个不同标定位置包括：设备4个不同时共面的极限基准位置。
在本发明实施例中，以四个不同时共面的极限基准位置标定归一化参数，与可变空间为可变二维平面类似，人眼模拟辅助相机移动到驾驶室内可变平面行程内的4个不同时共面的极限基准位置重复虚实标定方法，根据极限基准位置和对应的归一化坐标拟合归一化矩阵H normal2,基于DMS记录人眼在DMS相机中的坐标和拟合归一化矩阵H normal2，能够得到此时人眼在预设空间区域内的归一化坐标。
同理，在本发明实施例中，由于人眼的移动难以定量，因此以移动的极限位置作为标定依据，即以第一显示基准位置、第二显示基准位置、第三显示基准位置和第四显示基准位置为四个极限位置，人眼在驾驶室内可移动的P个不同时共面的极限位置(P>＝4)，之后分别进行标定工作，得到P组的标定参数，P组标定参数分别对应DMS拍摄的人眼在驾驶室内的P个三维归一化位置的值，结合P组标定参数组以及对应的P个三维归一化位置的值拟合确定归一化参数标定对应关系，本申请不限制建立对应 关系的数学方法，例如，采用插值或拟合算法。随后以DMS拍摄的人眼在驾驶室内三维归一化位置的值作为单一自变量，可以获得人眼在驾驶室内任意位置时对应的标定参数。
在随后的对齐步骤中，以人眼的移动为例，在本发明实施例中，车载计算机提取真实场景(如车道线)在行车记录仪图像(Cam adas)中所对应的像素点，根据此时人眼在DMS相机中的坐标，结合拟合归一化矩阵H normal2，确定人眼在预设平面区域内的归一化坐标(三维坐标)，根据此时人眼在预设平面区域内的归一化坐标以及归一化参数标定对应关系，计算人眼在当前行程时的标定参数，再将行车记录仪中的车道线像素点映射到AR-HUD虚像上的像素点，此时，在驾驶员视线中，AR-HUD虚像显示的车道线等将覆盖在驾驶员视线中真实的车道线，且可以根据不同驾驶员的习惯视线方向，上下调整AR-HUD的虚像画面，实现虚实对齐。
本申请实施例通过使用数学算法建立单一变量生成标定参数的函数，能够获取可移动设备任意位置下的标定参数，从而能够实现实时的动态对齐。可选的，对齐方法还包括：通过机械臂挂载人眼模拟辅助相机调整人眼模拟辅助相机的位置，并提供对应的三维归一化坐标。
在本发明实施例中，可以使用机械臂挂载辅助标定相机Cam eye，可控制机械臂实现模拟人眼在驾驶室三维空间内可行区域进行上下左右前后的移动。由于机械臂的引入，模拟人眼的Cam eye可以由机械臂或可探测深度的DMS相机给出它在可行区域中的三维归一化坐标(即预设平面区域内的归一化坐标，例如，定义可行区域某一个角为(0,0,0)，则距离这个角最远的角为(1,1,1))。
在本发明实施例中，若使用机械臂提供三维归一化坐标，则还需要标定机械臂坐标系和DMS相机坐标系的变换关系，若直接使用可探测深度的DMS相机提供三维归一化坐标，则无需两者之间的变换关系。
虽然本申请实施例仅仅说明了上述标定方式，但这些标定任意两者组合、或三者组合、或四者组合或五者组合也应当是本实施例的保护范围。
步骤S208，根据二维图像映射关系，将实体标识点投影至虚拟图像中，其中，虚拟图像中的虚拟标识物与视线聚焦区域中的实体参照物对齐。
图7是根据本发明实施例的一种可选的的对齐效果图，图7中101指示的框架为 AR-HUD虚像范围中所显示的虚拟车道线，虚拟车道线覆盖在Cam eye视场中的真实白色车道线(102)上。车载计算机通过语义识别算法(例如神经网络等)识别Cam adas图像中存在的车道线、行人、前方车辆等可辅助驾驶员驾驶决策的信息，并提取它们在Cam adas图像中所对应的像素，之后通过二维图像变换关系映射到AR-HUD虚像上，此时，在Cam eye视场中，AR-HUD虚像显示的车道线等将覆盖在Cam eye视场中真实的车道线，实现虚实对齐。
在本发明实施例中，可以能够实时处理输入的人眼位置及输出虚拟图像，实时性取决于DMS相机或Cam adas相机或AR-HUD显示屏中帧率最低者。
可选的，所述对齐方法还包括：结合所述二维图像映射关系和所述目标载具的姿态，将所述实体标识点投影至所述虚拟图像中。具体的，由于实际使用场景存在复杂的情况，目标载具整体状态存在姿态变化，例如，当车辆刹车或起步时，车头对应存在抬起或低头的改变，若仅依赖于正常平稳车头下的标定参数而不考虑目标载具的姿态，将导致用户视线中场景和虚像面的场景不一致，将影响对齐的效果。本申请实施例在二维图像映射关系的基础上进一步结合目标载具的姿态修正对齐，提高了虚实对齐的精度和鲁棒性，进一步扩大的应用场景范围，此外，本申请并不限制目标载具姿态的获取方式，例如，可采用外部传感器实时目标载具的姿态。
本申请实施例通过虚拟标定图样和实物标定图样，无需三维重建即可建立用户视场下行车记录仪采集的图像和显示设备显示的虚像图之间的二维图像像素对应关系，仅需开闭显示设备即可实现，操作简单的同时提高了计算效率和标定速度，大幅降低了对硬件算力的要求。此外，人眼模拟辅助相机安装于人眼位置用于仿真人眼，有助于实现标定流程工业化，丰富了使用场景。
实施例二
本实施例中提供的一种用于显示设备的对齐装置包含了多个实施单元，每个实施单元对应于上述实施例一中的各个实施步骤。
图8是根据本发明实施例的一种用于显示设备的对齐装置的示意图，如图8所示，该对齐装置可以包括：第一确定单元81、第一获取单元82、第一确定单元83、对齐单元84，其中，
第一确定单元81，设置为确定目标载具中的用户的视线聚焦区域，其中，视线聚 焦区域内至少包括一个实体参照物；
第一获取单元82，设置为获取第一相机对原始场景进行采集的拍摄图像，并提取拍摄图像中的实体标识点；
第一确定单元83，设置为通过预先得到的标定参数，确定拍摄图像和在用户当前视场中虚拟图像之间的二维图像映射关系，其中，虚拟图像为显示设备待投影的图像；
对齐单元84，设置为根据二维图像映射关系，将实体标识点投影至虚拟图像中，其中，虚拟图像中的虚拟标识物与视线聚焦区域中的实体参照物对齐。
上述用于显示设备的对齐装置，可以先通过第一确定单元81确定目标载具中的用户的视线聚焦区域，其中，视线聚焦区域内至少包括一个实体参照物，通过第一获取单元82获取第一相机对原始场景进行采集的拍摄图像，并提取拍摄图像中的实体标识点，通过第一确定单元83通过预先得到的标定参数，确定拍摄图像和在用户当前视场中虚拟图像之间的二维图像映射关系，其中，虚拟图像为显示设备待投影的图像，通过对齐单元84根据二维图像映射关系，将实体标识点投影至虚拟图像中，其中，虚拟图像中的虚拟标识物与视线聚焦区域中的实体参照物对齐。在该实施例中，预先进行标定得到标定参数后，通过拍摄装置对载具外的视线聚焦区域的实景进行拍摄，将拍摄到的画面内容的实体标识点投影到虚拟图像上，使得当前时刻用户视野中的虚拟图像中虚拟标识物与视线聚焦区域中的实体参照物对齐，用户眼中的虚拟图像与用户眼中虚拟图像所覆盖的真实场景关联，达到虚实对齐的“沉浸式”增强现实体验效果，提高了用户的体验和使用兴趣，进而解决了相关技术中无法使驾乘人员能够在视线移动的同时得到良好的虚实对齐的“沉浸式”增强现实的体验，降低了驾乘人员的使用兴趣的技术问题。
可选地，第一确定单元包括：第一确定模块，设置为基于用户的视线投放主体的坐标，确定用户对目标载具前方的聚焦注视线；注视点确定模块，设置为将聚焦注视线与目标载具前方地面所处的平面进行交汇处理，得到用户的注视位置点；第二确定模块，设置为基于注视位置点，确定目标载具中的用户的视线聚焦区域。
可选地，根据显示设备和用户眼睛的位置状态，结合人眼模拟辅助相机通过虚实对齐标定方法确定标定参数。
可选地，对齐装置还包括：第三确定单元，设置为在显示设备和用户眼睛的位置保持在固定位置时，在该固定位置上结合人眼模拟辅助相机通过虚实开闭标定方法确定标定参数；或者，第四确定单元，设置为在显示设备和/或用户眼睛的位置改变时，通过调整位置发生改变的设备至多个不同标定位置重复虚实开闭标定方法并确定归一 化参数标定对应关系，根据设备的实际位置以及归一化参数标定对应关系确定标定参数。
可选地，显示设备和用户眼睛的实际位置通过驾驶员监控***获取。
可选地，标定参数包括：第一相机采集的第一图像与人眼模拟辅助相机采集的第二图像之间的第一二维图像映射关系，人眼模拟辅助相机采集的第三图像与虚拟标定图样之间的第二二维图像映射关系。
可选地，第三确定单元包括：第三确定模块，设置为通过控制显示设备的开启与关闭，结合实物标定图样和虚拟标定图样，根据人眼模拟辅助相机和第一相机捕获的图像确定标定参数。
可选地，第三确定模块包括：第一确定子模块，设置为在关闭显示设备后，结合实物标定图样，确定第一相机采集的第一图像与人眼模拟辅助相机采集的第二图像之间的第三二维图像映射关系；第二确定子模块，设置为在开启显示设备后，结合虚拟标定图样，确定人眼模拟辅助相机采集的第三图像与虚拟标定图样之间的第四二维图像映射关系；第三确定子模块，设置为将第三二维图像映射关系和第四二维图像映射关系表征为标定参数。
可选地，对齐装置还包括：代替单元，设置为实物标定图样由在人眼模拟辅助相机视场中高于且平行于实物标定图样的等效实物标定图样代替。
可选地，第三确定模块还包括：第四确定子模块，设置为在关闭显示设备后，通过等效实物标定图样还原位于地面等效的虚拟标定图样并投影至第一相机生成的第四图像，以及人眼模拟辅助相机采集关于等效实物标定图样的第五图像，确定第四图像和第五图像之间的第五二维图像映射关系；第五确定子模块，设置为在开启显示设备后，结合虚拟标定图样，确定人眼模拟辅助相机采集的第六图像与虚拟标定图样之间的第六二维图像映射关系；第六确定子模块，设置为将第五二维图像映射关系和第六二维图像映射关系表征为标定参数。
可选地，用于显示设备的对齐装置还用于：在关闭显示设备后，通过等效实物标定图样还原位于地面等效的虚拟标定图样并投影至第一相机生成的第四图像，以及人眼模拟辅助相机采集关于等效实物标定图样的第五图像，确定第四图像和第五图像之间的第五二维图像映射关系；在开启显示设备后，结合虚拟标定图样，确定人眼模拟辅助相机采集的第六图像与虚拟标定图样之间的第六二维图像映射关系；将第五二维图像映射关系和第六二维图像映射关系表征为标定参数。
可选地，用于显示设备的对齐装置还用于：根据标定获得的第一相机、人眼模拟 辅助相机和等效实物标定图样的第一空间位姿，确定等效实物标定图样的离地高度值和预设缩放比；根据离地高度值和预设缩放比，结合光线传播原理将等效实物标定图样还原位于地面等效的虚拟标定图样以及计算虚拟标定图样的第二空间位姿；结合第二空间位姿将虚拟标定图样投影至第一相机成像面生成第四图像。
可选地，归一化参数标定对应关系是任意归一化参数与一一对应的标定参数之间映射关系，其中，归一化参数为显示设备和/或人眼模拟辅助相机的位置移动量占可变行程的比例。
可选地，位置发生改变的设备包括：显示设备和/或人眼模拟辅助相机。
可选地，当显示设备和/或用户眼睛的位置改变时，用于显示设备的对齐装置还用于：调整位置发生改变的设备至可变空间中的至少2个不同标定位置重复虚实开闭标定方法并确定归一化参数标定对应关系；基于设备的实际位置确定在可变空间上的归一化位置参数，根据归一化位置参数和归一化参数标定对应关系确定标定参数。
可选地，当可变空间为可变直线行程时，用于显示设备的对齐装置还用于：调整设备至可变直线行程内的至少2个不同标定位置重复虚实开闭标定方法并确定归一化参数标定对应关系，其中，至少2个不同标定位置包括：设备在可变直线行程的最高显示基准位置和最低显示基准位置。
可选地，当可变空间为可变平面行程时，用于显示设备的对齐装置还用于：调整设备至可变平面行程内的至少3个不同标定位置重复虚实开闭标定方法并确定归一化参数标定对应关系，其中，至少3个不同标定位置包括：设备3个不同时共线的极限基准位置。
可选地，当可变空间为可变三维空间行程时，用于显示设备的对齐装置还用于：调整设备至可变三维空间行程内的至少4个不同标定位置重复虚实开闭标定方法并确定归一化参数标定对应关系，其中，至少4个不同标定位置包括：设备4个不同时共面的极限基准位置。
可选地，用于显示设备的对齐装置还用于：调整位置发生改变的设备至可变空间中的至少2个不同标定位置，并在不同标定位置上重复虚实开闭标定方法获得标定参数组，其中，标定参数组中包含各标定参数与各标定位置一一对应；根据全部标定位置和可变空间确定归一化参数组；结合标定参数组以及归一化参数组拟合确定归一化参数标定对应关系。
可选地，用于显示设备的对齐装置还用于：通过机械臂挂载人眼模拟辅助相机调整人眼模拟辅助相机的位置，并提供对应的三维归一化坐标。
根据本发明实施例的另一方面，还提供了一种车载显示***，包括：驾驶员监控***，设置为追踪目标载具中的用户的视线聚焦区域，其中，视线聚焦区域内至少包括一个实体参照物；行车记录仪，设置为对视线聚焦区域进行拍摄，得到拍摄图像；车载控制器，分别与驾驶员监控***和行车记录仪连接，执行上述任意一项的用于显示设备的对齐方法；平视显示器，设置为将虚拟图像投影至用户的正前方的预设位置，其中，虚拟图像中的虚拟标识物与视线聚焦区域中的实体参照物对齐。
根据本发明实施例的另一方面，还提供了一种可穿戴设备，包括：视线追踪装置，设置为追踪目标载具中的用户的视线聚焦区域，其中，视线聚焦区域内至少包括一个实体参照物；采集装置，设置为对视线聚焦区域进行拍摄，得到拍摄图像；终端控制器，分别与视线追踪装置和采集装置连接，执行上述任意一项的用于显示设备的对齐方法；平视显示器，设置为将虚拟图像投影至用户的正前方的预设位置，其中，虚拟图像中的虚拟标识物与视线聚焦区域中的实体参照物对齐。
本公开解决了相关技术中无法使用户能够在视线移动的同时得到良好的虚实对齐的“沉浸式”增强现实的体验，降低了用户的使用兴趣的技术问题。
根据本发明实施例的另一方面，还提供了一种车载控制设备，包括：处理器；以及存储器，设置为存储处理器的可执行指令；其中，处理器配置为经由执行可执行指令来执行上述任意一项的用于显示设备的对齐方法。
根据本发明实施例的另一方面，还提供了一种计算机可读存储介质，计算机可读存储介质包括存储的计算机程序，其中，在计算机程序运行时控制计算机可读存储介质所在设备执行上述任意一项的用于显示设备的对齐方法。
上述本发明实施例序号仅仅为了描述，不代表实施例的优劣。
在本发明的上述实施例中，对各个实施例的描述都各有侧重，某个实施例中没有详述的部分，可以参见其他实施例的相关描述。
在本申请所提供的几个实施例中，应该理解到，所揭露的技术内容，可通过其它的方式实现。其中，以上所描述的装置实施例仅仅是示意性的，例如所述单元的划分，可以为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如多个单元或组件可以结合或者可以集成到另一个***，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，单元或模块的间接耦合或通信连接，可以是电性或其它的形式。
所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到 多个单元上。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的目的。
另外，在本发明各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。上述集成的单元既可以采用硬件的形式实现，也可以采用软件功能单元的形式实现。
所述集成的单元如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个计算机可读取存储介质中。基于这样的理解，本发明的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的全部或部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得一台计算机设备(可为个人计算机、服务器或者网络设备等)执行本发明各个实施例所述方法的全部或部分步骤。而前述的存储介质包括：U盘、只读存储器(ROM，Read-Only Memory)、随机存取存储器(RAM，Random Access Memory)、移动硬盘、磁碟或者光盘等各种可以存储程序代码的介质。
以上所述仅是本发明的优选实施方式，应当指出，对于本技术领域的普通技术人员来说，在不脱离本发明原理的前提下，还可以做出若干改进和润饰，这些改进和润饰也应视为本发明的保护范围。

Claims (34)

  1. 一种用于显示设备的对齐方法，包括：
    确定目标载具中的用户的视线聚焦区域，其中，所述视线聚焦区域内至少包括一个实体参照物；
    获取第一相机对原始场景进行采集的拍摄图像，并提取所述拍摄图像中的实体标识点；
    通过预先得到的标定参数，确定所述拍摄图像和在用户当前视场中虚拟图像之间的二维图像映射关系，其中，所述虚拟图像为显示设备待投影的图像；
    根据所述二维图像映射关系，将所述实体标识点投影至所述虚拟图像中，其中，所述虚拟图像中的虚拟标识物与所述视线聚焦区域中的实体参照物对齐。
  2. 根据权利要求1所述的对齐方法，其中，确定目标载具中的用户的视线聚焦区域的步骤，包括：
    基于所述用户的视线投放主体的坐标，确定所述用户对所述目标载具前方的聚焦注视线；
    将所述聚焦注视线与所述目标载具前方地面所处的平面进行交汇处理，得到所述用户的注视位置点；
    基于所述注视位置点，确定所述目标载具中的用户的视线聚焦区域。
  3. 根据权利要求1所述的对齐方法，其中，根据所述显示设备和用户眼睛的位置状态，结合人眼模拟辅助相机通过虚实对齐标定方法确定所述标定参数。
  4. 根据权利要求3所述的对齐方法，其中，所述对齐方法还包括：
    当所述显示设备和所述用户眼睛的位置保持在固定位置时，在该固定位置上结合所述人眼模拟辅助相机通过虚实开闭标定方法确定所述标定参数；或者，
    当所述显示设备和/或所述用户眼睛的位置改变时，通过调整位置发生改变的设备至多个不同标定位置重复所述虚实开闭标定方法并确定归一化参数标定对应关系，根据所述设备的实际位置以及所述归一化参数标定对应关系确定所述标定参数。
  5. 根据权利要求4所述的对齐方法，其中，所述显示设备和所述用户眼睛的实际位置 通过驾驶员监控***获取。
  6. 根据权利要求3所述的对齐方法，其中，所述标定参数包括：第一相机采集的第一图像与所述人眼模拟辅助相机采集的第二图像之间的第一二维图像映射关系，所述人眼模拟辅助相机采集的第三图像与虚拟标定图样之间的第二二维图像映射关系。
  7. 根据权利要求3所述的对齐方法，其中，结合人眼模拟辅助相机通过虚实对齐标定方法确定所述标定参数，包括：通过控制所述显示设备的开启与关闭，结合实物标定图样和虚拟标定图样，根据所述人眼模拟辅助相机和所述第一相机捕获的图像确定所述标定参数。
  8. 根据权利要求7所述的对齐方法，其中，通过控制所述显示设备的开启与关闭，结合实物标定图样和虚拟标定图样，根据人眼模拟辅助相机和所述第一相机捕获的图像确定所述标定参数，包括：
    在关闭所述显示设备后，结合所述实物标定图样，确定所述第一相机采集的第一图像与所述人眼模拟辅助相机采集的第二图像之间的第三二维图像映射关系；
    在开启所述显示设备后，结合所述虚拟标定图样，确定所述人眼模拟辅助相机采集的第三图像与所述虚拟标定图样之间的第四二维图像映射关系；
    将所述第三二维图像映射关系和所述第四二维图像映射关系表征为所述标定参数。
  9. 根据权利要求7所述的对齐方法，其中，所述实物标定图样放置于所述目标载具前方，所述虚拟标定图样显示在所述显示设备投影的虚像面上，其中，所述实物标定图样包括至少一个实物特征点，所述虚拟标定图样中包括至少一个虚拟特征点。
  10. 根据权利要求7所述的对齐方法，其中，所述对齐方法还包括：所述实物标定图样由在所述人眼模拟辅助相机视场中高于且平行于所述实物标定图样的等效实物标定图样代替。
  11. 根据权利要求10所述的对齐方法，其中，通过控制所述显示设备的开启与关闭，结合所述等效实物标定图样和所述虚拟标定图样，根据所述人眼模拟辅助相机和所述第一相机捕获的图像确定所述标定参数，包括：
    在关闭所述显示设备后，通过等效实物标定图样还原位于地面等效的虚拟标定图样并投影至所述第一相机生成的第四图像，以及所述人眼模拟辅助相机采集 关于所述等效实物标定图样的第五图像，确定所述第四图像和所述第五图像之间的第五二维图像映射关系；
    在开启所述显示设备后，结合所述虚拟标定图样，确定所述人眼模拟辅助相机采集的第六图像与所述虚拟标定图样之间的第六二维图像映射关系；
    将所述第五二维图像映射关系和所述第六二维图像映射关系表征为所述标定参数。
  12. 根据权利要求11所述的对齐方法，其中，通过等效实物标定图样还原位于地面等效的虚拟标定图样并投影至所述第一相机生成的第四图像，包括：
    根据标定获得的所述第一相机、所述人眼模拟辅助相机和所述等效实物标定图样的第一空间位姿，确定等效实物标定图样的离地高度值和预设缩放比；
    根据所述离地高度值和所述预设缩放比，结合光线传播原理将所述等效实物标定图样还原位于地面等效的所述虚拟标定图样以及计算所述虚拟标定图样的第二空间位姿；
    结合所述第二空间位姿将所述虚拟标定图样投影至所述第一相机成像面生成所述第四图像。
  13. 根据权利要求4所述的对齐方法，其中，所述归一化参数标定对应关系是任意归一化参数与一一对应的标定参数之间映射关系，其中，所述归一化参数为所述显示设备和/或所述人眼模拟辅助相机的位置移动量占可变行程的比例。
  14. 根据权利要求4所述的对齐方法，其中，所述位置发生改变的设备包括：所述显示设备和/或所述人眼模拟辅助相机。
  15. 根据权利要求4所述的对齐方法，其中，当所述显示设备和/或所述用户眼睛的位置改变时，通过调整位置发生改变的设备至多个不同标定位置重复所述虚实开闭标定方法并确定归一化参数标定对应关系，根据所述设备的实际位置以及所述归一化参数标定对应关系确定所述标定参数，包括：
    调整所述位置发生改变的设备至可变空间中的至少2个不同标定位置重复所述虚实开闭标定方法并确定所述归一化参数标定对应关系；
    基于所述设备的实际位置确定在所述可变空间上的归一化位置参数，根据所述归一化位置参数和所述归一化参数标定对应关系确定所述标定参数。
  16. 根据权利要求15所述的对齐方法，其中，当所述可变空间为可变直线行程时，所 述对齐方法包括：
    调整所述设备至可变直线行程内的至少2个不同标定位置重复所述虚实开闭标定方法并确定所述归一化参数标定对应关系，其中，所述至少2个不同标定位置包括：所述设备在可变直线行程的最高显示基准位置和最低显示基准位置。
  17. 根据权利要求15所述的对齐方法，其中，当所述可变空间为可变平面行程时，所述对齐方法包括：
    调整所述设备至可变平面行程内的至少3个不同标定位置重复所述虚实开闭标定方法并确定所述归一化参数标定对应关系，其中，所述至少3个不同标定位置包括：所述设备3个不同时共线的极限基准位置。
  18. 根据权利要求15所述的对齐方法，其中，当所述可变空间为可变三维空间行程时，所述对齐方法包括：
    调整所述设备至可变三维空间行程内的至少4个不同标定位置重复所述虚实开闭标定方法并确定所述归一化参数标定对应关系，其中，所述至少4个不同标定位置包括：所述设备4个不同时共面的极限基准位置。
  19. 根据权利要求15所述的对齐方法，其中，调整发生位置改变的设备至可变空间中的至少2个不同标定位置重复所述虚实开闭标定方法并确定所述归一化参数标定对应关系，包括：
    调整所述位置发生改变的设备至所述可变空间中的至少2个不同标定位置，并在所述不同标定位置上重复所述虚实开闭标定方法获得标定参数组，其中，所述标定参数组中包含各标定参数与各标定位置一一对应；
    根据全部所述标定位置和所述可变空间确定归一化参数组；
    结合所述标定参数组以及所述归一化参数组拟合确定所述归一化参数标定对应关系。
  20. 根据权利要求4所述的对齐方法，其中，所述对齐方法还包括：通过机械臂挂载所述人眼模拟辅助相机调整所述人眼模拟辅助相机的位置，并提供对应的三维归一化坐标。
  21. 根据权利要求1所述的对齐方法，其中，所述对齐方法还包括：
    结合所述二维图像映射关系和所述目标载具的姿态，将所述实体标识点投影至所述虚拟图像中。
  22. 一种用于显示设备的对齐装置，包括：
    第一确定单元，设置为确定目标载具中的用户的视线聚焦区域，其中，所述视线聚焦区域内至少包括一个实体参照物；
    第一获取单元，设置为获取第一相机对原始场景进行采集的拍摄图像，并提取所述拍摄图像中的实体标识点；
    第一确定单元，设置为通过预先得到的标定参数，确定所述拍摄图像和在用户当前视场中虚拟图像之间的二维图像映射关系，其中，所述虚拟图像为显示设备待投影的图像；
    对齐单元，设置为根据所述二维图像映射关系，将所述实体标识点投影至所述虚拟图像中，其中，所述虚拟图像中的虚拟标识物与所述视线聚焦区域中的实体参照物对齐。
  23. 根据权利要求22所述的对齐装置，其中，所述第一确定单元包括：
    第一确定模块，设置为基于所述用户的视线投放主体的坐标，确定所述用户对所述目标载具前方的聚焦注视线；
    注视点确定模块，设置为将所述聚焦注视线与所述目标载具前方地面所处的平面进行交汇处理，得到所述用户的注视位置点；
    第二确定模块，设置为基于所述注视位置点，确定所述目标载具中的用户的视线聚焦区域。
  24. 根据权利要求22所述的对齐装置，其中，根据所述显示设备和用户眼睛的位置状态，结合人眼模拟辅助相机通过虚实对齐标定方法确定所述标定参数。
  25. 根据权利要求24所述的对齐装置，其中，所述对齐装置还包括：
    第三确定单元，设置为在所述显示设备和所述用户眼睛的位置保持在固定位置时，在该固定位置上结合所述人眼模拟辅助相机通过虚实开闭标定方法确定所述标定参数；或者，
    第四确定单元，设置为在所述显示设备和/或所述用户眼睛的位置改变时，通过调整位置发生改变的设备至多个不同标定位置重复所述虚实开闭标定方法并确定归一化参数标定对应关系，根据所述设备的实际位置以及所述归一化参数标定对应关系确定所述标定参数。
  26. 根据权利要求24所述的对齐装置，其中，所述显示设备和所述用户眼睛的实际位 置通过驾驶员监控***获取。
  27. 根据权利要求24所述的对齐装置，其中，所述标定参数包括：第一相机采集的第一图像与所述人眼模拟辅助相机采集的第二图像之间的第一二维图像映射关系，所述人眼模拟辅助相机采集的第三图像与虚拟标定图样之间的第二二维图像映射关系。
  28. 根据权利要求24所述的对齐装置，其中，第三确定单元包括：第三确定模块，设置为通过控制所述显示设备的开启与关闭，结合实物标定图样和虚拟标定图样，根据所述人眼模拟辅助相机和所述第一相机捕获的图像确定所述标定参数。
  29. 根据权利要求28所述的对齐装置，其中，第三确定模块包括：
    第一确定子模块，设置为在关闭所述显示设备后，结合所述实物标定图样，确定所述第一相机采集的第一图像与所述人眼模拟辅助相机采集的第二图像之间的第三二维图像映射关系；
    第二确定子模块，设置为在开启所述显示设备后，结合所述虚拟标定图样，确定所述人眼模拟辅助相机采集的第三图像与所述虚拟标定图样之间的第四二维图像映射关系；
    第三确定子模块，设置为将所述第三二维图像映射关系和所述第四二维图像映射关系表征为所述标定参数。
  30. 根据权利要求29所述的对齐装置，其中，所述对齐装置还包括：代替单元，设置为所述实物标定图样由在所述人眼模拟辅助相机视场中高于且平行于所述实物标定图样的等效实物标定图样代替。
  31. 根据权利要求30所述的对齐装置，其中，第三确定模块还包括：
    第四确定子模块，设置为在关闭所述显示设备后，通过等效实物标定图样还原位于地面等效的虚拟标定图样并投影至所述第一相机生成的第四图像，以及所述人眼模拟辅助相机采集关于所述等效实物标定图样的第五图像，确定所述第四图像和所述第五图像之间的第五二维图像映射关系；
    第五确定子模块，设置为在开启所述显示设备后，结合所述虚拟标定图样，确定所述人眼模拟辅助相机采集的第六图像与所述虚拟标定图样之间的第六二维图像映射关系；
    第六确定子模块，设置为将所述第五二维图像映射关系和所述第六二维图像 映射关系表征为所述标定参数。
  32. 一种车载显示***，包括：
    驾驶员监控***，设置为追踪目标载具中的用户的视线聚焦区域，其中，所述视线聚焦区域内至少包括一个实体参照物；
    行车记录仪，设置为对所述视线聚焦区域进行拍摄，得到拍摄图像；
    车载控制器，分别与所述驾驶员监控***和所述行车记录仪连接，执行权利要求1至21中任意一项所述的用于显示设备的对齐方法；
    平视显示器，设置为将虚拟图像投影至所述用户的正前方的预设位置，其中，所述虚拟图像中的虚拟标识物与所述视线聚焦区域中的实体参照物对齐。
  33. 一种车载控制设备，包括：
    处理器；以及
    存储器，设置为存储所述处理器的可执行指令；
    其中，所述处理器配置为经由执行所述可执行指令来执行权利要求1至21中任意一项所述的用于显示设备的对齐方法。
  34. 一种计算机可读存储介质，所述计算机可读存储介质包括存储的计算机程序，其中，在所述计算机程序运行时控制所述计算机可读存储介质所在设备执行权利要求1至21中任意一项所述的用于显示设备的对齐方法。
PCT/CN2022/125475 2021-10-28 2022-10-14 用于显示设备的对齐方法及对齐装置、车载显示*** WO2023071834A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202111267017.X 2021-10-28
CN202111267017.XA CN114022565A (zh) 2021-10-28 2021-10-28 用于显示设备的对齐方法及对齐装置、车载显示***

Publications (1)

Publication Number Publication Date
WO2023071834A1 true WO2023071834A1 (zh) 2023-05-04

Family

ID=80058470

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/125475 WO2023071834A1 (zh) 2021-10-28 2022-10-14 用于显示设备的对齐方法及对齐装置、车载显示***

Country Status (2)

Country Link
CN (1) CN114022565A (zh)
WO (1) WO2023071834A1 (zh)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116295021A (zh) * 2023-05-25 2023-06-23 齐鲁工业大学(山东省科学院) 一种单目线结构光***中相机与激光器位置关系计算方法
CN117456012A (zh) * 2023-12-20 2024-01-26 绘见科技(深圳)有限公司 虚拟相机的视场角标定方法及装置、设备及存储介质

Families Citing this family (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114022565A (zh) * 2021-10-28 2022-02-08 虹软科技股份有限公司 用于显示设备的对齐方法及对齐装置、车载显示***
CN114821497A (zh) * 2022-02-24 2022-07-29 广州文远知行科技有限公司 目标物位置的确定方法、装置、设备及存储介质
CN115586648A (zh) * 2022-09-05 2023-01-10 华为技术有限公司 一种调节方法、装置和运载工具
CN115665400B (zh) * 2022-09-06 2024-05-28 东软集团股份有限公司 增强现实抬头显示成像方法、装置、设备以及存储介质
CN118092828A (zh) * 2022-11-25 2024-05-28 北京罗克维尔斯科技有限公司 信息显示方法、装置、设备、存储介质及车辆
CN115826766B (zh) * 2023-02-16 2023-04-21 北京睛采智能科技有限公司 基于显示模拟器的眼动目标获取装置、方法和***

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110148224A (zh) * 2019-04-04 2019-08-20 精电(河源)显示技术有限公司 Hud图像显示方法、装置及终端设备
CN110298924A (zh) * 2019-05-13 2019-10-01 西安电子科技大学 一种ar***中用于显示检测信息的坐标变换方法
WO2020166376A1 (ja) * 2019-02-15 2020-08-20 ソニー株式会社 画像処理装置、画像処理方法、及び、プログラム
CN113129224A (zh) * 2019-12-31 2021-07-16 精工爱普生株式会社 显示***、电子设备、移动体和显示方法
CN113240592A (zh) * 2021-04-14 2021-08-10 重庆利龙科技产业(集团)有限公司 基于ar-hud动态眼位下计算虚像平面的畸变矫正方法
US20210260998A1 (en) * 2018-08-08 2021-08-26 Bayerische Motoren Werke Aktiengesellschaft Method for Operating a Visual Field Display Device for a Motor Vehicle
CN114022565A (zh) * 2021-10-28 2022-02-08 虹软科技股份有限公司 用于显示设备的对齐方法及对齐装置、车载显示***

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20210260998A1 (en) * 2018-08-08 2021-08-26 Bayerische Motoren Werke Aktiengesellschaft Method for Operating a Visual Field Display Device for a Motor Vehicle
WO2020166376A1 (ja) * 2019-02-15 2020-08-20 ソニー株式会社 画像処理装置、画像処理方法、及び、プログラム
CN110148224A (zh) * 2019-04-04 2019-08-20 精电(河源)显示技术有限公司 Hud图像显示方法、装置及终端设备
CN110298924A (zh) * 2019-05-13 2019-10-01 西安电子科技大学 一种ar***中用于显示检测信息的坐标变换方法
CN113129224A (zh) * 2019-12-31 2021-07-16 精工爱普生株式会社 显示***、电子设备、移动体和显示方法
CN113240592A (zh) * 2021-04-14 2021-08-10 重庆利龙科技产业(集团)有限公司 基于ar-hud动态眼位下计算虚像平面的畸变矫正方法
CN114022565A (zh) * 2021-10-28 2022-02-08 虹软科技股份有限公司 用于显示设备的对齐方法及对齐装置、车载显示***

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116295021A (zh) * 2023-05-25 2023-06-23 齐鲁工业大学(山东省科学院) 一种单目线结构光***中相机与激光器位置关系计算方法
CN117456012A (zh) * 2023-12-20 2024-01-26 绘见科技(深圳)有限公司 虚拟相机的视场角标定方法及装置、设备及存储介质
CN117456012B (zh) * 2023-12-20 2024-03-26 绘见科技(深圳)有限公司 虚拟相机的视场角标定方法及装置、设备及存储介质

Also Published As

Publication number Publication date
CN114022565A (zh) 2022-02-08

Similar Documents

Publication Publication Date Title
WO2023071834A1 (zh) 用于显示设备的对齐方法及对齐装置、车载显示***
CN109688392B (zh) Ar-hud光学投影***及映射关系标定方法和畸变矫正方法
CN107554425B (zh) 一种增强现实车载平视显示器ar-hud
US8994558B2 (en) Automotive augmented reality head-up display apparatus and method
CN103050028B (zh) 具有立体视觉随动功能的驾驶模拟器
EP4339938A1 (en) Projection method and apparatus, and vehicle and ar-hud
CN113421346B (zh) 一种增强驾驶感的ar-hud抬头显示界面的设计方法
CN109649275A (zh) 一种基于增强现实的驾驶辅助***及方法
CN108171673A (zh) 图像处理方法、装置、车载抬头显示***及车辆
CN104536579A (zh) 交互式三维实景与数字图像高速融合处理***及处理方法
KR20120067854A (ko) 차량용 증강현실 디스플레이 시스템 및 차량용 증강현실 디스플레이 방법
JPH1165431A (ja) 景観ラベル付カーナビゲーション装置およびシステム
KR101583663B1 (ko) 차량용 카메라의 교정판 제공 방법
CN112242009A (zh) 显示效果融合方法、***、存储介质及主控单元
KR20160046480A (ko) 차량 외부 영상 출력 제어 장치 및 방법
CN102200445B (zh) 实时扩增实境装置及其实时扩增实境方法
CN112703527A (zh) 平视显示器（hud）内容控制***和方法
CN115525152A (zh) 图像处理方法及***、装置、电子设备和存储介质
CN115984122A (zh) 一种hud背光显示***及方法
JP2019053603A (ja) 表示制御プログラム、装置、及び方法
CN109859270A (zh) 一种人眼三维坐标定位方法及分离式双目摄像装置
CN115223231A (zh) 视线方向检测方法和装置
CN111263133B (zh) 信息处理方法及***
Li et al. Distortion correction algorithm of ar-hud virtual image based on neural network model of spatial continuous mapping
CN209290277U (zh) 辅助驾驶***

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22885712

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 2022885712

Country of ref document: EP

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2022885712

Country of ref document: EP

Effective date: 20240528