CN116095298A - Vehicle exterior scene display method, head-mounted display device, terminal device and storage medium - Google Patents

Vehicle exterior scene display method, head-mounted display device, terminal device and storage medium Download PDF

Info

Publication number
CN116095298A
CN116095298A CN202111310750.5A CN202111310750A CN116095298A CN 116095298 A CN116095298 A CN 116095298A CN 202111310750 A CN202111310750 A CN 202111310750A CN 116095298 A CN116095298 A CN 116095298A
Authority
CN
China
Prior art keywords
vehicle
image
display
panoramic image
head
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202111310750.5A
Other languages
Chinese (zh)
Inventor
罗伟
韦安阳
唐辉
王国锋
徐世宏
王颖杰
廖慧红
尹道志
顾鹏云
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Zhejiang Geely Holding Group Co Ltd
Geely Automobile Research Institute Ningbo Co Ltd
Original Assignee
Zhejiang Geely Holding Group Co Ltd
Geely Automobile Research Institute Ningbo Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Zhejiang Geely Holding Group Co Ltd, Geely Automobile Research Institute Ningbo Co Ltd filed Critical Zhejiang Geely Holding Group Co Ltd
Priority to CN202111310750.5A priority Critical patent/CN116095298A/en
Publication of CN116095298A publication Critical patent/CN116095298A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/30Image reproducers
    • H04N13/332Displays for viewing with the aid of special glasses or head-mounted displays [HMD]
    • H04N13/344Displays for viewing with the aid of special glasses or head-mounted displays [HMD] with head-mounted left-right displays
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/106Processing image signals
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/106Processing image signals
    • H04N13/156Mixing image signals
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20Image signal generators
    • H04N13/204Image signal generators using stereoscopic image cameras
    • H04N13/243Image signal generators using stereoscopic image cameras using three or more 2D image sensors
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/30Image reproducers
    • H04N13/366Image reproducers using viewer tracking
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00Television systems
    • H04N7/18Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
    • H04N7/181Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast for receiving images from a plurality of remote sources

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Controls And Circuits For Display Device (AREA)

Abstract

The invention discloses a vehicle exterior scene display method, a head-mounted display device, a terminal device and a storage medium, wherein the method comprises the following steps: receiving image display information sent by a terminal device, and acquiring an external panoramic image corresponding to a target vehicle according to the image display information; acquiring the head gesture of a riding user corresponding to the target vehicle, and determining the observation angle of the riding user according to the head gesture; and determining a display picture of the head-mounted display device according to the observation angle and the panoramic image outside the vehicle, and updating the display content of the display device according to the display picture. The invention realizes that the user can observe the environment picture outside the car at multiple angles in the car, and expands the visual field of the user.

Description

Vehicle exterior scene display method, head-mounted display device, terminal device and storage medium
Technical Field
The present invention relates to the field of image processing technologies, and in particular, to a vehicle exterior scene display method, a head-mounted display device, a terminal device, and a storage medium.
Background
At present, when a driver drives a vehicle, a vehicle body can shield a driving view, so that a view blind area appears in the driving view of the driver, the driver cannot exactly observe scene conditions outside the vehicle, traffic accidents are easy to cause, and great potential safety hazards are brought to the driver and other people.
Disclosure of Invention
The embodiment of the application aims to solve the technical problem that a vehicle body of a vehicle shields a driving visual field of a driver and traffic accidents are easy to cause by providing a vehicle exterior scene display method, a head-mounted display device, a terminal device and a storage medium.
The embodiment of the application provides an out-of-car scene display method, which is applied to head-mounted display equipment and comprises the following steps:
receiving image display information sent by terminal equipment, and acquiring an external panoramic image corresponding to a target vehicle according to the image display information;
acquiring the head gesture of a riding user corresponding to the target vehicle, and determining the observation angle of the riding user according to the head gesture;
and determining a display picture of the head-mounted display device according to the observation angle and the panoramic image outside the vehicle, and updating the display content of the head-mounted display device according to the display picture.
In an embodiment, the step of obtaining the panoramic image of the outside of the vehicle corresponding to the target vehicle according to the image display information includes:
when the image display information does not include an outside panoramic image, determining an outside field Jing Tuxiang according to the image information, and generating the outside panoramic image according to the outside scene image;
And when the image display information comprises an outside panoramic image, acquiring the outside panoramic image corresponding to the display information.
In an embodiment, the step of determining the display screen of the head-mounted display device according to the viewing angle and the panoramic image outside the vehicle comprises:
acquiring a shooting angle corresponding to the observation angle;
and acquiring a display picture corresponding to the shooting angle from the panoramic image outside the vehicle.
In an embodiment, the method for displaying an external scene of a vehicle further includes:
receiving a mixed panoramic image sent by the terminal equipment, wherein the mixed panoramic image comprises an out-of-vehicle panoramic image and a target vehicle image;
when a display instruction of the target vehicle is received, updating the display content of the head-mounted display device according to a mixed display picture corresponding to the observation angle in the mixed panoramic image;
and when receiving a hiding instruction of the target vehicle, updating the display content of the head-mounted display device according to a display picture corresponding to the observation angle in the mixed panoramic image.
The embodiment of the application provides an out-of-car scene display method, which is applied to terminal equipment and comprises the following steps:
Acquiring a vehicle external field Jing Tuxiang corresponding to a target vehicle shot by each camera according to different shooting angles;
synthesizing the exterior scene images corresponding to the shooting angles to obtain exterior panoramic images, and taking the exterior panoramic images as image display information;
or taking the scene images outside the vehicle corresponding to the shooting angles as the image display information;
and sending the image display information to head-mounted display equipment so that the head-mounted display equipment can acquire the panoramic image outside the vehicle according to the image display information and update display content according to the panoramic image outside the vehicle.
In an embodiment, the step of synthesizing the exterior scene images corresponding to the shooting angles to obtain the exterior panoramic image includes:
acquiring the same characteristic points of the scene images outside the vehicle at each shooting angle;
splicing the exterior scene images corresponding to the shooting angles according to the same characteristic points to generate spliced images;
processing the spliced images according to a preset processing mode to obtain the panoramic image outside the vehicle; the preset processing mode at least comprises image correction, image equalization compensation and image frequency band fusion.
In an embodiment, the method for displaying an external scene of a vehicle further includes:
the method for displaying the scene outside the vehicle further comprises the following steps:
acquiring a three-dimensional model of the target vehicle;
generating a hybrid panoramic image according to the three-dimensional model and the out-of-vehicle panoramic image;
and sending the mixed panoramic image to a head-mounted display device so that the head-mounted display device can update the display content according to the mixed panoramic image.
In addition, to achieve the above object, the present invention also provides a head-mounted display device including: the method comprises the steps of a memory, a processor and an off-vehicle scene display program which is stored in the memory and can run on the processor, wherein the off-vehicle scene display program is executed by the processor to realize the off-vehicle scene display method.
In addition, to achieve the above object, the present invention also provides a terminal device including: the method comprises the steps of a memory, a processor and an off-vehicle scene display program which is stored in the memory and can run on the processor, wherein the off-vehicle scene display program is executed by the processor to realize the off-vehicle scene display method.
In addition, in order to achieve the above object, the present invention also provides a storage medium having stored thereon an outside scene display program which, when executed by a processor, implements the steps of the above outside scene display method.
The technical scheme of the external scene display method, the head-mounted display device, the terminal device and the storage medium provided in the embodiment of the application has at least the following technical effects or advantages:
the technical scheme that the image display information sent by the terminal equipment is adopted, the external panoramic image corresponding to the target vehicle is obtained according to the image display information, the head gesture of the riding user corresponding to the target vehicle is obtained, the observation angle of the riding user is determined according to the head gesture, the display picture of the head-mounted display equipment is determined according to the observation angle and the external panoramic image, and the display content of the display equipment is updated according to the display picture is adopted, so that the technical problems that the driving vision of a driver is blocked by the vehicle body of the vehicle, traffic accidents are easy to cause are solved, the clear understanding of the traffic dynamic conditions in all directions outside the vehicle can be assisted, the visual field of the driver is expanded, the occurrence of traffic accidents is avoided, convenience is brought to the driver, and the driving safety of the driver is facilitated to be ensured.
Drawings
FIG. 1 is a flow chart of a first embodiment of an out-of-vehicle scene display method according to the present invention;
FIG. 2 is a flow chart of a second embodiment of the method for displaying an external scene of a vehicle according to the present invention;
FIG. 3 is a flowchart of a third embodiment of an out-of-vehicle scene display method according to the present invention;
FIG. 4 is a flowchart of a fourth embodiment of an out-of-vehicle scene display method according to the present invention;
FIG. 5 is a flowchart of a fifth embodiment of an out-of-vehicle scene display method according to the present invention;
fig. 6 is a schematic structural diagram of a hardware running environment according to an embodiment of the present invention.
Detailed Description
In order that the above-described aspects may be better understood, exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.
The embodiments of the present invention provide embodiments of an off-vehicle scene display method, and it should be noted that although a logic sequence is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown or described herein.
As shown in fig. 1, in a first embodiment of the present application, an off-vehicle scene display method of the present application is applied to a head-mounted display device, and includes the steps of:
Step S210: and receiving image display information sent by the terminal equipment, and acquiring an external panoramic image corresponding to the target vehicle according to the image display information.
In this embodiment, the head-mounted display device refers to a display device that is applied to MR technology (Mixed Reality technology), and that can be worn on the eyes, such as MR glasses. The image display information can be an exterior scene image corresponding to the target vehicle or an exterior panoramic image synthesized by the exterior scene images, wherein the exterior panoramic image is an exterior 360-degree panoramic image, and images of various angles outside the vehicle can be seen through the exterior panoramic image.
A plurality of cameras are mounted on the vehicle, and the cameras are used for shooting the external environment image of the vehicle, wherein the cameras are mounted on the vehicle for shooting the external environment image of the vehicle and can be regarded as target vehicles. All cameras are mounted on the target vehicle in a scattered manner in a surrounding manner according to the outline of the target vehicle, each camera corresponds to one shooting angle, and different cameras shoot the scene images outside the vehicle corresponding to the target vehicle according to different shooting angles. The terminal device is a device that processes an image, such as a server, a control device on the target vehicle side, or the like. And uploading the external scene images shot by the cameras to the terminal equipment, synthesizing the external scene images corresponding to the shooting angles by the terminal equipment according to the external scene images to obtain an external panoramic image, taking the obtained external panoramic image as image display information, namely the image display information is the external panoramic image, and then sending the image display information to the head-mounted display equipment. Or the terminal device does not synthesize the exterior scene images corresponding to the shooting angles, but takes the exterior scene images corresponding to the shooting angles as image display information, wherein the image display information comprises exterior fields Jing Tuxiang corresponding to the shooting angles, namely the image display information comprises a plurality of exterior scene images corresponding to the shooting angles which are shot simultaneously, and then the image display information is sent to the head-mounted display device. And after the head-mounted display equipment receives the image display information sent by the terminal equipment, acquiring the panoramic image outside the vehicle.
The image display information may be an external panoramic image, or may include external scene images corresponding to each shooting angle. After receiving the image display information, the head-mounted display device needs to determine whether the image display information includes an outside panoramic image. Namely, the steps of acquiring the panoramic image outside the vehicle corresponding to the target vehicle according to the image display information are as follows:
judging whether the image display information comprises an outside panoramic image;
when the image display information does not include an outside panoramic image, determining an outside field Jing Tuxiang according to the image information, and generating the outside panoramic image according to the outside scene image;
and when the image display information comprises an outside panoramic image, acquiring the outside panoramic image corresponding to the display information.
Specifically, whether the image display information comprises an exterior panoramic image is judged, if not, exterior scene images corresponding to all shooting angles included in the image display information are obtained, and then the exterior scene images corresponding to all shooting angles are synthesized to generate the exterior panoramic image. The same feature points of the exterior scene images of each shooting angle need to be acquired when the exterior panoramic images are generated, wherein the same feature points refer to the same parts between every two exterior scene images, for example, the camera 1 and the camera 2 are adjacent cameras, the exterior scene images shot by the camera 1 and the camera 2 have the same tree, and the tree is the same feature point of the exterior scene images shot by the camera 1 and the camera 2. After the same feature points are obtained, the exterior scene images of all shooting angles are spliced according to the same feature points to generate spliced images, the generated spliced images can be theoretically used as exterior panoramic images, but in consideration of actual conditions, as a target vehicle moves, things in the shot scene are static and also move, and the camera can shake and tilt to different degrees, the content in the spliced images possibly becomes wavy, so that the spliced images need to be subjected to image correction, the spliced images after correction can display normal left eye display pictures, and then the corrected spliced images are subjected to image equalization compensation and image frequency band fusion processing, so that the exterior panoramic images are obtained. The image equalization compensation and the image frequency band fusion processing of the corrected spliced image comprise the steps of adjusting illumination and tone of the corrected spliced image, removing joint edges, vignetting and parallax among the images, and beautifying the panoramic image outside the vehicle.
Step S220: and acquiring the head gesture of the riding user corresponding to the target vehicle, and determining the observation angle of the riding user according to the head gesture.
In this embodiment, the riding user may be a driver driving the target vehicle, or a person riding the target vehicle, and after wearing the head-mounted display device, the riding user may see a stereoscopic (3D) real environment picture at all angles outside the vehicle without shielding in the vehicle, as if the blocking of the vehicle body is transmitted. Specifically, the head-mounted display device acquires the panoramic image outside the vehicle according to the image display information, and simultaneously acquires the head gesture of the user taking the vehicle, wherein the head gesture can be represented by the three-dimensional coordinates of the head-mounted display device. For example, after the head-mounted display device is worn by the riding user, when the riding user sits down according to a correct sitting posture (the riding user faces the sitting posture right in front of the target vehicle), the head-mounted display device takes the three-dimensional coordinate at the moment as a reference coordinate, the reference coordinate can be understood as an origin coordinate, when the head of the riding user rotates in different directions such as leftwards, rightwards and backwards, the head posture of the riding user changes, that is, the equipment posture of the head-mounted display device also changes synchronously, that is, the three-dimensional coordinate of the head-mounted display device also changes, the head-mounted display device can acquire the three-dimensional coordinate of the head-mounted display device in real time, then the rotation angle of the head-mounted display device can be calculated according to the reference coordinate and the three-dimensional coordinate acquired in real time, and after the rotation angle is acquired, the observation angle of the riding user is also acquired, that is the head-mounted display device rotation angle can be used as the observation angle of the riding user. Wherein the rotation angle of the head-mounted display device, that is, the rotation angle of the head of the riding user.
Step S230: and determining the display picture of the head-mounted display device according to the observation angle and the panoramic image outside the vehicle.
Step S240: and updating the display content of the head-mounted display device according to the display picture.
In this embodiment, after determining the viewing angle of the riding user, the viewing angle ranges of the left lens and the right lens in the head-mounted display device are obtained, and the display screen is obtained from the panoramic image outside the vehicle according to the viewing angle and the viewing angle ranges of the left lens and the right lens. When the stereoscopic left-eye display screen is displayed by the head-mounted display device, the head-mounted display device needs to display the left-eye and right-eye screens, and the images of the left eye and the right eye are continuously and alternately displayed on the screen, so that the real stereoscopic 3D screen can be seen by adding the physiological characteristic of human eye persistence. Therefore, the display frames comprise a display frame corresponding to the left eye of the riding user and a display frame corresponding to the right eye, the application is called a left eye display frame and a right eye display frame, after the left eye display frame and the right eye display frame are obtained, the display frame and the right eye display frame are adopted to update the display content of the head-mounted display device, the display content is the original display of the external environment content of the head-mounted display device, the display content comprises the original left eye display frame and the original right eye display frame, and the riding user can see the real-time external environment content after the display content is displayed.
For example, the riding user is a driver, the driver wears the head-mounted display device to see the window direction of the co-driver, the window direction of the co-driver is the observation angle of the driver, after the head-mounted display device acquires the left eye display screen and the right eye display screen corresponding to the window direction of the co-driver from the panoramic image outside the vehicle, the head-mounted display device continuously and alternately displays the left eye display screen and the right eye display screen corresponding to the window direction of the co-driver on the left eye screen and the right eye screen after updating the original left eye display screen and the original right eye display screen by adopting the acquired left eye display screen and right eye display screen corresponding to the left eye display screen and the right eye display screen, and the driver can see the real-time stereoscopic screen outside the window of the co-driver, and the stereoscopic screen is the screen in the real-time environment outside the window of the co-driver. Therefore, the driver or the passengers can clearly see the real environment picture outside the vehicle without getting off, the driver can know the traffic condition outside the vehicle without restraint during driving, the occurrence of traffic accidents is avoided, and convenience is brought to the driving of the driver.
According to the technical scheme, the driver can be assisted to clearly know the traffic dynamic conditions in all directions outside the vehicle, the visual field range of the driver is expanded, traffic accidents are avoided, convenience is brought to driving of the driver, and driving safety of the driver is guaranteed.
As shown in fig. 2, in the second embodiment of the present application, step S230 includes the steps of:
step S231: and acquiring a shooting angle corresponding to the observation angle.
Step S232: and acquiring a display picture corresponding to the shooting angle from the panoramic image outside the vehicle.
In this embodiment, since each camera is fixedly mounted at a different position on the target vehicle, a target vehicle space coordinate system can be established around the target vehicle, then three-dimensional coordinates of each camera in the target vehicle space coordinate system are measured, and the shooting angles of each camera are determined by taking the three-dimensional coordinates of the preset camera as a reference. After the three-dimensional coordinates of the head-mounted display device are obtained, the observation angle of the riding user can be obtained, and then the observation angle of the riding user is judged to fall into the ranges of the shooting angles of the cameras. For example, if the observation angle of the riding user falls between the photographing angles corresponding to the cameras 3 and 4, then the photographing angle corresponding to the observation angle of the riding user is a range value; the observation angle of the riding user is equal to the shooting angle of the camera 3, and the shooting angle corresponding to the observation angle of the riding user is a fixed value, so that a corresponding display picture is obtained from the panoramic image outside the automobile according to the shooting angle corresponding to the observation angle of the riding user. For example, if the viewing angle of the riding user is equal to the photographing angle of the camera 3, the external field Jing Tuxiang photographed by the camera 3 is obtained from the external panoramic image, and the external scene image photographed by the camera 3 can be understood as a display screen, which refers to a left-eye display screen and a right-eye display screen.
As shown in fig. 3, in a third embodiment of the present application, the method for displaying an external scene of a vehicle of the present application further includes the following steps:
step 250: and receiving the mixed panoramic image sent by the terminal equipment.
In this embodiment, in order to enable a riding user to timely know the condition of a target vehicle without getting off and assist a driver in backing, the terminal device further fuses the target vehicle into an out-of-vehicle panoramic image to generate a hybrid panoramic image, and sends the hybrid panoramic image to the head-mounted display device, so that the riding user can clearly see not only an out-of-vehicle real environment picture, but also a stereoscopic external vehicle body picture of the target vehicle, such as the direction of a vehicle body, the steering direction of wheels of the target vehicle, and the like. Specifically, a three-dimensional model of the target vehicle is stored in the terminal device in advance, after an out-of-vehicle panoramic image is generated, the three-dimensional model of the target vehicle is obtained, and then the three-dimensional model and the out-of-vehicle panoramic image are fused to obtain a mixed panoramic image, namely the mixed panoramic image not only comprises the out-of-vehicle panoramic image but also comprises the target vehicle image. After the panoramic image outside the vehicle is obtained, an environment space coordinate system of the panoramic image outside the vehicle is established, and the real coordinate of the target vehicle is obtained by positioning the real target vehicle in the environment space coordinate system. Because the ratio of the size of the three-dimensional model to the size of the real target vehicle is fixed, the real coordinates are reduced according to the ratio of the size of the three-dimensional model to the size of the real target vehicle, the coordinates of the three-dimensional model in an environment space coordinate system are obtained, the three-dimensional model and the vehicle exterior panoramic image are overlapped according to the coordinates of the three-dimensional model in the environment space coordinate system, and the mixed panoramic image is obtained, wherein the visible mixed panoramic image comprises the vehicle exterior panoramic image and the target vehicle image. The terminal device then transmits the mixed panoramic image to a head-mounted display device, which can simultaneously display a real environment screen according to the mixed panoramic image, and can also display an external body screen of the target vehicle, for example, the orientation of the body.
Step 260: and when a display instruction of the target vehicle is received, updating the display content of the head-mounted display device according to a mixed display picture corresponding to the observation angle in the mixed panoramic image.
Step 270: and when receiving a hiding instruction of the target vehicle, updating the display content of the head-mounted display device according to a display picture corresponding to the observation angle in the mixed panoramic image.
In this embodiment, the head-mounted display device has a display function and a hiding function of the target vehicle, where the display function of the target vehicle means that a picture in a real environment and an external body picture of the target vehicle can be displayed at the same time, the hiding function of the target vehicle means that only the real environment picture is displayed and no external body picture of the target vehicle is displayed, and an implementation manner of displaying the real environment picture is the same as that of the first embodiment, and is not repeated here.
Specifically, the instruction for triggering the display function and the hiding function of the target vehicle is preset, and the display instruction and the hiding instruction are respectively triggered by means of voice, gestures, keys and the like. For example, when the riding user outputs a voice for turning on the display function of the target vehicle, the viewing angle of the riding user is obtained, then a shooting angle corresponding to the viewing angle is obtained, and further, according to the shooting angle corresponding to the viewing angle, a mixed display picture corresponding to the viewing angle is obtained from the mixed panoramic image, wherein the mixed display picture comprises a display picture and the target vehicle picture, and the mixed display picture refers to a left-eye mixed display picture and a right-eye mixed display picture. Then, the display content of the head-mounted display device is updated by adopting the left-eye mixed display picture and the right-eye mixed display picture, and the left-eye mixed display picture and the right-eye mixed display picture are continuously and alternately displayed on a left-eye screen and a right-eye screen of the head-mounted display device, so that a riding user can simultaneously see a three-dimensional real environment picture and a three-dimensional external car body picture of a target car. When the user is a driver, the driver can clearly see the states of the vehicle body and the wheels when the vehicle is reversed after wearing the head-mounted display device, so that the vehicle is convenient to reverse.
For another example, if the riding user outputs a voice for turning on the hiding function of the target vehicle, the three-dimensional model in the hybrid panoramic image is hidden, and the hybrid panoramic image after hiding the three-dimensional model can be understood as an outside panoramic image. And then the observation angle of the riding user is obtained, then the shooting angle corresponding to the observation angle is obtained, and further the display picture corresponding to the observation angle is obtained from the mixed panoramic image according to the shooting angle corresponding to the observation angle, wherein the display picture refers to a left eye display picture and a right eye display picture. The left eye display picture and the right eye display picture are continuously and alternately displayed on the left eye screen and the right eye screen of the head-mounted display device, and a riding user can simultaneously see the stereoscopic reality environment picture.
According to the technical scheme, the three-dimensional real environment picture can be seen by a riding user under the condition of not getting off, and the three-dimensional external car body picture of the target car can be seen, so that driving and reversing are facilitated.
As shown in fig. 4, in a fourth embodiment of the present application, an off-vehicle scene display method of the present application is applied to a terminal device, and the off-vehicle scene display method includes:
Step 310: and acquiring the scene images of the outside of the vehicle corresponding to the target vehicles shot by each camera according to different shooting angles.
In this embodiment, a plurality of cameras are mounted on the vehicle, and the cameras can be regarded as target vehicles by the vehicle for capturing the external environment image of the vehicle. All cameras are mounted on the target vehicle in a scattered manner in a surrounding manner according to the outline of the target vehicle, each camera corresponds to one shooting angle, and different cameras shoot the scene images outside the vehicle corresponding to the target vehicle according to different shooting angles. The terminal device is a device for processing an image, and may or may not be provided on the target vehicle, and the terminal device may be a server, a control device on the target vehicle side, or the like. And uploading the external scene images shot by the cameras to the terminal equipment by the cameras, and obtaining the external scene images uploaded by the cameras by the terminal equipment.
Step 320: synthesizing the exterior scene images corresponding to the shooting angles to obtain exterior panoramic images, and taking the exterior panoramic images as image display information; or taking the scene images outside the vehicle corresponding to the shooting angles as the image display information.
Step 330: and sending the image display information to head-mounted display equipment so that the head-mounted display equipment can acquire the panoramic image outside the vehicle according to the image display information and update display content according to the panoramic image outside the vehicle.
The terminal equipment synthesizes the exterior scene images corresponding to the shooting angles to obtain an exterior panoramic image, takes the obtained exterior panoramic image as image display information, namely the image display information is the exterior panoramic image, and then sends the image display information to the head-mounted display equipment. Or the terminal device does not synthesize the exterior scene images corresponding to the shooting angles, but takes the exterior scene images corresponding to the shooting angles as image display information, wherein the image display information comprises exterior fields Jing Tuxiang corresponding to the shooting angles, namely the image display information comprises a plurality of exterior scene images corresponding to the shooting angles which are shot simultaneously, and then the image display information is sent to the head-mounted display device.
The step of synthesizing the exterior scene images corresponding to the shooting angles by the terminal equipment to obtain the exterior panoramic image comprises the following steps:
Acquiring the same characteristic points of the scene images outside the vehicle at each shooting angle;
splicing the exterior scene images corresponding to the shooting angles according to the same characteristic points to generate spliced images;
processing the spliced images according to a preset processing mode to obtain the panoramic image outside the vehicle; the preset processing mode at least comprises image correction, image equalization compensation and image frequency band fusion.
When the terminal equipment synthesizes the panoramic images outside the vehicle, the same characteristic points of the scene images outside the vehicle at all shooting angles are required to be obtained, wherein the same characteristic points refer to the same parts between every two scene images outside the vehicle, for example, the camera 1 and the camera 2 are mutually adjacent cameras, the scene images outside the vehicle shot by the camera 1 and the camera 2 have the same tree, and the tree is the same characteristic point of the scene images outside the vehicle shot by the camera 1 and the camera 2. After the same feature points are obtained, the exterior scene images of all shooting angles are spliced according to the same feature points to generate spliced images, the generated spliced images can be theoretically used as exterior panoramic images, but in consideration of actual conditions, as a target vehicle moves, things in the shot scene are static and also move, and the camera can shake and tilt to different degrees, the content in the spliced images possibly becomes wavy, so that the spliced images need to be subjected to image correction, the spliced images after correction can display normal left eye display pictures, and then the corrected spliced images are subjected to image equalization compensation and image frequency band fusion processing, so that the exterior panoramic images are obtained. The image equalization compensation and the image frequency band fusion processing of the corrected spliced image comprise the steps of adjusting illumination and tone of the corrected spliced image, removing joint edges, vignetting and parallax among the images, and beautifying the panoramic image outside the vehicle.
Further, after receiving the image display information sent by the terminal device, the head-mounted display device acquires an outside panoramic image, acquires the outside panoramic image according to the image display information, and acquires a display picture corresponding to the observation angle of the riding user from the outside panoramic image. After receiving the image display information, the head-mounted display device needs to judge whether the image display information comprises an outside panoramic image, if so, the image display information is the outside panoramic image, and the outside panoramic image can be directly obtained; if the panoramic image is not included, acquiring the exterior scene images corresponding to the shooting angles included in the image display information, and then synthesizing the exterior panoramic image according to the exterior scene images corresponding to the shooting angles, wherein the process of synthesizing the exterior panoramic image is the same as that of synthesizing the exterior panoramic image by the terminal equipment, and the description is omitted.
Further, the head-mounted display device updates display content of the head-mounted display device by using the display picture corresponding to the acquired observation angle, then displays the updated display content, and a riding user can clearly see the stereoscopic scene picture outside the vehicle corresponding to the observation angle through the updated display content, so that the riding user can observe the scene outside the vehicle without getting off the vehicle, and the driving and the viewing of the scenery outside the vehicle in the vehicle are facilitated.
As shown in fig. 5, in a fifth embodiment of the present application, the method for displaying an external scene of a vehicle of the present application further includes the steps of:
step 340: and acquiring a three-dimensional model of the target vehicle.
Step 350: and generating a mixed panoramic image according to the three-dimensional model and the panoramic image outside the vehicle.
In this embodiment, a three-dimensional model of a target vehicle is stored in a terminal device in advance, after an external panoramic image is generated, the three-dimensional model of the target vehicle is obtained, and then the three-dimensional model and the external panoramic image are fused to obtain a hybrid panoramic image, that is, the hybrid panoramic image not only includes the external panoramic image but also includes the target vehicle image. After the panoramic image outside the vehicle is obtained, an environment space coordinate system of the panoramic image outside the vehicle is established, and the real target vehicle is positioned in the environment space coordinate system to obtain the real coordinate of the target vehicle. Because the ratio of the three-dimensional model of the target vehicle to the size of the target vehicle in reality is fixed, the real coordinates are reduced according to the ratio of the three-dimensional model to the size of the target vehicle in reality, the coordinates of the three-dimensional model in the environment space coordinate system are obtained, the three-dimensional model and the panoramic image outside the vehicle are overlapped according to the coordinates of the three-dimensional model in the environment space coordinate system, the mixed panoramic image is obtained, and then the mixed panoramic image is sent to the head-mounted display device, so that the head-mounted display device can display the real environment image and the external body image of the target vehicle, such as the direction of the body, according to the mixed panoramic image.
Step 360: and sending the mixed panoramic image to a head-mounted display device so that the head-mounted display device can update the display content according to the mixed panoramic image.
The head-mounted display device has a display function and a hiding function of a target vehicle, a three-dimensional model in the mixed panoramic image can be hidden through the hiding function, the mixed panoramic image after hiding can be understood as an out-of-vehicle panoramic image, and a user can see a three-dimensional real environment picture through the head-mounted display device. The display function can display the three-dimensional model in the mixed panoramic image, and the riding user can simultaneously see the stereoscopic real environment picture and the stereoscopic external car body picture of the target car through the head-mounted display device. The head-mounted display device is provided with instructions for triggering the display function and the hiding function of the target vehicle, wherein the instructions are respectively a display instruction and a hiding instruction, and the display instruction and the hiding instruction can be triggered by means of voice, gestures, keys and the like. After the display instruction is triggered, the head-mounted display device acquires the observation angle of the riding user, then acquires the shooting angle corresponding to the observation angle, further acquires the mixed display picture corresponding to the observation angle from the mixed panoramic image according to the shooting angle corresponding to the observation angle, updates the display content of the head-mounted display device by adopting the acquired mixed display picture, and displays the updated display content, so that the riding user can see the stereoscopic real environment picture and the stereoscopic external car body picture of the target car in the car at the same time. After the hidden command is triggered, the head-mounted display device acquires the observation angle of the riding user, then acquires the shooting angle corresponding to the observation angle, further acquires the display picture corresponding to the observation angle from the mixed panoramic image according to the shooting angle corresponding to the observation angle, then updates the display content of the head-mounted display device by adopting the acquired display picture, and displays the updated display content, so that the riding user can see the stereoscopic real environment picture in the car, but can not see the stereoscopic external car body picture of the target car.
According to the technical scheme, the three-dimensional real environment picture can be seen by a riding user under the condition of not getting off, and the three-dimensional external car body picture of the target car can be seen, so that driving and reversing are facilitated.
As shown in fig. 6, fig. 6 is a schematic structural diagram of a hardware running environment according to an embodiment of the present invention.
It should be noted that fig. 6 may be a schematic structural diagram of a hardware running environment of the head-mounted display device or the terminal device.
As an implementation manner, as shown in fig. 6, an embodiment of the present invention relates to a head-mounted display device, where the head-mounted display device includes: a processor 1001, such as a CPU, memory 1002, a communications bus 1003. Wherein the communication bus 1003 is used to enable connectivity communications between these components.
The memory 1002 may be a high-speed RAX memory or a stable memory (non-volatile Xerory), such as a disk memory. As shown in fig. 6, an off-vehicle scene display program may be included in a memory 1002 as one storage medium; and the processor 1001 may be configured to call an off-vehicle scene display program stored in the memory 1002 and perform the following operations:
Receiving image display information sent by terminal equipment, and acquiring an external panoramic image corresponding to a target vehicle according to the image display information;
acquiring the head gesture of a riding user corresponding to the target vehicle, and determining the observation angle of the riding user according to the head gesture;
and determining a display picture of the head-mounted display device according to the observation angle and the panoramic image outside the vehicle, and updating the display content of the head-mounted display device according to the display picture.
Further, the processor 1001 may be configured to call an off-vehicle scene display program stored in the memory 1002, and perform the following operations:
when the image display information does not include an outside panoramic image, determining an outside field Jing Tuxiang according to the image information, and generating the outside panoramic image according to the outside scene image;
and when the image display information comprises an outside panoramic image, acquiring the outside panoramic image corresponding to the display information.
Further, the processor 1001 may be configured to call an off-vehicle scene display program stored in the memory 1002, and perform the following operations:
acquiring a shooting angle corresponding to the observation angle;
And acquiring a display picture corresponding to the shooting angle from the panoramic image outside the vehicle.
Further, the processor 1001 may be configured to call an off-vehicle scene display program stored in the memory 1002, and perform the following operations:
receiving a mixed panoramic image sent by the terminal equipment, wherein the mixed panoramic image comprises an out-of-vehicle panoramic image and a target vehicle image;
when a display instruction of the target vehicle is received, updating the display content of the head-mounted display device according to a mixed display picture corresponding to the observation angle in the mixed panoramic image;
and when receiving a hiding instruction of the target vehicle, updating the display content of the head-mounted display device according to a display picture corresponding to the observation angle in the mixed panoramic image.
Further, as another implementation manner, as shown in fig. 6, an embodiment of the present invention may relate to a terminal device, where the terminal device includes: a processor 1001, such as a CPU, memory 1002, a communications bus 1003. Wherein the communication bus 1003 is used to enable connectivity communications between these components.
The memory 1002 may be a high-speed RAX memory or a stable memory (non-volatile Xerory), such as a disk memory. As shown in fig. 6, an off-vehicle scene display program may be included in a memory 1002 as one storage medium; and the processor 1001 may be configured to call an off-vehicle scene display program stored in the memory 1002 and perform the following operations:
Acquiring a vehicle external field Jing Tuxiang corresponding to a target vehicle shot by each camera according to different shooting angles;
synthesizing the exterior scene images corresponding to the shooting angles to obtain exterior panoramic images, and taking the exterior panoramic images as image display information;
or taking the scene images outside the vehicle corresponding to the shooting angles as the image display information;
and sending the image display information to head-mounted display equipment so that the head-mounted display equipment can acquire the panoramic image outside the vehicle according to the image display information and update display content according to the panoramic image outside the vehicle.
Further, the processor 1001 may be configured to call an off-vehicle scene display program stored in the memory 1002, and perform the following operations:
acquiring the same characteristic points of the scene images outside the vehicle at each shooting angle;
splicing the exterior scene images corresponding to the shooting angles according to the same characteristic points to generate spliced images;
processing the spliced images according to a preset processing mode to obtain the panoramic image outside the vehicle; the preset processing mode at least comprises image correction, image equalization compensation and image frequency band fusion.
Further, the processor 1001 may be configured to call an off-vehicle scene display program stored in the memory 1002, and perform the following operations:
acquiring a three-dimensional model of the target vehicle;
generating a hybrid panoramic image according to the three-dimensional model and the out-of-vehicle panoramic image;
and sending the mixed panoramic image to a head-mounted display device so that the head-mounted display device can update the display content according to the mixed panoramic image.
Further, the present invention also provides a head-mounted display device including: the method comprises the steps of a memory, a processor and an off-vehicle scene display program which is stored in the memory and can run on the processor, wherein the off-vehicle scene display program is executed by the processor to realize the off-vehicle scene display method.
Further, the present invention also provides a terminal device including: the method comprises the steps of a memory, a processor and an off-vehicle scene display program which is stored in the memory and can run on the processor, wherein the off-vehicle scene display program is executed by the processor to realize the off-vehicle scene display method.
Further, the present invention also provides a storage medium, on which an external scene display program is stored, and the external scene display program when executed by a processor implements the steps of the external scene display method described above.
It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
It should be noted that in the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention may be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by one and the same item of hardware. The use of the words first, second, third, etc. do not denote any order. These words may be interpreted as names.
While preferred embodiments of the present invention have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. It is therefore intended that the following claims be interpreted as including the preferred embodiments and all such alterations and modifications as fall within the scope of the invention.
It will be apparent to those skilled in the art that various modifications and variations can be made to the present invention without departing from the spirit or scope of the invention. Thus, it is intended that the present invention also include such modifications and alterations insofar as they come within the scope of the appended claims or the equivalents thereof.

Claims (10)

1. An out-of-vehicle scene display method, characterized by being applied to a head-mounted display device, comprising:
receiving image display information sent by terminal equipment, and acquiring an external panoramic image corresponding to a target vehicle according to the image display information;
acquiring the head gesture of a riding user corresponding to the target vehicle, and determining the observation angle of the riding user according to the head gesture;
and determining a display picture of the head-mounted display device according to the observation angle and the panoramic image outside the vehicle, and updating the display content of the head-mounted display device according to the display picture.
2. The method of claim 1, wherein the step of acquiring the outside panorama image corresponding to the target vehicle according to the image display information comprises:
when the image display information does not include an outside panoramic image, determining an outside field Jing Tuxiang according to the image information, and generating the outside panoramic image according to the outside scene image;
and when the image display information comprises an outside panoramic image, acquiring the outside panoramic image corresponding to the display information.
3. The method of claim 1, wherein the step of determining a display of the head mounted display device from the viewing angle and the off-board panoramic image comprises:
acquiring a shooting angle corresponding to the observation angle;
and acquiring a display picture corresponding to the shooting angle from the panoramic image outside the vehicle.
4. The method of claim 1, wherein the off-board scene display method further comprises:
receiving a mixed panoramic image sent by the terminal equipment, wherein the mixed panoramic image comprises an out-of-vehicle panoramic image and a target vehicle image;
when a display instruction of the target vehicle is received, updating the display content of the head-mounted display device according to a mixed display picture corresponding to the observation angle in the mixed panoramic image;
And when receiving a hiding instruction of the target vehicle, updating the display content of the head-mounted display device according to a display picture corresponding to the observation angle in the mixed panoramic image.
5. The method for displaying the scene outside the vehicle is characterized by being applied to terminal equipment, and comprises the following steps:
acquiring a vehicle external field Jing Tuxiang corresponding to a target vehicle shot by each camera according to different shooting angles;
synthesizing the exterior scene images corresponding to the shooting angles to obtain exterior panoramic images, and taking the exterior panoramic images as image display information;
or taking the scene images outside the vehicle corresponding to the shooting angles as the image display information;
and sending the image display information to head-mounted display equipment so that the head-mounted display equipment can acquire the panoramic image outside the vehicle according to the image display information and update display content according to the panoramic image outside the vehicle.
6. The method of claim 5, wherein the step of synthesizing the exterior scene image corresponding to each of the photographing angles to obtain the exterior panoramic image comprises:
acquiring the same characteristic points of the scene images outside the vehicle at each shooting angle;
Splicing the exterior scene images corresponding to the shooting angles according to the same characteristic points to generate spliced images;
processing the spliced images according to a preset processing mode to obtain the panoramic image outside the vehicle; the preset processing mode at least comprises image correction, image equalization compensation and image frequency band fusion.
7. The method of claim 5, wherein the off-board scene display method further comprises:
acquiring a three-dimensional model of the target vehicle;
generating a hybrid panoramic image according to the three-dimensional model and the out-of-vehicle panoramic image;
and sending the mixed panoramic image to a head-mounted display device so that the head-mounted display device can update the display content according to the mixed panoramic image.
8. A head-mounted display device, comprising: a memory, a processor and an off-board scene display program stored on the memory and operable on the processor, which when executed by the processor, implements the steps of the off-board scene display method of any one of claims 1-4.
9. A terminal device, comprising: a memory, a processor and an off-board scene display program stored on the memory and operable on the processor, which when executed by the processor, implements the steps of the off-board scene display method of any one of claims 5-7.
10. A storage medium having stored thereon an off-vehicle scene display program which, when executed by a processor, implements the steps of the off-vehicle scene display method of any of claims 1-7.
CN202111310750.5A 2021-11-06 2021-11-06 Vehicle exterior scene display method, head-mounted display device, terminal device and storage medium Pending CN116095298A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202111310750.5A CN116095298A (en) 2021-11-06 2021-11-06 Vehicle exterior scene display method, head-mounted display device, terminal device and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202111310750.5A CN116095298A (en) 2021-11-06 2021-11-06 Vehicle exterior scene display method, head-mounted display device, terminal device and storage medium

Publications (1)

Publication Number Publication Date
CN116095298A true CN116095298A (en) 2023-05-09

Family

ID=86206873

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202111310750.5A Pending CN116095298A (en) 2021-11-06 2021-11-06 Vehicle exterior scene display method, head-mounted display device, terminal device and storage medium

Country Status (1)

Country Link
CN (1) CN116095298A (en)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2006074105A (en) * 2004-08-31 2006-03-16 Equos Research Co Ltd Image display apparatus for vehicle periphery
CN103692973A (en) * 2012-09-27 2014-04-02 富士通天株式会社 Image generating apparatus, image display system and image generating method
CN106339980A (en) * 2016-08-22 2017-01-18 乐视控股(北京)有限公司 Automobile-based VR display device and method and automobile
CN107009961A (en) * 2017-03-03 2017-08-04 北京汽车股份有限公司 Automobile and automobile transparent cockpit system
CN110341597A (en) * 2018-04-02 2019-10-18 杭州海康威视数字技术股份有限公司 A kind of vehicle-mounted panoramic video display system, method and Vehicle Controller

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2006074105A (en) * 2004-08-31 2006-03-16 Equos Research Co Ltd Image display apparatus for vehicle periphery
CN103692973A (en) * 2012-09-27 2014-04-02 富士通天株式会社 Image generating apparatus, image display system and image generating method
CN106339980A (en) * 2016-08-22 2017-01-18 乐视控股(北京)有限公司 Automobile-based VR display device and method and automobile
CN107009961A (en) * 2017-03-03 2017-08-04 北京汽车股份有限公司 Automobile and automobile transparent cockpit system
CN110341597A (en) * 2018-04-02 2019-10-18 杭州海康威视数字技术股份有限公司 A kind of vehicle-mounted panoramic video display system, method and Vehicle Controller

Similar Documents

Publication Publication Date Title
CN105103034B (en) Display
US6947059B2 (en) Stereoscopic panoramic image capture device
JP4642723B2 (en) Image generating apparatus and image generating method
JP4257356B2 (en) Image generating apparatus and image generating method
KR100911066B1 (en) Image display system, image display method and recording medium
EP1179958A1 (en) Image processing device and monitoring system
JP6899875B2 (en) Information processing device, video display system, information processing device control method, and program
CN105959665A (en) Panoramic 3D video generation method for virtual reality equipment
CN106339980A (en) Automobile-based VR display device and method and automobile
JP2017204674A (en) Imaging device, head-mounted display, information processing system, and information processing method
WO2004068865A1 (en) Steroscopic panoramic image capture device
JP2019102936A (en) Display device, electronic mirror, control method of display device, and display control program
JP6768412B2 (en) Vehicle visibility device and vehicle visibility image display method
JP6649010B2 (en) Information processing device
JP2019095916A (en) Image generation device, head-mounted display, image generation system, image generation method, and program
CN110194109B (en) Method for displaying vehicle external image, image system and vehicle
JP2019102935A (en) Display device, electronic mirror, control method of display device, and display control program
WO2013133057A1 (en) Image processing apparatus, method, and program
CN116095298A (en) Vehicle exterior scene display method, head-mounted display device, terminal device and storage medium
CN112215917A (en) Vehicle-mounted panorama generation method, device and system
US11422621B2 (en) Method and system for displaying at least one spatial section depending on an eye position of a person
CN218877068U (en) Digital rearview mirror with image synthesis function, vehicle-mounted unit system and vehicle
CN118264773A (en) Vehicle-mounted video acquisition system, visual field linkage method and wearable electronic equipment
CN113438466A (en) Method, system, device and computer readable storage medium for widening external view field
JP2023179496A (en) Image processing device and image processing method

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination