CN115685616A - Display device and eye movement tracking system - Google Patents
Display device and eye movement tracking system Download PDFInfo
- Publication number
- CN115685616A CN115685616A CN202211331001.5A CN202211331001A CN115685616A CN 115685616 A CN115685616 A CN 115685616A CN 202211331001 A CN202211331001 A CN 202211331001A CN 115685616 A CN115685616 A CN 115685616A
- Authority
- CN
- China
- Prior art keywords
- mini led
- infrared
- led module
- display device
- liquid crystal
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Liquid Crystal (AREA)
Abstract
The present invention relates to a display device, characterized in that the display device includes: the liquid crystal display device comprises a liquid crystal panel and a Mini LED unit, wherein the Mini LED unit is arranged on one side of the liquid crystal panel and at least comprises: and the infrared Mini LED module is used for emitting infrared light. The infrared Mini LED module is arranged in the display device to emit infrared light to irradiate the eyes of a user so as to obtain a clear infrared eye image. Further, through setting up the blue light Mini LED module of permeable infrared light and the green glow Mini LED module of permeable infrared light, the infrared light directly sees through blue light Mini LED module and green glow Mini LED module and directly shines in user's people's eye to fuse infrared light filling scheme and screen display scheme mutually, in order to improve the luminous utilization ratio of infrared lamp, reduce the infrared lamp loss, can improve display quality simultaneously.
Description
Technical Field
The present invention relates to an eye tracking system and method, and more particularly, to a display device and an eye tracking system.
Background
The eye tracking technology is widely applied to the fields of human-computer interaction, virtual reality, vehicle auxiliary driving, human factor analysis, psychological research and the like. Eye tracking is a process of identifying what and how a person looks at, and from the physiological structure of the eyeball, humans mainly acquire visual data through the foveal region, which can only provide an angle of view of about 1-2 degrees. Although this region occupies only a very small portion of the visual field, the information recorded by this region contains 50% of the available visual information that is transmitted to the brain via the optic nerve. Thus, the human vision and attention system works around a primary object, focusing the optical imaging of the object of interest into a central recess.
Eye tracking devices can be largely classified into two types, classified by the usage scenario: a screen type eye movement instrument and a wearable eye movement instrument. The screen-type eye tracker is placed at a distance from the user to track the user's eye movements. The wearable eye tracker collects eye movement behaviors of a user in a real environment by integrating the eye movement tracking system and the scene camera on a light-weight frame, such as glasses or a helmet, so that the detection precision of the sight line direction is high, the user can move freely, and the ecological effectiveness of an experiment is maximized.
The display device and the camera in the existing eye movement tracking product are usually mutually independent, the camera is generally arranged below the display screen and upwards provided with an inclination angle, so that the eyes of a shot image are not right opposite to the camera and have an obvious inclination angle. The camera is arranged in the screen, and the problem that clear human eye images are difficult to obtain due to insufficient light brightness exists. Therefore, it is necessary to provide an eye movement tracking scheme that can not only capture an image of a human eye directly against the human eye, but also overcome the defect of insufficient brightness and make it difficult to obtain a clear image of the human eye.
Disclosure of Invention
The invention aims to provide an eye movement tracking scheme which can be used for shooting eyes to obtain eye images, overcomes the defect of insufficient brightness and is difficult to obtain clear eye images.
According to an aspect of the present invention, there is provided a display device including:
a liquid crystal panel having a plurality of liquid crystal cells,
a Mini LED unit disposed at one side of the liquid crystal panel, and the Mini LED unit at least includes: and the infrared Mini LED module is used for emitting infrared light.
More preferably, the display device further includes:
and the infrared driving unit is connected with the infrared Mini LED module and is used for controlling the infrared Mini LED module to be turned on or turned off or adjusting the brightness.
More preferably, the liquid crystal panel includes:
a normal display area covering a part of a display area of the liquid crystal panel;
the display area under the screen covers the other part of the display area of the liquid crystal panel;
and recording the direction vertical to the liquid crystal panel as a first direction, and observing along the first direction, wherein the infrared Mini LED module is positioned in the normal display area.
More preferably, the Mini LED unit further comprises:
a red Mini LED module, a blue Mini LED module and a green Mini LED module,
and observing along the first direction, wherein the blue light Mini LED module and the green light Mini LED module which are positioned in the normal display area can both transmit infrared light.
Preferably, the blue Mini LED module and the green Mini LED module are disposed between the infrared Mini LED module and the liquid crystal panel.
More preferably, the display device further includes:
and the LED control unit is respectively connected with the red light Mini LED module, the blue light Mini LED module and the green light Mini LED module.
More preferably, the display device further includes:
the optical diffusion layer is arranged on one side, away from the liquid crystal panel, of the Mini LED unit;
and the quantum film is arranged on one side of the Mini LED unit close to the liquid crystal panel.
An eye tracking system comprising the display device, the system further comprising:
the infrared camera under the screen, set up in one side of LCD panel observes along the first direction, infrared camera is located under the screen in the display area under the LCD panel's the screen, just infrared camera is used for shooing user's people's eye under the screen to obtain infrared people's eye image.
More preferably, the system further comprises:
the image shooting module is used for controlling the infrared camera under the screen to shoot the eyes of the user illuminated by the infrared light to obtain an infrared eye image;
and the image processing module is used for carrying out image processing on the infrared human eye image to obtain the position of the current gazing of the human eyes of the user on the liquid crystal screen.
The technical effects are as follows:
an infrared Mini LED module is arranged in a display device to emit infrared light which irradiates eyes of a user so as to obtain a clear infrared eye image;
through setting up the blue light Mini LED module of permeable infrared light and the green glow Mini LED module of permeable infrared light, and with infrared Mini LED module set up in blue light Mini LED module with green glow Mini LED module deviates from one side of liquid crystal display panel, the infrared light directly sees through blue light Mini LED module and green glow Mini LED module and directly shines in user's people's eye to fuse infrared light filling scheme and screen display scheme mutually, in order to improve the luminous utilization ratio of infrared lamp, reduce the infrared lamp loss, can improve display quality simultaneously.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts.
Fig. 1 is a schematic structural diagram of a display device according to an embodiment of the invention;
fig. 2 is a block diagram of a positional relationship structure between a Mini LED unit and a liquid crystal panel according to an embodiment of the present invention;
FIG. 3 is a schematic structural diagram of a liquid crystal panel according to an embodiment of the invention;
fig. 4 is a connection block diagram of an LED control unit according to an embodiment of the present invention;
fig. 5 is a connection block diagram of an infrared driving unit according to an embodiment of the present invention;
FIG. 6 is a block diagram illustrating the connection of an eye tracking system according to an embodiment of the present invention;
the reference numbers indicate: 100. a display device; 10. a liquid crystal panel; 20. a Mini LED unit; 21. an infrared Mini LED module; 30. an infrared drive unit; 11. a normal display area; 12. an off-screen display area; 22. a red Mini LED module; 23. a blue Mini LED module; 24. a green Mini LED module; 40. an LED control unit; 50. an optical diffusion layer; 60. a quantum film; 200. an eye tracking system; 210. an infra-red camera under the screen; 230. an image capturing module; 240. an image processing module; f1, first direction.
Detailed Description
To facilitate an understanding of the invention, the invention will now be described more fully with reference to the accompanying drawings. Preferred embodiments of the present invention are shown in the drawings. This invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete.
It will be understood that when an element is referred to as being "secured to" another element, it can be directly on the other element or intervening elements may also be present. When an element is referred to as being "connected" to another element, it can be directly connected to the other element or intervening elements may also be present. The terms "vertical," "horizontal," "left," "right," and the like as used herein are for illustrative purposes only.
Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used herein, the term "and/or" includes any and all combinations of one or more of the associated listed items.
The present embodiment provides a display device 100, and the display device 100 is applied to an eye movement tracking scheme, and the present embodiment takes an application scenario of VR glasses as an example. The display device 100 corresponds to a built-in display screen of VR glasses, and when a user wears the VR glasses, the relative positions of the eyes and the display device 100 are relatively fixed. The VR glasses have a characteristic that when a user rotates the head, the image on the display device 100 rotates along with the rotation of the head. More specifically, when the head of the person rotates, the head of the person may be reflected in the VR glasses along with the rotation of the eyeball, that is, the change of the focus position of the eye on the display device 100 may be obtained, and therefore, the change of the focus position of the eye on the display screen may be calculated by analyzing the change of the image features of the eye. The human eye image needs to be shot to analyze the human eye image, and the definition of the human eye image images shows the accuracy of an analysis result. In this embodiment, the human eye image is obtained by means of shooting under the screen, that is, the under-screen camera is disposed in the display device 100, and the infrared Mini LED module 21 for infrared supplementary lighting is disposed in the display device 100.
Specifically, the display device 100 includes: a liquid crystal panel 10, a Mini LED unit,
the liquid crystal panel 10 is an IPS screen. The Mini LED unit is disposed at one side of the liquid crystal panel 10, and the Mini LED unit at least includes: the LED module comprises an infrared Mini LED module 21, wherein the infrared Mini LED module 21 is used for emitting infrared light. The infrared light emitted by the infrared Mini LED module 21 passes through the liquid crystal panel 10 and then irradiates the eyes of the user, so that the infrared light brightness at the eyes of the user is improved, and the definition of the infrared image of the eyes of the user shot by the infrared camera 210 under the screen is improved.
More preferably, the display device 100 further includes: an infrared drive unit 30.
Specifically, the infrared driving unit 30 is connected to the infrared Mini LED module 21, and is configured to control the infrared Mini LED module 21 to be turned on, turned off, or adjusted in brightness, so as to implement independent control of infrared supplementary lighting.
More preferably, the liquid crystal panel 10 includes: a normal display area 11, an off-screen display area 12.
The normal display area 11 refers to an area for normally displaying images and for infrared supplementary lighting on the liquid crystal panel 10, and the under-screen display area 12 refers to an area for setting the under-screen infrared camera 210 on the liquid crystal panel 10.
The normal display area 11 may be an area for only infrared supplementary lighting, for example, the infrared Mini LED module 21 is only disposed in the normal display area 11, and the red Mini LED module 22, the blue Mini LED module 23, and the green Mini LED module 24 for normally forming an image are not disposed, and the area may be disposed in one or more of the following areas of the liquid crystal panel 10: one or more of the four corner regions, a left side region, a right side region, an upper side region, a lower side region. Referring to fig. 3, in the present embodiment, an infrared Mini LED module 21 is provided in the left side region of the liquid crystal panel 10.
The normal display area 11 may also be an area that is used for both infrared light supplement and image display, for example, not only the infrared Mini LED module 21 is disposed in the normal display area 11, but also the red Mini LED module 22, the blue Mini LED module 23, and the green Mini LED module 24 are disposed in the normal display area 11, wherein the blue Mini LED module 23 and the green Mini LED module 24 can transmit infrared light, and therefore, the infrared Mini LED module 21 is disposed only under the blue Mini LED module 23 and the green Mini LED module 24, so that the normal display area 11 may be used for both infrared light supplement and normal image display.
It should be noted that the red Mini LED module 22 may also be disposed in the normal display area 11, but the red Mini LED module 22 cannot pass through infrared light, so that the area corresponding to the red Mini LED module 22 cannot be used for infrared supplementary lighting. This is because the blue Mini LED module 23 and the green Mini LED module 24 use gallium nitride (GaN) as a substrate, and light can be transmitted through gallium nitride (GaN) without being absorbed or significantly attenuated, but the red Mini LED module 22 uses aluminum indium gallium phosphide (AlInGaP) and opaque gallium arsenide (GaAs) as a substrate, and infrared light cannot pass through the red Mini LED module 22 and irradiate to eyes of a user.
Specifically, the light transmittance of the liquid crystal panel 10 is between 50% and 100%, optionally, the light transmittance of the liquid crystal panel 10 is 50%, 60%, 70%, 80%, 90%, 95%, 96%, 97%, 98%, 99%, or 100%, and in this embodiment, the light transmittance of the liquid crystal panel 10 is 98%.
Specifically, the infrared Mini LED module 21 emits infrared light having a wavelength of 930nm to 950nm. The wavelength of the infrared light emitted by the infrared Mini LED module 21 may be selected to be 930nm, 940nm, 950nm, or the like, and in this embodiment, the infrared light having the wavelength of the infrared light H emitted by the infrared Mini LED module 21 of 940nm and 940nm may be selected to avoid the red exposure phenomenon.
In this embodiment, the wavelength of the infrared light H received by the off-screen infrared camera 210 is 930nm to 950nm. Optionally, the wavelength of the infrared light H received by the under-screen infrared camera 210 may be 930nm, 940nm, 950nm, or the like, and the wavelength of the infrared light H received by the under-screen infrared camera 210 in this embodiment may be 940nm.
Specifically, the normal display area 11 of the liquid crystal panel 10 covers a part of the display area of the liquid crystal panel 10, and the off-screen display area 12 of the liquid crystal panel 10 covers another part of the display area of the liquid crystal panel 10. The direction perpendicular to the liquid crystal panel 10 is denoted as a first direction F1, and the infrared Mini LED module 21 is located in the normal display area 11 when viewed along the first direction F1.
More preferably, the Mini LED unit further comprises: a red Mini LED module 22, a blue Mini LED module 23 and a green Mini LED module 24.
When viewed along the first direction F1, both the blue Mini LED module 23 and the green Mini LED module 24 in the normal display area 11 can transmit infrared light. And the blue Mini LED module 23 and the green Mini LED module 24 are disposed between the infrared Mini LED module 21 and the liquid crystal panel 10. Therefore, the infrared light emitted by the infrared Mini LED module 21 passes through the blue Mini LED module 23 and the green Mini LED module 24, further passes through the liquid crystal panel 10, and irradiates the eyes of the user.
More preferably, the display device 100 further includes: the LED control unit 40 and the LED control unit 40 are respectively connected with the red light Mini LED module 22, the blue light Mini LED module 23 and the green light Mini LED module 24.
More preferably, the display device 100 further includes: an optical diffusion layer 50 and a quantum film 60.
The optical diffusion layer 50 is disposed on a side of the Mini LED unit away from the liquid crystal panel 10. The quantum film 60 is disposed on one side of the Mini LED unit close to the liquid crystal panel 10.
The present embodiment further provides an eye tracking system 200, including the display device 100, the system further including: an infra-red camera 210 under the screen.
Wherein, infrared camera 210 sets up in one side of LCD panel 10 under the screen, and observe along first direction F1, infrared camera 210 is located display area 12 under the screen, and infrared camera 210 is used for shooing user's people's eyes under the screen to obtain infrared people's eye image.
More preferably, the system further comprises: an image shooting module 230 and an image processing module 240.
The image capturing module 230 is configured to control the under-screen infrared camera 210 to capture eyes of a user illuminated by infrared light, so as to obtain an infrared eye image. The image processing module 240 is configured to perform image processing on the infrared human eye image to obtain a position where the human eyes of the user currently gaze on the liquid crystal screen.
Therefore, the infrared Mini LED module 21 arranged in the display device 100 emits infrared light to irradiate the eyes of the user, so as to obtain a clear infrared eye image; through setting up blue light Mini LED module 23 and the green glow Mini LED module 24 of permeable infrared light, and set up infrared Mini LED module 21 in blue light Mini LED module 23 with green glow Mini LED module 24 deviates from one side of liquid crystal display panel 10, the infrared light directly sees through blue light Mini LED module 23 and green glow Mini LED module 24 and directly shines in user's people's eye, thereby fuse infrared light filling scheme and screen display scheme mutually, in order to improve the luminous utilization ratio of infrared lamp, reduce the infrared lamp loss, can improve display quality simultaneously.
The above-mentioned embodiments only express several embodiments of the present invention, and the description thereof is more specific and detailed, but not construed as limiting the claims. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the inventive concept, which falls within the scope of the present invention. Therefore, the protection scope of the present patent shall be subject to the appended claims.
Claims (10)
1. A display device, characterized in that the display device comprises:
a liquid crystal panel having a plurality of liquid crystal cells,
a Mini LED unit disposed at one side of the liquid crystal panel, and the Mini LED unit at least includes: and the infrared Mini LED module is used for emitting infrared light.
2. The display device according to claim 1, characterized in that the display device further comprises:
and the infrared driving unit is connected with the infrared Mini LED module and is used for controlling the infrared Mini LED module to be turned on or turned off or adjusting the brightness.
3. The display device according to claim 2, wherein the liquid crystal panel comprises:
a normal display area covering a part of a display area of the liquid crystal panel;
the under-screen display area covers the other part of the display area of the liquid crystal panel;
and recording the direction vertical to the liquid crystal panel as a first direction, and observing along the first direction, wherein the infrared Mini LED module is positioned in the normal display area.
4. The display device of claim 3, wherein the Mini LED unit further comprises:
a red Mini LED module, a blue Mini LED module and a green Mini LED module,
and observing along the first direction, wherein the blue light Mini LED module and the green light Mini LED module which are positioned in the normal display area can both transmit infrared light.
5. The display device of claim 4, wherein the blue Mini LED module and the green Mini LED module are disposed between the infrared Mini LED module and the liquid crystal panel.
6. The display device according to claim 5, further comprising:
and the LED control unit is respectively connected with the red light Mini LED module, the blue light Mini LED module and the green light Mini LED module.
7. The display device according to claim 1, further comprising:
and the optical diffusion layer is arranged on one side of the Mini LED unit, which is deviated from the liquid crystal panel.
8. The display device according to claim 1, characterized in that the display device further comprises:
and the quantum film is arranged on one side of the Mini LED unit close to the liquid crystal panel.
9. An eye tracking system comprising the display device of any one of claims 1-8, the system further comprising:
the infrared camera under the screen, set up in one side of LCD panel observes along the first direction, infrared camera is located under the screen in the display area under the LCD panel's the screen, just infrared camera is used for shooing user's people's eye under the screen to obtain infrared people's eye image.
10. The eye tracking system of claim 9, further comprising:
the image shooting module is used for controlling the infrared camera under the screen to shoot the eyes of the user illuminated by the infrared light to obtain an infrared eye image;
and the image processing module is used for carrying out image processing on the infrared human eye image to obtain the position of the current gazing of the human eyes of the user on the liquid crystal screen.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211331001.5A CN115685616A (en) | 2022-10-27 | 2022-10-27 | Display device and eye movement tracking system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211331001.5A CN115685616A (en) | 2022-10-27 | 2022-10-27 | Display device and eye movement tracking system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115685616A true CN115685616A (en) | 2023-02-03 |
Family
ID=85045728
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211331001.5A Pending CN115685616A (en) | 2022-10-27 | 2022-10-27 | Display device and eye movement tracking system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115685616A (en) |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103713738A (en) * | 2013-12-17 | 2014-04-09 | 武汉拓宝电子***有限公司 | Man-machine interaction method based on visual tracking and gesture recognition |
| CN109871782A (en) * | 2019-01-28 | 2019-06-11 | 深圳市力川智控科技有限公司 | Method and system based on the identification of Micro LED liquid crystal display iris capturing |
| CN110333779A (en) * | 2019-06-04 | 2019-10-15 | Oppo广东移动通信有限公司 | Control method, terminal and storage medium |
| CN112631432A (en) * | 2021-01-05 | 2021-04-09 | Oppo广东移动通信有限公司 | Screen control method, wearable device and storage medium |
| US20210227674A1 (en) * | 2020-01-21 | 2021-07-22 | Brightlogic, Inc. | Light Emitting Diode Package and Electronic Display |
| CN114035323A (en) * | 2021-10-29 | 2022-02-11 | 歌尔光学科技有限公司 | A display panel and intelligent glasses for eyeball is tracked |
| CN114624881A (en) * | 2020-12-08 | 2022-06-14 | 三星电子株式会社 | Gaze point display device |
| CN114660803A (en) * | 2022-03-07 | 2022-06-24 | 上海鱼微阿科技有限公司 | Eye tracking system under screen |
| CN114740994A (en) * | 2022-03-25 | 2022-07-12 | 厦门天马微电子有限公司 | Display screen and display device |
-
2022
- 2022-10-27 CN CN202211331001.5A patent/CN115685616A/en active Pending
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103713738A (en) * | 2013-12-17 | 2014-04-09 | 武汉拓宝电子***有限公司 | Man-machine interaction method based on visual tracking and gesture recognition |
| CN109871782A (en) * | 2019-01-28 | 2019-06-11 | 深圳市力川智控科技有限公司 | Method and system based on the identification of Micro LED liquid crystal display iris capturing |
| CN110333779A (en) * | 2019-06-04 | 2019-10-15 | Oppo广东移动通信有限公司 | Control method, terminal and storage medium |
| US20210227674A1 (en) * | 2020-01-21 | 2021-07-22 | Brightlogic, Inc. | Light Emitting Diode Package and Electronic Display |
| CN114624881A (en) * | 2020-12-08 | 2022-06-14 | 三星电子株式会社 | Gaze point display device |
| CN112631432A (en) * | 2021-01-05 | 2021-04-09 | Oppo广东移动通信有限公司 | Screen control method, wearable device and storage medium |
| CN114035323A (en) * | 2021-10-29 | 2022-02-11 | 歌尔光学科技有限公司 | A display panel and intelligent glasses for eyeball is tracked |
| CN114660803A (en) * | 2022-03-07 | 2022-06-24 | 上海鱼微阿科技有限公司 | Eye tracking system under screen |
| CN114740994A (en) * | 2022-03-25 | 2022-07-12 | 厦门天马微电子有限公司 | Display screen and display device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11809628B2 (en) | See-through computer display systems | |
| US11796799B2 (en) | See-through computer display systems | |
| US11786105B2 (en) | Content presentation in head worn computing | |
| US6227667B1 (en) | Apparatus for recording the retina reflex image and for superimposing of additional images in the eye | |
| CN107894666B (en) | Head-mounted multi-depth stereo image display system and display method | |
| US11269182B2 (en) | Content presentation in head worn computing | |
| JP6308940B2 (en) | System and method for identifying eye tracking scene reference position | |
| CN103376551B (en) | Small-hole-projection-type near-eye display | |
| CN109597201A (en) | Fresnel component for the light-redirecting in eyes tracking system | |
| JPWO2015012280A1 (en) | Gaze detection device | |
| JP2010250789A (en) | Display device with camera | |
| CN109725462B (en) | Display device, display apparatus, and driving method of display device | |
| CN204595329U (en) | Head-mounted display apparatus | |
| CN109725416B (en) | Eyeball tracking optical system, head-mounted equipment and imaging method | |
| US20230258929A1 (en) | See-through computer display systems | |
| CN114153066A (en) | Head-up display device and head-up display system | |
| CN115685616A (en) | Display device and eye movement tracking system | |
| CN111700586B (en) | Eye movement tracking device and electronic device using same | |
| RU2301436C2 (en) | Wide-angle virtual helmet with possibility of combination of real and virtual space | |
| JP2001242417A (en) | Pupil position detector and image display device using the same | |
| JPH11249588A (en) | Head-mounted display | |
| CN116149051A (en) | Lens assembly, infrared light supplementing method and eye movement tracking method and system | |
| CN116068756A (en) | Display device and head-mounted display equipment suitable for eye movement tracking |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |