CN116012905A - Driver road anger disease recognition and warning system - Google Patents
Driver road anger disease recognition and warning system Download PDFInfo
- Publication number
- CN116012905A CN116012905A CN202211203388.6A CN202211203388A CN116012905A CN 116012905 A CN116012905 A CN 116012905A CN 202211203388 A CN202211203388 A CN 202211203388A CN 116012905 A CN116012905 A CN 116012905A
- Authority
- CN
- China
- Prior art keywords
- data
- driver
- emotion
- anger
- server
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Traffic Control Systems (AREA)
Abstract
The invention discloses a driver road anger disease recognition and warning system. The driver road anger disease recognition and warning system comprises: the system comprises a multi-sensor data acquisition system, an edge calculation and storage system and an Internet of things communication system; compared with the traditional video monitoring equipment, the system has the advantages that the system integrates the functions of geographic position acquisition, emotion recognition, background reporting and the like besides a single video data acquisition and storage function, enriches the incontrollable emotion details of a driver, is convenient for follow-up tracking and backtracking, and improves monitoring effectiveness. The emotion recognition subsystem fully combines the advantages of edge calculation and the Internet of things system, a lightweight deep learning model is deployed at the embedded equipment, meanwhile, the situation of a driver and updating of OTA programs are reported by means of a wireless network, the mobility and expandability of data are enhanced, monitoring and overall planning are facilitated, and data support is improved for subsequent cloud big data analysis.
Description
Technical Field
The invention relates to the technical field of intelligent traffic and Internet of vehicles, in particular to a driver road anger recognition and warning system.
Background
The automobile monitoring is a video recording device applied to an automobile, and can completely record the driving condition outside the automobile and the driving state of a driver in the automobile. The method has important effects on guaranteeing the running safety of the vehicle and analyzing and identifying road traffic accidents. At present, the traditional automobile monitoring mainly relates to the storage of video recordings inside and outside an automobile, does not analyze videos, has single functions and does not fully exert the computing power of edge equipment.
The driver road anger syndrome recognition and warning system provided by the invention is used for detecting and classifying driver emotion, is suitable for universal portable embedded equipment, and is also suitable for real-time networking information reporting of the equipment, so that data backup is realized, traffic managers can conveniently master all driver states, labor cost is reduced, effective test means are provided for scientific research, production and daily maintenance of automobiles and timely searching for reasons of sudden traffic accidents, and vehicle faults are reduced, so that basis is provided for product design and fault analysis. Has important practical effect and significance for further researching the perfect design, fault analysis, cost reduction and traffic management of various vehicles.
Disclosure of Invention
The invention aims to overcome the defects of the prior art and provides a driver road anger disease identification and warning system.
The aim of the invention is realized by the following technical scheme: a driver road anger recognition and warning system;
in view of the above, the driver road anger recognition and warning system acquires the video voice and navigation sensor, realizes local storage and cloud synchronization of data, enhances data mobility and expandability, is convenient for monitoring and overall planning, and also improves data support for subsequent cloud big data analysis.
The system comprises a multi-sensor data acquisition system, an edge calculation and storage system and an Internet of things communication system, wherein:
the multi-sensor acquisition system comprises an SOC core board, an embedded software acquisition module and a peripheral circuit; the method comprises the steps of acquiring video, voice and geographic position data in the driving process for subsequent processing; the embedded software acquisition module runs Linux; the method comprises the steps of establishing a sub-thread newly, starting a timer to enable a peripheral circuit interface to acquire Beidou navigation longitude and latitude and time data, converting a data stream into video and extracting voice by using an FFmpeg tool;
the edge computing and storing system comprises a driver emotion recognition module, a parameter storing module and an image and video storing module; the method comprises the steps that data collected by a multi-sensor collection system are used for identifying the emotion of a driver in real time by using a neural network model, and data fragments are recorded and stored locally;
the communication system of the Internet of things comprises a wireless data transparent transmission module and a server communication module; the wireless data transparent transmission module is connected with a mobile network to realize a networking function; the server communication module realizes, receives the server issuing instruction, uses the specific URL link to download the model file, realizes the OTA updating of the software, and sends the datagram to the back-end server at regular time by the MQTT so as to carry out warning processing.
Further, the peripheral circuit comprises a UART interface for acquiring the longitude and latitude and time data of Beidou navigation, a USB camera interface for acquiring the image and video data, a microphone interface for acquiring the voice data, a UART interface for connecting with LTE transparent transmission equipment and an SD card interface.
Further, the driver emotion recognition module is used for judging driver emotion of road anger emotion and storing the driver emotion judgment and the SD card; the parameter storage module is used for realizing SD card storage of vehicle longitude and latitude and time information; the image and video storage module is used for storing the camera data into a video or picture.
Further, the emotion recognition process of the driver emotion recognition module includes: preprocessing data, extracting features, mixing multiple modes and judging emotion;
data preprocessing: acquiring a picture in a video, inputting an MTCNN network to detect a human face, and cutting the picture by using an OpenCV function to obtain a face picture of a driver; extracting semantic information by using a voice recognition algorithm;
feature extraction: inputting the face picture of the driver into a VGG network, analyzing the expression of the driver according to the characteristics of each part of the face, and selecting an anger probability value as an anger expression classification result value s1 according to the output result;
continuously extracting the voice of a driver by a background process, updating the MFCC, PLP-CC, fundamental frequency features, formant features and short-time energy extracted by using a library of library audio processing in real time, vectorizing the extracted features, carrying out vector splicing, and mapping the vector into a voice classification result value s2;
at the same time, the semantic information is compared with a local keyword library, the local keyword library integrates short sentences and words which are common in the road anger condition in a plurality of databases, and a coding value p is given according to the occurrence frequency of one word in the semantic information in each database i The code value represents the anger level, wherein i represents the ith word in the semantic information appearing in the database, and the code value p of each phrase in the semantic information is used 1 ...p i Giving a weight value w, w= (p) 1 +p 2 +...+p i )/i；
Multimode mixing; the two sub-threads for feature extraction start to communicate, the thread which completes calculation firstly waits for the end of the calculation of another score, then fusion is carried out, and the real-time total score is calculated and given, namely, the road anger degree x, x=s1 (1-w) +s2×w;
emotion judgment: when the anger threshold is set and the road anger level is greater than the set anger threshold, it is determined that the driver exhibits anger emotion.
Further, the MQTT communication flow in the server communication module includes:
message receiving, newly building a sub-thread, subscribing the theme of the MQTT server and outputting the received message as a receiving file;
reading a receiving file, and retransmitting a key when the key is out of date; when receiving the update program or the update model, the URL address is read from the additional content and downloaded to realize OTA upgrading;
the method comprises the steps of sending a message, starting a timer, sending a heartbeat data packet to an MQTT theme b1 of a server at intervals of 10 minutes, creating a sub-thread for monitoring emotion recognition results, and sending a road anger record data packet to an MQTT theme b2 of the server if a road anger emotion is generated; the data package is stored in a JSON format, wherein the JSON format comprises data numbers and data, the data are arranged in an array, and single data comprise parameter names, units and numerical values; the road anger data packet comprises geographic position information and time information recorded by a system clock.
Further, the MQTT in the communication system of the Internet of things sends datagrams to the back-end server at regular time so as to carry out warning processing, specifically;
creating a sub-thread monitoring road anger emotion detection result, and once a data packet is received, indicating that a system detects road anger emotion and gives warning through a server manager, wherein the server can give warning to the manager, and the manager can warn a driver.
The invention has the beneficial effects that:
the driver road anger recognition and warning terminal comprises three subsystems, and the hardware structure and the software flow are clear and complete. The computing capability of the edge equipment is fully exerted, multidimensional analysis is carried out on the video monitoring image, the road anger emotion judgment is carried out by utilizing the video image and the voice characteristics, when a driver generates the road anger emotion, the terminal sends an alarm data packet to the server, the warning function is realized, and the manager can check the road anger emotion conveniently. When the device runs normally, idle heartbeat packets can occur to the server, and the normal running of the device is indicated.
Drawings
FIG. 1 is a diagram of a peripheral circuit for identifying and warning driver's road anger according to an embodiment of the present invention;
FIG. 2 is a workflow diagram of an embedded software acquisition module provided by an embodiment of the present invention;
FIG. 3 is a working step diagram of a driver emotion recognition module according to an embodiment of the present invention;
FIG. 4 is a diagram of the communication working process of the MQTT communication module of the server provided by the embodiment of the present invention;
fig. 5 is a structural frame diagram of a driver road anger recognition and warning system according to an embodiment of the present invention.
Detailed Description
In order to more particularly describe the present invention, embodiments of the present invention will be described in further detail with reference to the accompanying drawings.
As shown in FIG. 5, the function subsystem of the driver road anger recognition and warning system provided by the invention comprises a multi-sensor data acquisition system, an edge calculation and storage system and a communication system connected with the Internet of things; the multi-sensor acquisition system is used for acquiring video, voice and geographic position data in the driving process; the edge computing and storing system has the functions that firstly, video data and voice data are preprocessed, then a neural network model is used for analyzing the emotion of a driver, and if the emotion of road anger is detected, a data segment is recorded and stored locally; the internet of things communication system is used for reporting data at regular time, realizing cloud information synchronization, and receiving a cloud data packet to realize OTA upgrading and model updating;
the multi-sensor data acquisition system comprises an SOC core board, an embedded software acquisition module and a peripheral circuit, wherein the peripheral circuit is composed of a voice IIS interface connected with a microphone, an eMMC interface connected with an SD card, a USB interface connected with a camera, a UART interface connected with Beidou navigation and a UART interface connected with transparent transmission equipment, as shown in figure 1.
As shown in fig. 2, the embedded software acquisition module is operated based on a Linux operation system, after a driver road anger identification and warning system starts to operate, firstly, initializing a transparent transmission module to establish network connection with a server, secondly, the embedded software acquisition module acquires a geographic position from a peripheral circuit, and then, creating three sub-threads t1, t2 and t3, wherein t1 is used for starting a 1s timer to periodically acquire geographic position information of Beidou navigation; t2 is used for processing the video data stream, and converting the camera data stream into video by using an FFmpeg tool; t3 captures voice data from the IIS interface.
The edge computing and storing system comprises a driver emotion recognition module, a parameter storing module and an image and video storing module; the driver emotion recognition module is used for judging whether the driver has road anger emotion or not; the parameter storage module is used for realizing SD card storage of vehicle longitude and latitude and time information; and the image and video storage module is used for storing the data of the camera as a video or a picture when the driver is detected to generate the anger emotion.
As shown in fig. 3, the operation of the driver emotion recognition module of the present invention includes the steps of:
step 1, data preprocessing, wherein a t2 thread selects one picture with highest quality from 5 adjacent pictures as a sample in a picture sequence in a video data stream according to judging rules such as definition, whether the picture sequence contains a complete face or not, inputs the sample into an MTCNN network to detect the face, and cuts the face by using an OpenCV function to obtain a face picture p1 of a driver; the t3 thread reads the voice data and filters out the ambient noise signals, then extracts the semantic information m1 by using a voice recognition algorithm, and simultaneously divides the voice signals into a plurality of short-time frame signal sets v1. In particular, short-time frame lengths are 10-30ms, and speech signals have time variability, but are considered relatively stable for short periods of time, which are 10-30ms.
And 2, feature extraction, wherein a t2 thread inputs p1 into a VGG network, the network is composed of four convolution layers and three full connection layers, a one-dimensional vector with the length of 7 is finally output, and the probability values of 7 expressions (surprise, sadness, neutrality, happiness, fear, aversion and anger) corresponding to an input image are contained. According to the finally output vector, calculating an anger probability value, and multiplying the anger probability value by 100 to obtain an anger expression score s1;
the t3 thread will extract the sound features of v1, specifically, extracting the sound features of a plurality of short-time frame signals includes: the MFCC, PLP-CC, fundamental frequency feature, formant feature and short-time energy extracted by using a library of library audio processing are vectorized, vector stitching is carried out on the extracted features, and then the vector is mapped into a score s2;
the t3 thread compares the semantic information m1 with a local keyword library to judge whether a road anger scene word appears, specifically, the local keyword library is integrated by a word set for emotion analysis of a known net, a simplified Chinese emotion polarity dictionary of Taiwan university, a positive desense dictionary (Qinghua university), comprises short sentences and words common under the road anger condition, and gives different coding values according to the occurrence times of the same word in three data sets. Specifically, if a word appears in all three data sets, its code value is 0.9; appears in both data sets, its code value is 0.7; only in one data set, its code value is 0.5. The code value represents the extent to which the phrase or word symbolizes the road anger emotion. For example, when the semantic information m1 includes driving scene sentences such as "stopover", "will not start" or anger word, it is found that 3 times of comparison occur in the local keyword library, and at this time, a weight value w, w= (p1+p2+p3)/3 is given according to the code values p1, p2, p3 of the word library phrases.
And 3, carrying out multi-mode fusion, wherein t2 and t3 threads are communicated, the thread which completes calculation firstly waits for the end of the calculation of another score value, then carrying out fusion, and calculating to give a real-time total score, namely the road anger degree x, wherein x=s1 (1-w) +s2×w. Specifically, the values of s1 and s2 range from 0 to 100, and the weight value ranges from 0 to 1. The calculation result x is also in the range of 0 to 100.
And 4, judging the emotion, namely comparing the fusion result value x with a set anger threshold value, and judging that the driver has road anger emotion if the fusion result value x exceeds 75 in the embodiment.
The communication system of the Internet of things comprises a wireless data transparent transmission module and a server communication module; the wireless data transparent transmission module exchanges data with the equipment through a UART interface and is connected to a 4G mobile network to realize a networking communication function; the server communication module is used for sending the MQTT data packet to the back-end server at regular time, receiving the server issuing instruction and a specific URL link downloading model file, and realizing OTA updating of software;
as shown in fig. 4, the working process of the server communication module for implementing MQTT communication includes the following steps:
step 1, message reception, comprising the following sub-steps:
step 11, creating a sub-thread by the main process, subscribing the MQTT theme of the back-end server, and if no message exists, circularly waiting; if so, the received message is saved as a file and the process proceeds to step 12.
And step 12, analyzing and processing the received message. Reading the file saved in the step 11, and retransmitting the key if a key expiration instruction is received; if an update program instruction or an update model instruction is received, the URL address is read from the additional content of the message, and the corresponding file is downloaded, so that one OTA upgrade is realized.
And step 2, sending a message, starting a timer by a main process, sending a heartbeat data packet to the MQTT theme b1 of the server at intervals of 10 minutes, creating a sub-thread for monitoring emotion recognition results, and if a road anger emotion is generated, sending a road anger record data packet to the MQTT theme b2 of the server. Specifically, the data packet is transmitted in a JSON format, wherein the data packet comprises a data number and data, the data is arranged in an array, and the single data comprises a parameter name, a unit and a numerical value; specifically, the heartbeat data packet is a blank data packet which does not transmit effective data, and the parameter information is null; the parameter information of the road anger record data packet comprises geographical position information acquired by the sub-thread t1 and time information recorded by a system clock.
The two data packets are encrypted and then sent to the corresponding MQTT theme of the server, a server manager can know whether the system operates normally or not from the MQTT theme b1, and can know when and in what geographic position the system detects that the driver generates the road anger emotion from the MQTT theme b2. The MQTT theme b1 indicates that the system is operating properly once a packet is received. Similarly, once the MQTT theme b2 receives the data packet, it indicates that the system detects the anger emotion, and the server sends a warning to the administrator, so that the administrator can warn the driver or assist the traffic management department in dividing responsibility when a traffic accident occurs. Specifically, the warning method includes the steps of notifying a short message or a telephone by purchasing an operator service, or displaying system information in real time by using a front-end webpage, and displaying the system information in detail in an interface when road anger data occurs.
The previous description of the embodiments is provided to facilitate a person of ordinary skill in the art in order to make and use the present invention; the foregoing description of the preferred embodiments of the invention is not intended to be limiting, but rather is intended to cover all modifications, equivalents, and alternatives falling within the spirit and principles of the invention.
The above-described embodiments are intended to illustrate the present invention, not to limit it, and any modifications and variations made thereto are within the spirit of the invention and the scope of the appended claims.
Claims (6)
1. The system is characterized by comprising a multi-sensor data acquisition system, an edge calculation and storage system and an Internet of things communication system, wherein:
the multi-sensor acquisition system comprises an SOC core board, an embedded software acquisition module and a peripheral circuit; the method comprises the steps of acquiring video, voice and geographic position data in the driving process for subsequent processing; the embedded software acquisition module runs Linux; the method comprises the steps of establishing a sub-thread newly, starting a timer to enable a peripheral circuit interface to acquire Beidou navigation longitude and latitude and time data, converting a data stream into video and extracting voice by using an FFmpeg tool;
the edge computing and storing system comprises a driver emotion recognition module, a parameter storing module and an image and video storing module; the method comprises the steps that data collected by a multi-sensor collection system are used for identifying the emotion of a driver in real time by using a neural network model, and data fragments are recorded and stored locally;
the communication system of the Internet of things comprises a wireless data transparent transmission module and a server communication module; the wireless data transparent transmission module is connected with a mobile network to realize a networking function; the server communication module realizes, receives the server issuing instruction, uses the specific URL link to download the model file, realizes the OTA updating of the software, and sends the datagram to the back-end server at regular time by the MQTT so as to carry out warning processing.
2. The driver road anger recognition and warning system according to claim 1, wherein the peripheral circuit comprises a UART interface for acquiring the longitude and latitude and time data of Beidou navigation, a USB camera interface for acquiring the image and video data, a microphone interface for acquiring the voice data, a UART interface for connecting the LTE transparent transmission equipment, and an SD card interface.
3. The driver road anger syndrome recognition and warning system according to claim 1, wherein the driver emotion recognition module is configured to implement driver emotion determination and SD card storage of road anger emotion; the parameter storage module is used for realizing SD card storage of vehicle longitude and latitude and time information; the image and video storage module is used for storing the camera data into a video or picture.
4. The driver road anger recognition and warning system of claim 3, wherein the emotion recognition process of the driver emotion recognition module comprises: preprocessing data, extracting features, mixing multiple modes and judging emotion;
data preprocessing: acquiring a picture in a video, inputting an MTCNN network to detect a human face, and cutting the picture by using an OpenCV function to obtain a face picture of a driver; extracting semantic information by using a voice recognition algorithm;
feature extraction: inputting the face picture of the driver into a VGG network, analyzing the expression of the driver according to the characteristics of each part of the face, and selecting an anger probability value as an anger expression classification result value s1 according to the output result;
continuously extracting the voice of a driver by a background process, updating the MFCC, PLP-CC, fundamental frequency features, formant features and short-time energy extracted by using a library of library audio processing in real time, vectorizing the extracted features, carrying out vector splicing, and mapping the vector into a voice classification result value s2;
at the same time, the semantic information is compared with a local keyword library, the local keyword library integrates short sentences and words which are common in the road anger condition in a plurality of databases, and a coding value p is given according to the occurrence frequency of one word in the semantic information in each database i The code value represents the anger level, wherein i represents the ith word in the semantic information appearing in the database, and the code value p of each phrase in the semantic information is used 1 ...p i Giving a weight value w, w= (p) 1 +p 2 +...+p i )/i；
Multimode mixing; the two sub-threads for feature extraction start to communicate, the thread which completes calculation firstly waits for the end of the calculation of another score, then fusion is carried out, and the real-time total score is calculated and given, namely, the road anger degree x, x=s1 (1-w) +s2×w;
emotion judgment: when the anger threshold is set and the road anger level is greater than the set anger threshold, it is determined that the driver exhibits anger emotion.
5. The driver road anger recognition and warning system according to claim 1, wherein the MQTT communication flow in the server communication module comprises:
message receiving, newly building a sub-thread, subscribing the theme of the MQTT server and outputting the received message as a receiving file;
reading a receiving file, and retransmitting a key when the key is out of date; when receiving the update program or the update model, the URL address is read from the additional content and downloaded to realize OTA upgrading;
the method comprises the steps of sending a message, starting a timer, sending a heartbeat data packet to an MQTT theme b1 of a server at intervals of 10 minutes, creating a sub-thread for monitoring emotion recognition results, and sending a road anger record data packet to an MQTT theme b2 of the server if a road anger emotion is generated; the data package is stored in a JSON format, wherein the JSON format comprises data numbers and data, the data are arranged in an array, and single data comprise parameter names, units and numerical values; the road anger data packet comprises geographic position information and time information recorded by a system clock.
6. The system for identifying and warning road anger of drivers according to claim 5, wherein the MQTT in the internet of things communication system sends data messages to the back-end server at regular time so as to perform warning processing, specifically;
creating a sub-thread monitoring road anger emotion detection result, and once a data packet is received, indicating that a system detects road anger emotion and gives warning through a server manager, wherein the server can give warning to the manager, and the manager can warn a driver.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211203388.6A CN116012905A (en) | 2022-09-29 | 2022-09-29 | Driver road anger disease recognition and warning system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211203388.6A CN116012905A (en) | 2022-09-29 | 2022-09-29 | Driver road anger disease recognition and warning system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116012905A true CN116012905A (en) | 2023-04-25 |
Family
ID=86032424
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211203388.6A Pending CN116012905A (en) | 2022-09-29 | 2022-09-29 | Driver road anger disease recognition and warning system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116012905A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117657170A (en) * | 2024-02-02 | 2024-03-08 | 江西五十铃汽车有限公司 | Intelligent safety and whole vehicle control method and system for new energy automobile |
-
2022
- 2022-09-29 CN CN202211203388.6A patent/CN116012905A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117657170A (en) * | 2024-02-02 | 2024-03-08 | 江西五十铃汽车有限公司 | Intelligent safety and whole vehicle control method and system for new energy automobile |
| CN117657170B (en) * | 2024-02-02 | 2024-05-17 | 江西五十铃汽车有限公司 | Intelligent safety and whole vehicle control method and system for new energy automobile |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106297785B (en) | Intelligent service system based on Internet of vehicles | |
| US20200007638A1 (en) | Method and apparatus for pushing information | |
| EP2377051A2 (en) | Generating predilection cohorts | |
| CN110099246A (en) | Monitoring and scheduling method, apparatus, computer equipment and storage medium | |
| CN109473119B (en) | Acoustic target event monitoring method | |
| CN112744174B (en) | Vehicle collision monitoring method, device, equipment and computer readable storage medium | |
| CN112785837A (en) | Method and device for recognizing emotion of user when driving vehicle, storage medium and terminal | |
| CN107800793A (en) | Car networking environment down train music active pushing system | |
| CN111951560B (en) | Service anomaly detection method, method for training service anomaly detection model and method for training acoustic model | |
| CN116012905A (en) | Driver road anger disease recognition and warning system | |
| CN111540222A (en) | Intelligent interaction method and device based on unmanned vehicle and unmanned vehicle | |
| CN110070893A (en) | A kind of system, method and apparatus carrying out sentiment analysis using vagitus | |
| CN115312061A (en) | Voice question-answer method and device in driving scene and vehicle-mounted terminal | |
| CN114353819A (en) | Navigation method, device, equipment, storage medium and program product for vehicle | |
| CN111191498A (en) | Behavior recognition method and related product | |
| CN116932919A (en) | Information pushing method, device, electronic equipment and computer readable medium | |
| CN109817224A (en) | A kind of voice sensitive word monitor system and method | |
| CN117407507A (en) | Event processing method, device, equipment and medium based on large language model | |
| CN111267759A (en) | Automobile intelligent interaction system for realizing AI interaction | |
| CN113221990B (en) | Information input method and device and related equipment | |
| CN113609956B (en) | Training method, recognition device, electronic equipment and storage medium | |
| CN114944152A (en) | Vehicle whistling sound identification method | |
| CN113592262B (en) | Safety monitoring method and system for network appointment | |
| CN110956072B (en) | Driving skill training method based on big data analysis | |
| CN114332798A (en) | Processing method and related device for network car booking environment information |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |