CN112923938A - Map optimization method, device, storage medium and system - Google Patents
Map optimization method, device, storage medium and system Download PDFInfo
- Publication number
- CN112923938A CN112923938A CN202110190388.6A CN202110190388A CN112923938A CN 112923938 A CN112923938 A CN 112923938A CN 202110190388 A CN202110190388 A CN 202110190388A CN 112923938 A CN112923938 A CN 112923938A
- Authority
- CN
- China
- Prior art keywords
- laser data
- frame
- data frame
- environment
- information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
- G01C21/00—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
- G01C21/26—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 specially adapted for navigation in a road network
- G01C21/28—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 specially adapted for navigation in a road network with correlation of data from several navigational instruments
- G01C21/30—Map- or contour-matching
- G01C21/32—Structuring or formatting of map data
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Landscapes
- Engineering & Computer Science (AREA)
- Radar, Positioning & Navigation (AREA)
- Remote Sensing (AREA)
- Automation & Control Theory (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Image Analysis (AREA)
- Traffic Control Systems (AREA)
Abstract
The invention discloses a map optimization method, a map optimization device, a storage medium and a map optimization system, wherein in the process of constructing a map based on laser data, first environment geometric information of a current laser data frame is determined; comparing the first environment geometric information with each second environment geometric information of the current map, and determining at least one scene candidate frame from each laser data frame for constructing the current map; matching first environment semantic information of a current laser data frame with second environment semantic information of each scene candidate frame, and determining a scene recognition frame from the scene candidate frames; and calculating the relative pose of the current laser data frame and the scene recognition frame, and adjusting the relative pose according to the matching result in the process of matching the structural feature points of the current laser data frame with the structural feature points of the scene recognition frame. By the scheme, robustness and accuracy of scene identification in the map construction process can be effectively guaranteed, and accurate positioning of the vehicle based on the optimized map is facilitated.
Description
Technical Field
The embodiment of the invention relates to the technical field of vehicle control, in particular to a map optimization method, a map optimization device, a map optimization storage medium and a map optimization system.
Background
In the field of unmanned intelligent driving at present, a robust and accurate positioning system is indispensable, and in L4-level intelligent driving, laser mapping positioning occupies an important position.
In the traditional method laser mapping system, the laser mapping system inevitably generates accumulated errors along with the accumulation of time; in a positioning system, a traditional method generally completes initialization of the positioning system through a GPS signal or coordinates of a last parking position recorded by the system, and in L4 intelligent driving, the GPS signal is limited by various scenes such as high buildings, high frames, shade streets, underground and the like, and a vehicle system cannot record the last parking position every time. Therefore, it is desirable to provide a robust and accurate scene recognition method.
In the related technology, the scene recognition of the laser mapping positioning system is divided into two parts, firstly, in the mapping process, when an unmanned automobile returns to the original position, the automobile matches the constructed partial map by sensing the surrounding environment information, and finds that the automobile returns to the original passing place; and secondly, in the positioning process, the unmanned automobile can quickly find the position of the unmanned automobile in the map by sensing surrounding information and matching with the map through sensing the surrounding information. Obviously, correct and rapid scene identification can be identified when the vehicle returns to the original passing place in the process of map construction, and the accumulated error caused by long-time operation is reduced in a pose map optimization mode; in the positioning process, the requirements for GPS signals and initial point poses can be reduced, and the use scenes and robustness guarantee of the algorithm are greatly increased. However, the wrong scene identification is carried out, wrong pose map optimization is carried out during map building, map deformation is easily caused, and a wrong environment map is constructed; when positioning, an incorrect initial pose can be directly given, and positioning failure is caused.
Disclosure of Invention
The invention provides a map optimization method, a map optimization device, a storage medium and a map optimization system, which can effectively ensure the robustness and accuracy of scene identification in the map construction process and are beneficial to accurate positioning of a vehicle based on an optimized map.
In a first aspect, an embodiment of the present invention provides a map optimization method, including:
in the process of constructing a map based on laser data, determining first environment geometric information of a current laser data frame;
comparing the first environment geometric information with each second environment geometric information of the current map, and determining at least one scene candidate frame from each laser data frame for constructing the current map; the second environment geometric information is environment geometric data corresponding to each laser data frame for constructing the current map;
determining first environment semantic information of the current laser data frame, matching the first environment semantic information with second environment semantic information of each scene candidate frame, and determining a scene identification frame from the scene candidate frames;
and calculating the relative pose of the current laser data frame and the scene recognition frame, and adjusting the relative pose according to a matching result in the process of matching the structural feature points of the current laser data frame and the structural feature points of the scene recognition frame so as to optimize the current map and generate a target map.
In a second aspect, an embodiment of the present invention further provides a map optimizing apparatus, including:
the environment information determining module is used for determining first environment geometric information of a current laser data frame in the process of constructing a map based on laser data;
a scene candidate frame determining module, configured to compare the first environment geometric information with each second environment geometric information of the current map, and determine at least one scene candidate frame from each laser data frame used for constructing the current map; the second environment geometric information is environment geometric data corresponding to each laser data frame for constructing the current map;
a scene recognition frame determining module, configured to determine first environment semantic information of the current laser data frame, match the first environment semantic information with second environment semantic information of each scene candidate frame, and determine a scene recognition frame from the scene candidate frames;
and the map optimization module is used for calculating the relative pose of the current laser data frame and the scene recognition frame, and adjusting the relative pose according to a matching result in the process of matching the structural feature point of the current laser data frame with the structural feature point of the scene recognition frame so as to optimize the current map and generate a target map.
In a third aspect, an embodiment of the present invention further provides a computer storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the map optimization method provided in any embodiment of the present invention.
In a fourth aspect, an embodiment of the present invention provides a map optimization system, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, the map optimization method according to an embodiment of the present invention is implemented.
The map optimization scheme provided by the invention comprises the following steps: in the process of constructing a map based on laser data, determining first environment geometric information of a current laser data frame; comparing the first environment geometric information with each second environment geometric information of the current map, and determining at least one scene candidate frame from each laser data frame for constructing the current map; the second environment geometric information is environment geometric data corresponding to each laser data frame for constructing the current map; determining first environment semantic information of the current laser data frame, matching the first environment semantic information with second environment semantic information of each scene candidate frame, and determining a scene identification frame from the scene candidate frames; and calculating the relative pose of the current laser data frame and the scene recognition frame, and adjusting the relative pose according to a matching result in the process of matching the structural feature points of the current laser data frame and the structural feature points of the scene recognition frame so as to optimize the current map and generate a target map. According to the technical scheme provided by the embodiment of the invention, the robustness and the accuracy of scene identification in the map construction process can be effectively ensured, and the vehicle can be accurately positioned based on the optimized map.
Drawings
Fig. 1 is a schematic flowchart of a map optimization method according to an embodiment of the present invention;
fig. 2 is a schematic flowchart of another map optimization method according to an embodiment of the present invention;
fig. 3 is a schematic flowchart of another map optimization method according to an embodiment of the present invention;
fig. 4 is a block diagram of a map optimization apparatus according to an embodiment of the present invention;
fig. 5 is a block diagram of a map optimization system according to an embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be further noted that, for the convenience of description, only some of the structures related to the present invention are shown in the drawings, not all of the structures.
Fig. 1 is a flowchart of a map optimization method according to an embodiment of the present invention, which may be implemented by a map optimization device, where the map optimization device may be implemented by software and/or hardware, and may be generally integrated in a map optimization system, and the map optimization system may be generally configured in a vehicle. As shown in fig. 1, the method includes:
The laser data is point cloud data acquired by a laser radar. In the embodiment of the invention, a map can be constructed based on a large amount of continuously collected laser data, and in the process of constructing the map based on the laser data, first environment geometric information of a current laser data frame is determined, wherein the first environment geometric information reflects geometric structure information of the current laser data frame.
Optionally, determining the first environment geometry information of the current laser data frame includes: extracting structural feature points of the current laser data frame; constructing a structure descriptor corresponding to the current laser data frame based on three-dimensional geometrical information among the structural feature points of the current laser data frame; wherein the structure descriptor is used for indicating the structure information of the structure characteristic point of the current laser data frame; constructing a bag of words for the current laser data frame based on the structural descriptor; and taking the structure descriptor and the word bag as first environment geometric information of the current laser data frame. Specifically, structural feature points are extracted from the current laser data frame, and a three-dimensional geometric structure formed by all the structural feature points is determined, wherein the three-dimensional geometric structure formed by all the structural feature points may include a planar structure and a stereo structure. And constructing a structure descriptor corresponding to the current laser data frame based on a three-dimensional geometrical structure (namely three-dimensional geometrical information) among the structural feature points of the current laser data frame, wherein the structure descriptor can reflect the structural information of the structural feature points of the current laser data frame. For example, the structure descriptor may be represented in a sequence, where the sequence includes attribute information of each structural feature point in the three-dimensional geometric structure represented by all the structural feature points, such as information of a plane point or a corner point. And constructing a word bag corresponding to the current laser data frame based on the structure descriptor, wherein the word bag can reflect the structure types presented by all the structure characteristic points in the current laser data frame. And taking the structure descriptor and the word bag as first environment geometric information of the current laser data frame. It is understood that the bag of words in the first environment geometry information may roughly reflect the structural class of the current laser data frame, and the structural descriptor may reflect the structural information of each structural feature point in the current laser data frame in more detail.
102, comparing the first environment geometric information with each second environment geometric information of the current map, and determining at least one scene candidate frame from each laser data frame for constructing the current map; and the second environment geometric information is environment geometric data corresponding to each laser data frame for constructing the current map.
In an embodiment of the present invention, the current map may be understood as a map constructed based on at least a previous laser data frame of the current laser data frame. And acquiring second environment geometric information corresponding to each laser data frame for constructing the current map, wherein the second environment geometric information comprises a structure descriptor corresponding to each laser data frame and a bag of words constructed based on the structure descriptor. And comparing the first environment geometric information of the current laser data frame with each second environment geometric information of the current map, and determining at least one scene candidate frame from each laser data frame for constructing the current map. Specifically, the word bag in the first environment geometric information may be respectively matched with the word bag in the second environment geometric information, a target word bag whose word bag matching degree is greater than a first matching degree threshold value is determined from the second environment geometric information, the second environment geometric information to which the target word bag belongs is taken as the first target environment geometric information, then the structure descriptor in the first environment geometric information is matched with the structure descriptor in the first target environment geometric information, the target structure descriptor whose descriptor matching degree is greater than a second matching degree threshold value is determined from the first target environment geometric information, the first target environment geometric information to which the target structure descriptor belongs is taken as the second target environment geometric information, and then the laser data frame corresponding to the second target environment geometric information is determined as the scene candidate frame. It can be understood that the scene candidate frame is a laser data frame with higher similarity of the environmental geometric information of the current map and the current laser data frame.
In the embodiment of the invention, first environment semantic information of the current laser data frame is determined, wherein the first environment semantic information may include semantic information of the current laser data frame and three-dimensional coordinate information of the semantic information in the current laser data. Specifically, the semantic information of the current laser data frame may be determined based on a deep learning manner, for example, the semantic information of the current laser data frame may be determined by an image recognition manner, and for example, the semantic information may include information such as a signboard and a ground identifier. And determining the three-dimensional coordinate information of the semantic information in the current laser data frame, if the semantic information is a signboard, namely the current laser data frame is image data containing the signboard, determining the three-dimensional coordinate information of the signboard in the current laser data frame. Similarly, second environment semantic information of each environment candidate frame is determined, where the second environment semantic information may include semantic information of the scene candidate frame and three-dimensional coordinate information of the semantic information in the scene candidate frame.
In the embodiment of the invention, the first environment semantic information of the current laser data frame is matched with the second environment semantic information of each scene candidate frame, and the scene candidate frame of which the matching degree of the environment semantic information is greater than a third preset matching degree threshold value is determined as the scene recognition frame. Specifically, the semantic information in the first environment semantic information may be matched with the semantics in the second environment semantic information, respectively, the target semantic information whose semantic matching degree is greater than the fourth matching degree threshold value may be determined from the second environment semantic information, and the second environment semantic information to which the target semantic information belongs is taken as the first target environment semantic information, then matching the three-dimensional coordinate information in the first environment semantic information with the three-dimensional coordinate information in the first target environment semantic information, determining target three-dimensional coordinate information with the three-dimensional coordinate matching degree larger than a fifth matching degree threshold value from the first target environment semantic information, and the semantic information of the first target environment to which the target three-dimensional coordinate information belongs is taken as the semantic information of the second target environment, and then determining the scene candidate frame corresponding to the semantic information of the second target environment as a scene recognition frame. It can be understood that the scene identification frame is a laser data frame with higher similarity with the environmental geometric information and the environmental semantic information of the current laser data frame in the current map.
And 104, calculating the relative pose of the current laser data frame and the scene recognition frame, and adjusting the relative pose according to a matching result in the process of matching the structural feature point of the current laser data frame with the structural feature point of the scene recognition frame so as to optimize the current map and generate a target map.
In the embodiment of the invention, the relative pose of the current laser data frame and the scene recognition frame is calculated according to the three-dimensional coordinate information in the first environment semantic information of the current laser data frame and the three-dimensional coordinate information in the environment semantic information of the scene recognition frame. And matching the structural feature points of the current laser data frame with the structural feature points of the scene recognition frame, and adjusting the relative pose of the current laser data frame relative to the scene recognition frame according to the matching result in the matching process, so as to complete the optimization of the current map, and taking the optimized current map as a target map.
In the embodiment of the invention, for each laser data frame for constructing the map, the map can be continuously optimized by the technical scheme, and the accumulated error of the constructed map can be effectively reduced.
The map optimization method provided by the invention is characterized in that in the process of constructing a map based on laser data, first environment geometric information of a current laser data frame is determined; comparing the first environment geometric information with each second environment geometric information of the current map, and determining at least one scene candidate frame from each laser data frame for constructing the current map; the second environment geometric information is environment geometric data corresponding to each laser data frame for constructing the current map; determining first environment semantic information of the current laser data frame, matching the first environment semantic information with second environment semantic information of each scene candidate frame, and determining a scene identification frame from the scene candidate frames; and calculating the relative pose of the current laser data frame and the scene recognition frame, and adjusting the relative pose according to a matching result in the process of matching the structural feature points of the current laser data frame and the structural feature points of the scene recognition frame so as to optimize the current map and generate a target map. According to the technical scheme provided by the embodiment of the invention, the robustness and the accuracy of scene identification in the map construction process can be effectively ensured, and the vehicle can be accurately positioned based on the optimized map.
In some embodiments, before comparing the first environment geometric information with each second environment geometric information of the current map, the method further includes: determining first position information and first acquisition time of a current laser data frame; determining second position information and second acquisition time of a last laser data frame of the current laser data frame; the last laser data frame is used for constructing the current map; calculating the distance between the current laser data frame and the corresponding position of the last laser data frame based on the first position information and the second position information, and calculating the acquisition time difference between the current laser data frame and the last laser data frame based on the first acquisition time and the second acquisition time; comparing the first environment geometric information with each second environment geometric information of the current map, including: and when the distance is greater than a preset distance threshold value and the acquisition time difference is greater than a preset time difference, comparing the first environment geometric information with each second environment geometric information of the current map. The advantage of this arrangement is that the efficiency of map optimization can be effectively improved.
In the embodiment of the invention, when the distance between the corresponding positions of the current laser data frame and the last laser data frame is very close or the acquisition time point is very close, the current laser data frame is not likely to be the data returning to the origin, and at this time, the loop optimization can be performed on the currently constructed map no matter the current laser data frame. For example, the first position information and the first collecting time of the current laser data frame are determined, and the second position information and the second collecting time of the previous laser data frame of the current laser data frame are determined, for example, the first position information corresponding to the current laser data frame and the second position information corresponding to the previous laser data frame may be determined by an image recognition technology. And calculating the distance between the current laser data frame and the position corresponding to the last laser data frame according to the first position information and the second position information, wherein the distance can be a Euclidean distance. And calculating the acquisition time difference between the current laser data frame and the previous laser data frame based on the first acquisition time and the second acquisition time. And when the distance is greater than a preset distance threshold value and the acquisition time difference is greater than a preset time difference, comparing the first environment geometric information with each second environment geometric information of the current map. And when the distance is smaller than the preset distance threshold or the acquisition time difference is smaller than the preset time difference, the current map can not be optimized based on the current laser data frame.
In some embodiments, after determining at least one scene candidate frame from among the frames of laser data used to construct the current map, further comprising: determining the continuity of the scene candidate frames, and eliminating discontinuous single frames in the scene candidate frames according to the continuity so as to update the scene candidate frames; comparing the third environmental geometric information of at least one laser data frame before the current laser data frame and/or at least one laser data frame after the current laser data frame with the fourth environmental geometric information of the updated scene candidate frame, and taking the scene candidate frame of which the matching degree of the environmental geometric information is greater than a preset matching degree threshold value as a target candidate frame; correspondingly, determining first environment semantic information of the current laser data frame, matching the first environment semantic information with second environment semantic information of each scene candidate frame, and determining a scene identification frame from the scene candidate frames, including: and determining first environment semantic information of the current laser data frame, matching the first environment semantic information with third environment semantic information of each target candidate frame, and determining a scene recognition frame from the target candidate frames. The advantage of this arrangement is that scene candidate frames with a high degree of matching with the environmental geometric information of the current laser data frame can be more accurately determined from the laser data frames used for constructing the current map.
In the embodiment of the invention, if the scene is the same scene, the scene candidate frames similar to the current laser data frame are continuous definitely. Therefore, the continuity of the scene candidate frames is determined, and discontinuous single frames in the scene candidate frames are eliminated according to the continuity so as to update the scene candidate frames. For example, the scene candidate frames are the 6 th frame laser data, the 7 th frame laser data, the 8 th frame laser data, the 9 th frame laser data and the 13 th frame laser data, and obviously, the 13 th frame laser data is a single frame data discontinuous from other scene candidate frames, so the 13 th frame laser data can be deleted from the scene candidate frames, and the updated scene candidate frames include: frame 6 laser data, frame 7 laser data, frame 8 laser data, and frame 9 laser data. And determining third environment geometrical information of at least a previous laser data frame of the current laser data frame and/or at least a next laser data frame of the current laser data frame, and determining fourth environment geometrical information of the updated scene candidate frame. The third environment geometric information and the fourth environment geometric information may include corresponding structure descriptors and word bags. And comparing the third environment geometric information with the fourth environment geometric information of the updated scene candidate frame, and taking the scene candidate frame of which the matching degree of the environment geometric information is greater than a preset matching degree threshold value as a target candidate frame. For example, in the updated scene candidate frames (the 6 th frame laser data, the 7 th frame laser data, the 8 th frame laser data, and the 9 th frame laser data), the scene candidate frame whose matching degree of the environmental geometric information is greater than the preset matching degree threshold includes the 7 th frame laser data and the 8 th frame laser data, and then the 7 th frame laser data and the 8 th frame laser data are used as target candidate frames. And then, matching the first environment semantic information of the current laser data frame with the environment semantic information of the target candidate frame, and further determining a scene recognition frame from the target candidate frame.
In some embodiments, after generating the target map, further comprising: acquiring a target laser data frame acquired by a vehicle, and determining fourth ring geometric information of the target laser data frame; comparing the fourth environmental geometric information with each fifth environmental geometric information of the target map, and determining at least one scene recognition candidate frame from each laser data frame for constructing the target map; the fifth environment geometric information is environment geometric data corresponding to each laser data frame for constructing the target map; determining fourth environment semantic information of the target laser data frame, matching the fourth environment semantic information with fifth environment semantic information of each scene recognition candidate frame, and determining a target scene recognition frame from the scene recognition candidate frames; and calculating a rough initial pose of the target laser data frame relative to the target scene identification frame, and adjusting the rough initial pose according to a matching result in the process of matching the structural feature points of the target laser data frame with the structural feature points of the target scene identification frame to determine the target initial pose of the vehicle.
In the embodiment of the present invention, a target laser data frame collected by a vehicle is obtained, wherein the target laser data frame may be collected by a laser radar configured on the vehicle. And determining fourth environment geometric information of the target laser data frame, wherein the fourth environment geometric information may include a corresponding structure descriptor and a corresponding bag of words, and a determination manner of the fourth environment geometric information is the same as that of the first environment geometric information, and is not described herein again. And comparing the fourth environmental geometric information with fifth environmental geometric information corresponding to each laser data frame for constructing the target map, and determining the laser data frame with the environmental geometric information larger than a preset threshold value from each laser data frame for constructing the target map as a scene recognition candidate frame. It can be understood that the scene recognition candidate frame is a data frame with a greater similarity to the environmental geometric information of the target laser data frame, among the laser data frames used for constructing the target map. And then further determining fourth environment semantic information of the target laser data and fifth environment semantic information of the scene candidate recognition frame. The fourth environment semantic information and the fifth environment semantic information may include semantic information of the corresponding laser data frame and three-dimensional coordinate information of the semantic information in the corresponding laser data frame. And matching the fourth environment semantic information of the target laser data frame with the fifth environment semantic information of each scene recognition candidate frame, and determining the data frame of which the environment semantic information is greater than a preset threshold value in the scene recognition candidate frames as the target scene recognition frame. It can be understood that the target scene identification frame is a data frame with greater similarity to the environmental geometric information and the environmental semantic information of the target laser data frame in each laser data frame used for constructing the target map. And calculating the rough initial pose of the target laser data frame relative to the target scene recognition frame according to the three-dimensional coordinate information in the fourth environment semantic information of the target laser data frame and the three-dimensional coordinate information in the environment semantic information of the target scene recognition frame. And in the process of matching the structural feature points of the target laser data frame with the structural feature points of the target scene recognition frame, adjusting the rough initial pose in real time according to the matching result, and taking the adjusted rough initial pose as the target initial pose of the vehicle.
In some embodiments, before acquiring the target laser data frame collected by the vehicle, the method further comprises: after a positioning function of the vehicle is started, judging whether real-time dynamic positioning (RTK) information exists or not; acquiring a target laser data frame acquired by a vehicle, comprising: and when the RTK information does not exist, acquiring a target laser data frame acquired by the vehicle. Optionally, when the RTK information exists, the RTK information is used as the target initial pose of the vehicle.
Fig. 2 is a schematic flow chart of another map optimization method according to an embodiment of the present invention, as shown in fig. 2, the method includes the following steps:
And step 203, constructing a bag of words of the current laser data frame based on the structure descriptor.
And step 204, taking the structure descriptor and the word bag as first environment geometric information of the current laser data frame.
Optionally, before comparing the first environment geometric information with each second environment geometric information of the current map, the method further includes: determining first position information and first acquisition time of a current laser data frame; determining second position information and second acquisition time of a last laser data frame of the current laser data frame; the last laser data frame is used for constructing the current map; calculating the distance between the current laser data frame and the corresponding position of the last laser data frame based on the first position information and the second position information, and calculating the acquisition time difference between the current laser data frame and the last laser data frame based on the first acquisition time and the second acquisition time; comparing the first environment geometric information with each second environment geometric information of the current map, including: and when the distance is greater than a preset distance threshold value and the acquisition time difference is greater than a preset time difference, comparing the first environment geometric information with each second environment geometric information of the current map.
And step 207, comparing the third environmental geometric information of at least one laser data frame before the current laser data frame and/or at least one laser data frame after the current laser data frame with the fourth environmental geometric information of the updated scene candidate frame, and taking the scene candidate frame of which the matching degree of the environmental geometric information is greater than a preset matching degree threshold value as the target candidate frame.
And 208, determining first environment semantic information of the current laser data frame, matching the first environment semantic information with third environment semantic information of each target candidate frame, and determining a scene recognition frame from the target candidate frames.
And 209, calculating the relative pose of the current laser data frame and the scene recognition frame, and adjusting the relative pose according to a matching result in the process of matching the structural feature point of the current laser data frame with the structural feature point of the scene recognition frame so as to optimize the current map and generate a target map.
Optionally, the environment semantic information includes semantic information of a laser data frame and three-dimensional coordinate information corresponding to the semantic information of the laser data frame.
The map optimization scheme provided by the embodiment of the invention can effectively ensure the robustness and accuracy of scene identification in the map construction process, and is beneficial to accurately positioning a vehicle based on an optimized map.
Fig. 3 is a schematic flow chart of another map optimization method according to an embodiment of the present invention, as shown in fig. 3, the method includes the following steps:
And step 303, determining the continuity of the scene candidate frames, and eliminating discontinuous single frames in the scene candidate frames according to the continuity so as to update the scene candidate frames.
And 305, determining first environment semantic information of the current laser data frame, matching the first environment semantic information with third environment semantic information of each target candidate frame, and determining a scene recognition frame from the target candidate frames.
And step 306, calculating the relative pose of the current laser data frame and the scene recognition frame, and adjusting the relative pose according to the matching result in the process of matching the structural feature points of the current laser data frame and the structural feature points of the scene recognition frame so as to optimize the current map and generate the target map.
And 308, acquiring a target laser data frame acquired by the vehicle, and determining fourth ring geometric information of the target laser data frame.
309, comparing the fourth environment geometric information with each fifth environment geometric information of the target map, and determining at least one scene recognition candidate frame from each laser data frame for constructing the target map; and the fifth environment geometric information is environment geometric data corresponding to each laser data frame for constructing the target map.
And 310, determining fourth environment semantic information of the target laser data frame, matching the fourth environment semantic information with fifth environment semantic information of each scene recognition candidate frame, and determining the target scene recognition frame from the scene recognition candidate frames.
And 311, calculating a rough initial pose of the target laser data frame relative to the target scene recognition frame, and adjusting the rough initial pose according to a matching result to determine the target initial pose of the vehicle in the process of matching the structural feature points of the target laser data frame with the structural feature points of the target scene recognition frame.
And step 312, taking the RTK information as the initial pose of the target of the vehicle.
Optionally, the environment semantic information includes semantic information of a laser data frame and three-dimensional coordinate information corresponding to the semantic information of the laser data frame.
The map optimization scheme provided by the embodiment of the invention not only can effectively ensure the robustness and accuracy of scene recognition in the map construction process, but also can effectively ensure the accuracy of scene recognition of a vehicle based on an optimized map.
Fig. 4 is a block diagram of a map optimization apparatus, which may be implemented by software and/or hardware, generally integrated in a map optimization system, and may analyze a vehicle emergency event by performing a map optimization method according to an embodiment of the present invention. As shown in fig. 4, the apparatus includes:
the environment information determining module 401 is configured to determine first environment geometric information of a current laser data frame in a process of constructing a map based on laser data;
a scene candidate frame determining module 402, configured to compare the first environment geometric information with each second environment geometric information of the current map, and determine at least one scene candidate frame from each laser data frame used for constructing the current map; the second environment geometric information is environment geometric data corresponding to each laser data frame for constructing the current map;
a scene identification frame determining module 403, configured to determine first environment semantic information of the current laser data frame, match the first environment semantic information with second environment semantic information of each scene candidate frame, and determine a scene identification frame from the scene candidate frames;
and a map optimization module 404, configured to calculate a relative pose of the current laser data frame and the scene recognition frame, and adjust the relative pose according to a matching result in a process of matching the structural feature point of the current laser data frame with the structural feature point of the scene recognition frame, so as to optimize the current map and generate a target map.
The map optimization device provided by the invention determines the first environment geometric information of the current laser data frame in the process of constructing the map based on the laser data; comparing the first environment geometric information with each second environment geometric information of the current map, and determining at least one scene candidate frame from each laser data frame for constructing the current map; the second environment geometric information is environment geometric data corresponding to each laser data frame for constructing the current map; determining first environment semantic information of the current laser data frame, matching the first environment semantic information with second environment semantic information of each scene candidate frame, and determining a scene identification frame from the scene candidate frames; and calculating the relative pose of the current laser data frame and the scene recognition frame, and adjusting the relative pose according to a matching result in the process of matching the structural feature points of the current laser data frame and the structural feature points of the scene recognition frame so as to optimize the current map and generate a target map. According to the technical scheme provided by the embodiment of the invention, the robustness and the accuracy of scene identification in the map construction process can be effectively ensured, and the vehicle can be accurately positioned based on the optimized map.
Optionally, the apparatus further comprises:
the first information determining module is used for determining first position information and first acquisition time of a current laser data frame before comparing the first environment geometric information with each second environment geometric information of a current map;
the second information determining module is used for determining second position information and second acquisition time of a last laser data frame of the current laser data frame; the last laser data frame is used for constructing the current map;
a relative information calculation module, configured to calculate a distance between a corresponding position of the current laser data frame and a corresponding position of the previous laser data frame based on the first position information and the second position information, and calculate an acquisition time difference between the current laser data frame and the previous laser data frame based on the first acquisition time and the second acquisition time;
the scene candidate frame determination module is configured to:
and when the distance is greater than a preset distance threshold value and the acquisition time difference is greater than a preset time difference, comparing the first environment geometric information with each second environment geometric information of the current map.
Optionally, the environment information determining module is configured to:
extracting structural feature points of the current laser data frame;
constructing a structure descriptor corresponding to the current laser data frame based on three-dimensional geometrical information among the structural feature points of the current laser data frame; wherein the structure descriptor is used for indicating the structure information of the structure characteristic point of the current laser data frame;
constructing a bag of words for the current laser data frame based on the structural descriptor;
and taking the structure descriptor and the word bag as first environment geometric information of the current laser data frame.
Optionally, the apparatus further comprises:
the continuity determining module is used for determining the continuity of the scene candidate frames after at least one scene candidate frame is determined from all laser data frames used for constructing the current map, and removing discontinuous single frames in the scene candidate frames according to the continuity so as to update the scene candidate frames;
the target candidate frame determining module is used for comparing the third environment geometric information of at least one previous laser data frame of the current laser data frame and/or at least one next laser data frame of the current laser data frame with the fourth environment geometric information of the updated scene candidate frame, and taking the scene candidate frame of which the matching degree of the environment geometric information is greater than a preset matching degree threshold value as the target candidate frame;
accordingly, the scene recognition frame determination module is configured to:
and determining first environment semantic information of the current laser data frame, matching the first environment semantic information with third environment semantic information of each target candidate frame, and determining a scene recognition frame from the target candidate frames.
Optionally, the apparatus further comprises:
the target laser data acquisition module is used for acquiring a target laser data frame acquired by a vehicle after a target map is generated and determining fourth ring geometric information of the target laser data frame;
a scene recognition candidate frame determination module, configured to compare the fourth environmental geometric information with each fifth environmental geometric information of the target map, and determine at least one scene recognition candidate frame from each laser data frame used for constructing the target map; the fifth environment geometric information is environment geometric data corresponding to each laser data frame for constructing the target map;
the target scene recognition frame determining module is used for determining fourth environment semantic information of the target laser data frame, matching the fourth environment semantic information with fifth environment semantic information of each scene recognition candidate frame, and determining a target scene recognition frame from the scene recognition candidate frames;
and the initial pose determining module is used for calculating a rough initial pose of the target laser data frame relative to the target scene identification frame, adjusting the rough initial pose according to a matching result in the process of matching the structural feature points of the target laser data frame with the structural feature points of the target scene identification frame, and determining the target initial pose of the vehicle.
Optionally, the apparatus further comprises:
the system comprises a judging module, a processing module and a processing module, wherein the judging module is used for judging whether real-time dynamic positioning RTK information exists or not before a target laser data frame acquired by a vehicle is acquired and after a positioning function of the vehicle is started;
the target laser data acquisition module is configured to:
and when the RTK information does not exist, acquiring a target laser data frame acquired by the vehicle.
Optionally, the environment semantic information includes semantic information of a laser data frame and three-dimensional coordinate information corresponding to the semantic information of the laser data frame.
Embodiments of the present invention also provide a storage medium containing computer-executable instructions, which when executed by a computer processor, perform a map optimization method, the method including:
in the process of constructing a map based on laser data, determining first environment geometric information of a current laser data frame;
comparing the first environment geometric information with each second environment geometric information of the current map, and determining at least one scene candidate frame from each laser data frame for constructing the current map; the second environment geometric information is environment geometric data corresponding to each laser data frame for constructing the current map;
determining first environment semantic information of the current laser data frame, matching the first environment semantic information with second environment semantic information of each scene candidate frame, and determining a scene identification frame from the scene candidate frames;
and calculating the relative pose of the current laser data frame and the scene recognition frame, and adjusting the relative pose according to a matching result in the process of matching the structural feature points of the current laser data frame and the structural feature points of the scene recognition frame so as to optimize the current map and generate a target map.
Storage medium-any of various types of memory devices or storage devices. The term "storage medium" is intended to include: mounting media such as CD-ROM, floppy disk, or tape devices; computer system memory or random access memory such as DRAM, DDRRAM, SRAM, EDORAM, Lanbas (Rambus) RAM, etc.; non-volatile memory such as flash memory, magnetic media (e.g., hard disk or optical storage); registers or other similar types of memory elements, etc. The storage medium may also include other types of memory or combinations thereof. In addition, the storage medium may be located in a first computer system in which the program is executed, or may be located in a different second computer system connected to the first computer system through a network (such as the internet). The second computer system may provide program instructions to the first computer for execution. The term "storage medium" may include two or more storage media that may reside in different locations, such as in different computer systems that are connected by a network. The storage medium may store program instructions (e.g., embodied as a computer program) that are executable by one or more processors.
Of course, the storage medium containing the computer-executable instructions provided by the embodiments of the present invention is not limited to the map optimization operation described above, and may also perform related operations in the map optimization method provided by any embodiments of the present invention.
The embodiment of the invention provides a map optimization system, and a map optimization device provided by the embodiment of the invention can be integrated in the map optimization system. Fig. 5 is a block diagram of a map optimization system according to an embodiment of the present invention. The map optimization system 500 may include: a memory 501, a processor 502 and a computer program stored on the memory 501 and executable on the processor, wherein the processor 502 implements the map optimization method according to the embodiment of the present invention when executing the computer program.
The map optimization system provided by the embodiment of the invention determines the first environment geometric information of the current laser data frame in the process of constructing the map based on the laser data; comparing the first environment geometric information with each second environment geometric information of the current map, and determining at least one scene candidate frame from each laser data frame for constructing the current map; the second environment geometric information is environment geometric data corresponding to each laser data frame for constructing the current map; determining first environment semantic information of the current laser data frame, matching the first environment semantic information with second environment semantic information of each scene candidate frame, and determining a scene identification frame from the scene candidate frames; and calculating the relative pose of the current laser data frame and the scene recognition frame, and adjusting the relative pose according to a matching result in the process of matching the structural feature points of the current laser data frame and the structural feature points of the scene recognition frame so as to optimize the current map and generate a target map. According to the technical scheme provided by the embodiment of the invention, the robustness and the accuracy of scene identification in the map construction process can be effectively ensured, and the vehicle can be accurately positioned based on the optimized map.
The map optimization device, the storage medium and the system provided in the above embodiments can execute the map optimization method provided in any embodiment of the present invention, and have corresponding functional modules and beneficial effects for executing the method. For technical details that are not described in detail in the above embodiments, reference may be made to a map optimization method provided in any embodiment of the present invention.
It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims (10)
1. A map optimization method, comprising:
in the process of constructing a map based on laser data, determining first environment geometric information of a current laser data frame;
comparing the first environment geometric information with each second environment geometric information of the current map, and determining at least one scene candidate frame from each laser data frame for constructing the current map; the second environment geometric information is environment geometric data corresponding to each laser data frame for constructing the current map;
determining first environment semantic information of the current laser data frame, matching the first environment semantic information with second environment semantic information of each scene candidate frame, and determining a scene identification frame from the scene candidate frames;
and calculating the relative pose of the current laser data frame and the scene recognition frame, and adjusting the relative pose according to a matching result in the process of matching the structural feature points of the current laser data frame and the structural feature points of the scene recognition frame so as to optimize the current map and generate a target map.
2. The method of claim 1, prior to comparing the first environment geometry information with respective second environment geometry information of a current map, further comprising:
determining first position information and first acquisition time of a current laser data frame;
determining second position information and second acquisition time of a last laser data frame of the current laser data frame; the last laser data frame is used for constructing the current map;
calculating the distance between the current laser data frame and the corresponding position of the last laser data frame based on the first position information and the second position information, and calculating the acquisition time difference between the current laser data frame and the last laser data frame based on the first acquisition time and the second acquisition time;
comparing the first environment geometric information with each second environment geometric information of the current map, including:
and when the distance is greater than a preset distance threshold value and the acquisition time difference is greater than a preset time difference, comparing the first environment geometric information with each second environment geometric information of the current map.
3. The method of claim 1, wherein determining the first environmental geometry information for the current frame of laser data comprises:
extracting structural feature points of the current laser data frame;
constructing a structure descriptor corresponding to the current laser data frame based on three-dimensional geometrical information among the structural feature points of the current laser data frame; wherein the structure descriptor is used for indicating the structure information of the structure characteristic point of the current laser data frame;
constructing a bag of words for the current laser data frame based on the structural descriptor;
and taking the structure descriptor and the word bag as first environment geometric information of the current laser data frame.
4. The method of claim 1, further comprising, after determining at least one scene candidate frame from among the frames of laser data used to construct the current map:
determining the continuity of the scene candidate frames, and eliminating discontinuous single frames in the scene candidate frames according to the continuity so as to update the scene candidate frames;
comparing the third environmental geometric information of at least one laser data frame before the current laser data frame and/or at least one laser data frame after the current laser data frame with the fourth environmental geometric information of the updated scene candidate frame, and taking the scene candidate frame of which the matching degree of the environmental geometric information is greater than a preset matching degree threshold value as a target candidate frame;
correspondingly, determining first environment semantic information of the current laser data frame, matching the first environment semantic information with second environment semantic information of each scene candidate frame, and determining a scene identification frame from the scene candidate frames, including:
and determining first environment semantic information of the current laser data frame, matching the first environment semantic information with third environment semantic information of each target candidate frame, and determining a scene recognition frame from the target candidate frames.
5. The method of claim 1, after generating the target map, further comprising:
acquiring a target laser data frame acquired by a vehicle, and determining fourth ring geometric information of the target laser data frame;
comparing the fourth environmental geometric information with each fifth environmental geometric information of the target map, and determining at least one scene recognition candidate frame from each laser data frame for constructing the target map; the fifth environment geometric information is environment geometric data corresponding to each laser data frame for constructing the target map;
determining fourth environment semantic information of the target laser data frame, matching the fourth environment semantic information with fifth environment semantic information of each scene recognition candidate frame, and determining a target scene recognition frame from the scene recognition candidate frames;
and calculating a rough initial pose of the target laser data frame relative to the target scene identification frame, and adjusting the rough initial pose according to a matching result in the process of matching the structural feature points of the target laser data frame with the structural feature points of the target scene identification frame to determine the target initial pose of the vehicle.
6. The method of claim 5, further comprising, prior to acquiring the frame of target laser data acquired by the vehicle:
after the positioning function of the vehicle is started, judging whether real-time dynamic positioning RTK information exists or not;
acquiring a target laser data frame acquired by a vehicle, comprising:
and when the RTK information does not exist, acquiring a target laser data frame acquired by the vehicle.
7. The method according to any one of claims 1-6, wherein the environment semantic information comprises semantic information of a laser data frame and three-dimensional coordinate information corresponding to the semantic information of the laser data frame.
8. A map optimization apparatus, comprising:
the environment information determining module is used for determining first environment geometric information of a current laser data frame in the process of constructing a map based on laser data;
a scene candidate frame determining module, configured to compare the first environment geometric information with each second environment geometric information of the current map, and determine at least one scene candidate frame from each laser data frame used for constructing the current map; the second environment geometric information is environment geometric data corresponding to each laser data frame for constructing the current map;
a scene recognition frame determining module, configured to determine first environment semantic information of the current laser data frame, match the first environment semantic information with second environment semantic information of each scene candidate frame, and determine a scene recognition frame from the scene candidate frames;
and the map optimization module is used for calculating the relative pose of the current laser data frame and the scene recognition frame, and adjusting the relative pose according to a matching result in the process of matching the structural feature point of the current laser data frame with the structural feature point of the scene recognition frame so as to optimize the current map and generate a target map.
9. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the map optimization method according to any one of claims 1 to 7.
10. A map optimization system comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor, when executing the computer program, implements the map optimization method of any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110190388.6A CN112923938B (en) | 2021-02-18 | 2021-02-18 | Map optimization method, device, storage medium and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110190388.6A CN112923938B (en) | 2021-02-18 | 2021-02-18 | Map optimization method, device, storage medium and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112923938A true CN112923938A (en) | 2021-06-08 |
| CN112923938B CN112923938B (en) | 2023-10-03 |
Family
ID=76169918
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110190388.6A Active CN112923938B (en) | 2021-02-18 | 2021-02-18 | Map optimization method, device, storage medium and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112923938B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113607157A (en) * | 2021-08-04 | 2021-11-05 | 上海高仙自动化科技发展有限公司 | Self-body positioning method and device, self-body, chip and storage medium |
| CN114413881A (en) * | 2022-01-07 | 2022-04-29 | 中国第一汽车股份有限公司 | Method and device for constructing high-precision vector map and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108415032A (en) * | 2018-03-05 | 2018-08-17 | 中山大学 | A kind of point cloud semanteme map constructing method based on deep learning and laser radar |
| US20190011566A1 (en) * | 2017-07-04 | 2019-01-10 | Baidu Online Network Technology (Beijing) Co., Ltd. | Method and apparatus for identifying laser point cloud data of autonomous vehicle |
| CN111723173A (en) * | 2020-06-15 | 2020-09-29 | 中国第一汽车股份有限公司 | Vehicle-mounted map making method and device, electronic equipment and storage medium |
| CN111928862A (en) * | 2020-08-10 | 2020-11-13 | 廊坊和易生活网络科技股份有限公司 | Method for constructing semantic map on line by fusing laser radar and visual sensor |
| CN112305554A (en) * | 2020-11-23 | 2021-02-02 | 中国科学院自动化研究所 | Laser odometer method, system and device based on directed geometric points and sparse frames |
-
2021
- 2021-02-18 CN CN202110190388.6A patent/CN112923938B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20190011566A1 (en) * | 2017-07-04 | 2019-01-10 | Baidu Online Network Technology (Beijing) Co., Ltd. | Method and apparatus for identifying laser point cloud data of autonomous vehicle |
| CN108415032A (en) * | 2018-03-05 | 2018-08-17 | 中山大学 | A kind of point cloud semanteme map constructing method based on deep learning and laser radar |
| CN111723173A (en) * | 2020-06-15 | 2020-09-29 | 中国第一汽车股份有限公司 | Vehicle-mounted map making method and device, electronic equipment and storage medium |
| CN111928862A (en) * | 2020-08-10 | 2020-11-13 | 廊坊和易生活网络科技股份有限公司 | Method for constructing semantic map on line by fusing laser radar and visual sensor |
| CN112305554A (en) * | 2020-11-23 | 2021-02-02 | 中国科学院自动化研究所 | Laser odometer method, system and device based on directed geometric points and sparse frames |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113607157A (en) * | 2021-08-04 | 2021-11-05 | 上海高仙自动化科技发展有限公司 | Self-body positioning method and device, self-body, chip and storage medium |
| CN113607157B (en) * | 2021-08-04 | 2024-04-16 | 上海高仙自动化科技发展有限公司 | Autonomous positioning method, apparatus, autonomous body, chip and storage medium |
| CN114413881A (en) * | 2022-01-07 | 2022-04-29 | 中国第一汽车股份有限公司 | Method and device for constructing high-precision vector map and storage medium |
| CN114413881B (en) * | 2022-01-07 | 2023-09-01 | 中国第一汽车股份有限公司 | Construction method, device and storage medium of high-precision vector map |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112923938B (en) | 2023-10-03 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11042762B2 (en) | Sensor calibration method and device, computer device, medium, and vehicle | |
| CN107677279B (en) | Method and system for positioning and establishing image | |
| US20200005487A1 (en) | Positioning method and robot using the same | |
| JP6595182B2 (en) | Systems and methods for mapping, locating, and attitude correction | |
| CN110782494A (en) | Visual SLAM method based on point-line fusion | |
| WO2019062649A1 (en) | Adaptive region division method and system | |
| US20230108621A1 (en) | Method and system for generating visual feature map | |
| CN112923938B (en) | Map optimization method, device, storage medium and system | |
| CN111986261B (en) | Vehicle positioning method and device, electronic equipment and storage medium | |
| CN113936198A (en) | Low-beam laser radar and camera fusion method, storage medium and device | |
| US11830218B2 (en) | Visual-inertial localisation in an existing map | |
| US20220398854A1 (en) | Lane line determination method and system, vehicle, and storage medium | |
| GB2599947A (en) | Visual-inertial localisation in an existing map | |
| CN114088081B (en) | Map construction method for accurate positioning based on multistage joint optimization | |
| CN115638787B (en) | Digital map generation method, computer readable storage medium and electronic device | |
| CN115375870B (en) | Loop detection optimization method, electronic equipment and computer readable storage device | |
| GB2599948A (en) | Initialising keyframes for visual-inertial localisation and/or mapping | |
| CN113325389A (en) | Unmanned vehicle laser radar positioning method, system and storage medium | |
| CN116563376A (en) | LIDAR-IMU tight coupling semantic SLAM method based on deep learning and related device | |
| CN115493612A (en) | Vehicle positioning method and device based on visual SLAM | |
| CN115063550A (en) | Semantic point cloud map construction method and system and intelligent robot | |
| Achar et al. | Large scale visual localization in urban environments | |
| CN108932275B (en) | Bayesian methods for geospatial object/feature detection | |
| CN115908498B (en) | Multi-target tracking method and device based on category optimal matching | |
| CN113189610A (en) | Map-enhanced autonomous driving multi-target tracking method and related equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |