WO2022041706A1 - Positioning method, positioning system, and vehicle - Google Patents

Positioning method, positioning system, and vehicle Download PDF

Info

Publication number
WO2022041706A1
WO2022041706A1 PCT/CN2021/082388 CN2021082388W WO2022041706A1 WO 2022041706 A1 WO2022041706 A1 WO 2022041706A1 CN 2021082388 W CN2021082388 W CN 2021082388W WO 2022041706 A1 WO2022041706 A1 WO 2022041706A1
Authority
WO
WIPO (PCT)
Prior art keywords
semantic information
positioning
pose
weight
semantic
Prior art date
Application number
PCT/CN2021/082388
Other languages
French (fr)
Chinese (zh)
Inventor
薛常亮
俞彦辉
王化友
周彦兴
温丰
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Publication of WO2022041706A1 publication Critical patent/WO2022041706A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
    • G01S17/88—Lidar systems specially adapted for specific applications
    • G01S17/93—Lidar systems specially adapted for specific applications for anti-collision purposes
    • G01S17/931—Lidar systems specially adapted for specific applications for anti-collision purposes of land vehicles
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
    • G01S17/02—Systems using the reflection of electromagnetic waves other than radio waves
    • G01S17/06—Systems determining position data of a target
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
    • G01S17/86—Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
    • G01S17/88—Lidar systems specially adapted for specific applications
    • G01S17/89—Lidar systems specially adapted for specific applications for mapping or imaging
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
    • G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
    • G01C21/00—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
    • G01C21/26—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 specially adapted for navigation in a road network
    • G01C21/28—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 specially adapted for navigation in a road network with correlation of data from several navigational instruments
    • G01C21/30—Map- or contour-matching
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
    • G01C21/00—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
    • G01C21/26—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 specially adapted for navigation in a road network
    • G01C21/28—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 specially adapted for navigation in a road network with correlation of data from several navigational instruments
    • G01C21/30—Map- or contour-matching
    • G01C21/32—Structuring or formatting of map data

Definitions

  • the present application relates to the technical field of automatic driving, and in particular, to a positioning method, a positioning system and a vehicle.
  • Autonomous driving needs to address three core questions of driving: where? (vehicle location); where to go? (determine the destination); how to get there? (route plan).
  • positioning technology is mainly used to solve the problem of "where?", and is one of the key technologies that are essential to realize autonomous driving.
  • Signal-based positioning technology uses satellite signals or 5G signals to achieve vehicle positioning, so it has the ability to global positioning and is currently the most widely used positioning technology.
  • the GNSS satellite signal is easily affected by the occlusion of tall buildings and mountains, the signal-based positioning technology cannot provide accurate positioning when the vehicle is driving in cities, tunnels and other road conditions, so it cannot meet the needs of full-scene autonomous driving.
  • the current research direction in the field of autonomous driving is to integrate signal-based positioning technology, dead reckoning-based positioning technology, and feature matching-based positioning technology to make up for the shortcomings of signal-based positioning technology.
  • the laser positioning technology based on laser feature matching supplements or replaces the signal-based positioning technology or the visual positioning technology based on visual feature matching supplements or replaces the signal-based positioning technology.
  • An independent technical route, the map data that laser positioning and visual positioning rely on are stored separately, and the algorithms run independently, resulting in excessive data storage overhead and computational overhead, high requirements for the vehicle's control unit ECU and other hardware systems, and low efficiency.
  • the embodiments of the present application provide a positioning method, a positioning system, and a vehicle, which can integrate a laser positioning technology based on laser feature matching and a visual positioning technology based on visual feature matching, so as to improve positioning efficiency and reduce the occurrence of data and computational overhead.
  • an embodiment of the present application provides a positioning method, the method includes: determining an initial pose of a vehicle, and generating N sampling points C 1 to C N around the initial pose, where N is a positive integer;
  • the current predicted pose P extracts the first laser feature and at least one visual semantic information from the positioning map; for any sampling point C n , n is a positive integer, n ⁇ N, according to its corresponding current estimated pose P n , the extracted
  • the first laser feature of the self-localization map is matched with the second laser feature to determine the first weight of the current estimated pose P n , and the second laser feature is extracted from the point cloud data collected by the lidar; and, for At any sampling point C n , according to the current estimated pose P n , at least one visual semantic information is matched with at least one pavement semantic information to determine the second weight of the current estimated pose P n , and the at least one pavement semantic information is derived from It is extracted from the image data collected by the camera; according to the current estimated pose P 1
  • the technical solutions provided by the embodiments of the present application can respectively determine the first weight of the currently estimated pose based on the matching of laser features, and determine the current estimated pose based on the matching of visual semantic information and road semantic information (ie, matching of visual features).
  • the second weight and then calculate the weighted average of the current estimated pose according to the first weight and the second weight, and use the weighted average as the current pose of the vehicle, thereby realizing the laser positioning based on laser feature matching.
  • the technology is integrated with the visual positioning technology based on visual feature matching, which improves the positioning efficiency.
  • the technical solutions provided by the embodiments of the present application encode and store laser features and visual semantic information in the same positioning map, which realizes the fusion of map data and reduces the data overhead and computing overhead generated in the positioning process.
  • the positioning picture includes a first color channel, and the first color channel is used to store coding of visual semantic information.
  • the method provided by the embodiment of the present application can decode the first color channel to obtain visual semantic information.
  • the positioning picture further includes a second color channel, where the second color channel is used to store the code of the first laser feature.
  • the method provided by the embodiment of the present application can decode the second color channel to obtain the first laser feature.
  • the encoding of the visual semantic information includes at least one of a marker bit, a type encoding, and a brightness encoding; the marker bit is used to represent the type of the road marking, and the type encoding is used to represent the content of the road marking, Luma information encoding is used to represent the luma information of a picture.
  • the marker bit is used to represent the type of the road marking
  • the type encoding is used to represent the content of the road marking
  • Luma information encoding is used to represent the luma information of a picture.
  • At least one piece of visual semantic information is extracted by the following steps: extracting a local positioning map from the positioning map according to the current predicted pose, the local positioning map includes M positioning pictures, and the M positioning pictures include the current prediction The first picture where the pose is located, and M-1 second pictures near the first picture, where M is a positive integer greater than 1; at least one visual semantic information is extracted from the local positioning map.
  • the visual semantic information can include information such as road signs around the current predicted pose, so as to be matched with the road semantic information in the image data around the vehicle collected by the camera to determine the second weight.
  • At least one piece of visual semantic information is matched with at least one piece of road semantic information, so as to determine the first position of the current estimated pose P n
  • Two weights including: determining at least one valid pavement semantic information from at least one pavement semantic information, and the number of pixels of each valid pavement semantic information is within a preset range; Projecting into the coordinate system of the local positioning map; determining the semantic relationship between at least one valid pavement semantic information and at least one visual semantic information; performing semantic matching on each pair of semantically related valid pavement semantic information and visual semantic information, according to the semantic matching As a result, the second weight is determined. In this way, some incomplete or oversized falsely detected visual semantic information can be filtered out by determining the effective pavement semantic information, so as to improve the efficiency of semantic matching and reduce the calculation amount of semantic matching.
  • determining the semantic relationship between at least one valid pavement semantic information and at least one visual semantic information includes: calculating the semantic weight of any valid pavement semantic information a i and the arbitrary visual semantic information b j Semantic weight; according to the difference between the semantic weight of the effective pavement semantic information a i and the semantic weight of the visual semantic information b j , the semantic correlation degree of the effective pavement semantic information a i and the visual semantic information b j is determined; when the semantic correlation When the degree is less than the preset first threshold, it is determined that the effective pavement semantic information a i and the visual semantic information b j have a semantic relationship.
  • the semantic relevance when the semantic relevance is less than the first threshold, it means that the semantic weight of the effective pavement semantic information a i and the semantic weight of the visual semantic information b j are relatively close, which further means that the effective pavement semantic information a i and the visual semantic information b j have Semantic association.
  • semantic matching is performed on each pair of semantically associated valid pavement semantic information and visual semantic information, and the second weight is determined according to the semantic matching result, including: separately calculating the valid pavement semantic information for each pair of semantic associations.
  • the matching distance between the pavement semantic information and the visual semantic information; the weighted summation of the calculated matching distances is used to obtain the total matching distance; the second weight is determined according to the total matching distance. In this way, the second weight can reflect the closeness of the current estimated pose P n to the real pose of the vehicle.
  • the current estimated pose P 1 -PN of the N sampling points C 1 -CN and their first weights and second weights the current estimated pose P 1 -
  • the weighted average of P N , and the weighted average value is taken as the current pose of the vehicle, including: using the first weight of the current estimated pose P 1 -P N to weight the current estimated pose P 1 -P N and averaging, Obtain the first weighted average; use the second weight of the current estimated pose P 1 -PN to weight the current estimated pose P 1 -PN to obtain the second weighted average;
  • the second weighted average is weighted and averaged to obtain a weighted average, and the weighted average is used as the current pose of the vehicle.
  • the fusion of the laser positioning technology based on laser feature matching and the visual positioning technology based on visual feature matching is realized.
  • the pavement semantic information includes: a pixel block containing at least one pavement mark, the number of pixels in the pixel block, and the type of pavement mark to which each pixel belongs. In this way, the type, size, etc. of the pavement markings can be determined through the pavement semantic information, so as to facilitate matching with the pavement semantic information.
  • the current predicted pose is determined by the following steps: determining the relative pose of the vehicle generated between the current time t and the first historical time t -1 according to the odometer data; The predicted pose corresponding to the first historical time t-1 is added to the relative pose to obtain the current predicted pose.
  • the N sampling points C 1 -C N are regenerated. In this way, the divergence phenomenon of sampling points with the increase of positioning times can be eliminated.
  • an embodiment of the present application provides a positioning system, including: a GNSS/INS combination module, a control unit, a memory, a lidar, and a camera installed on a vehicle; and a GNSS/INS combination module for determining the initial position of the vehicle attitude; memory, used to store the positioning map, the positioning map includes multiple positioning pictures spliced with each other, the positioning picture includes a color channel, and the color channel stores the encoding of the first laser feature and the encoding of visual semantic information; lidar, used for collecting point cloud data, where the point cloud data includes the second laser feature; a camera for collecting image data, where the image data includes at least one pavement semantic information; a control unit for generating N sampling points C 1 to C N around the initial pose , N is a positive integer; the control unit is also used to extract the first laser feature and at least one visual semantic information from the positioning map according to the current predicted pose of the vehicle; the control unit is also used for any sampling point C n , n is A positive
  • Matching is performed to determine the second weight of the current estimated pose P n , and at least one road semantic information is extracted from the image data collected by the camera; the control unit is further configured to The current estimated pose P 1 -PN and its first weight and second weight are calculated, and the weighted average of the current estimated pose P 1 -PN is calculated, and the weighted average is used as the current pose of the vehicle.
  • the technical solutions provided by the embodiments of the present application can respectively determine the first weight of the currently estimated pose based on the matching of laser features, and determine the current estimated pose based on the matching of visual semantic information and road semantic information (ie, matching of visual features).
  • the second weight and then calculate the weighted average of the current estimated pose according to the first weight and the second weight, and use the weighted average as the current pose of the vehicle, thereby realizing the laser positioning based on laser feature matching.
  • the technology is integrated with the visual positioning technology based on visual feature matching, which improves the positioning efficiency.
  • the technical solutions provided by the embodiments of the present application encode and store laser features and visual semantic information in the same positioning map, which realizes the fusion of map data and reduces the data overhead and computing overhead generated in the positioning process.
  • the positioning picture includes a first color channel, and the first color channel is used to store coding of visual semantic information.
  • the positioning system provided by the embodiment of the present application can decode the first color channel to obtain visual semantic information.
  • the positioning picture further includes a second color channel, where the second color channel is used to store the code of the first laser feature.
  • the positioning system provided in this embodiment of the present application can decode the second color channel to obtain the first laser feature.
  • the encoding of the visual semantic information includes at least one of a marker bit, a type encoding, and a brightness encoding; the marker bit is used to represent the type of the road marking, and the type encoding is used to represent the content of the road marking, Luma information encoding is used to represent the luma information of a picture.
  • the marker bit is used to represent the type of the road marking
  • the type encoding is used to represent the content of the road marking
  • Luma information encoding is used to represent the luma information of a picture.
  • the control unit when the control unit is used to extract at least one visual semantic information from the positioning map according to the current predicted pose of the vehicle: the control unit is specifically configured to extract from the positioning map according to the current predicted pose Local positioning map, the local positioning map includes M positioning pictures, the M positioning pictures include the first picture where the current predicted pose is located, and M-1 second pictures near the first picture, where M is a positive integer greater than 1;
  • the control unit is further used for extracting at least one visual semantic information from the local localization map.
  • the visual semantic information can include information such as road signs around the current predicted pose, so as to be matched with the road semantic information in the image data around the vehicle collected by the camera to determine the semantic relationship and the second weight.
  • the control unit when the control unit is configured to match at least one visual semantic information with at least one pavement semantic information according to the current estimated pose P n for any sampling point C n to determine the current estimated position
  • the control unit specifically configured to determine at least one valid road semantic information from the at least one road semantic information, and the number of pixels of each valid road semantic information is within a preset range
  • the control unit also is used for projecting at least one valid pavement semantic information into the coordinate system of the local positioning map according to the current estimated pose P n
  • the control unit is further adapted to determine the semantic association relationship between at least one valid pavement semantic information and at least one visual semantic information
  • the control unit is further configured to perform semantic matching on each pair of semantically associated valid pavement semantic information and visual semantic information, and determine the second weight according to the semantic matching result. In this way, some incomplete or oversized falsely detected visual semantic information can be filtered out by determining the effective pavement semantic information, so as to improve the efficiency of semantic matching and reduce the calculation amount of semantic matching.
  • control unit when the control unit is used to determine the semantic relationship between at least one valid pavement semantic information and at least one visual semantic information: the control unit is specifically adapted to calculate the semantics of any valid pavement semantic information a i weight and the semantic weight of any visual semantic information b j ; the control unit is also used to determine the effective pavement semantic information according to the difference between the semantic weight of the effective pavement semantic information a i and the semantic weight of the visual semantic information b j The semantic correlation degree of a i and the visual semantic information b j ; the control unit is further configured to determine that the effective pavement semantic information a i and the visual semantic information b j have semantic correlation when the semantic correlation degree is less than the preset first threshold.
  • the semantic relevance when the semantic relevance is less than the first threshold, it means that the semantic weight of the effective pavement semantic information a i and the semantic weight of the visual semantic information b j are relatively close, which further means that the effective pavement semantic information a i and the visual semantic information b j have Semantic association.
  • control unit when the control unit is used to perform semantic matching on each pair of semantically associated valid pavement semantic information and visual semantic information, and determine the second weight according to the semantic matching result: the control unit, specifically using It is used to calculate the matching distance of each pair of semantically associated effective pavement semantic information and visual semantic information respectively; the control unit is also used to calculate the weighted sum of each matching distance to obtain the total matching distance; the control unit is also used to The total matching distance determines the second weight. In this way, the second weight can reflect the closeness of the current predicted pose P n to the actual pose of the vehicle.
  • control unit when the control unit is used to calculate the current estimated poses P 1 -PN and their first weights and second weights according to the current estimated poses P 1 -PN of the N sampling points C 1 -CN The weighted average of the poses P 1 - P N , when the weighted average is used as the current pose of the vehicle: the control unit is specifically configured to use the first weight of the current estimated pose P 1 -PN to determine the current estimated pose.
  • the poses P 1 -PN are weighted and averaged to obtain a first weighted average value; the control unit is further configured to use the second weight of the current estimated pose P 1 -PN to weight the current estimated pose P 1 -PN averaging to obtain a second weighted average value; the control unit is further configured to perform a weighted average of the first weighted average value and the second weighted average value to obtain a weighted average value, and use the weighted average value as the current pose of the vehicle.
  • the first weight and the second weight to weight the current estimated pose, the fusion of the laser positioning technology based on laser feature matching and the visual positioning technology based on visual feature matching is realized.
  • the pavement semantic information includes: a pixel block containing at least one pavement mark, the number of pixels in the pixel block, and the type of pavement mark to which each pixel belongs. In this way, the type, size, etc. of the pavement markings can be determined through the pavement semantic information, so as to facilitate matching with the pavement semantic information.
  • the positioning system further includes an odometer; the control unit is further configured to determine the relative pose of the vehicle between the current time t and the first historical time t-1 according to the odometer data; control The unit is also used to add the predicted pose corresponding to the sampling point C n at the first historical time t-1 and the relative pose to obtain the current predicted pose.
  • control unit is further configured to regenerate the N sampling points C 1 -C N when there is a preset ratio of the first weight or the second weight is lower than the second threshold. In this way, the divergence phenomenon of sampling points with the increase of positioning times can be eliminated.
  • an embodiment of the present application provides a vehicle, and the vehicle includes the positioning system provided by the second aspect of the embodiment of the present application and various implementations thereof.
  • embodiments of the present application further provide a computer-readable storage medium, where instructions are stored in the computer-readable storage medium, and when the computer-readable storage medium runs on a computer, the computer executes the methods of the above aspects and their respective implementations.
  • the embodiments of the present application further provide a computer program product including instructions, which, when run on a computer, enables the computer to execute the methods of the above aspects and their respective implementations.
  • an embodiment of the present application further provides a chip system, where the chip system includes a processor for supporting the above-mentioned apparatus or system to implement the functions involved in the above-mentioned aspects, for example, generating or processing the functions involved in the above-mentioned method. information.
  • Fig. 1 is the schematic diagram of point cloud type positioning map
  • Fig. 2 is the schematic diagram of picture type positioning map
  • Figure 3 is a schematic diagram of the amount of pixels of a sparse feature map, a semi-dense map and a dense map;
  • FIG. 4 is a configuration diagram of a positioning system module of an autonomous vehicle
  • Fig. 6 is a coding format diagram of visual semantic information provided by an embodiment of the present application.
  • FIG. 7 is a schematic diagram of 8-bit encoding of visual semantic information corresponding to pixels in different regions on the positioning map
  • FIG. 8 is a hardware framework diagram of a positioning system for implementing a positioning method provided by an embodiment of the present application.
  • FIG. 9 is a flowchart of a positioning method provided by an embodiment of the present application.
  • FIG. 10 is a block diagram of data flow involved in the positioning method provided by the embodiment of the present application.
  • Figure 11 exemplarily provides a scheme for generating sampling points
  • Figure 12 is a schematic diagram of an odometer coordinate system
  • FIG. 13 is a schematic diagram of a process for determining a first weight provided by an embodiment of the present application.
  • FIG. 14 is a schematic diagram of a process for determining a second weight provided by an embodiment of the present application.
  • FIG. 16 is a flowchart of step S201 of the positioning method provided by the embodiment of the present application.
  • 17 is a schematic diagram of calculating the pixel area of road semantic information provided by an embodiment of the present application.
  • FIG. 18 is a flowchart of step S203 of the positioning method provided by the embodiment of the present application.
  • FIG. 19 is a flowchart of step S204 of the positioning method provided by the embodiment of the present application.
  • 20 is a schematic diagram of a matching distance provided by an embodiment of the present application.
  • step S502 of the positioning method provided by the embodiment of the present application.
  • FIG. 22 is a flowchart of step S105 of the positioning method provided by the embodiment of the present application.
  • FIG. 23 is a block diagram of software modules of a positioning system provided by an embodiment of the present application.
  • Autonomous vehicles autonomous vehicles or self-driving automobiles
  • unmanned driving computer driving
  • Autonomous driving can sense its surrounding environment with technologies such as sensors, global navigation satellite system (GNSS) and machine vision, and determine its own position, plan navigation routes, update map information, and avoid obstacles based on the sensing data.
  • GNSS global navigation satellite system
  • the ultimate goal is to drive the vehicle autonomously with little or no human activity.
  • autonomous driving needs to solve three core problems of driving: where? (vehicle location); where to go? (determine the destination); how to get there? (route plan).
  • positioning technology is mainly used to solve the problem of "where?", and is one of the key technologies that are essential to realize autonomous driving.
  • the positioning technologies for autonomous driving mainly include the following three types:
  • GNSS global navigation satellite system
  • BDS Beidou navigation satellite system
  • GPS global positioning system
  • the technology requires the vehicle to be equipped with sensors such as an inertial measurement unit (IMU) and wheel speedometers.
  • IMU inertial measurement unit
  • wheel speedometer can measure the rotational speed of the wheels.
  • the current pose (position and attitude) of the vehicle can be estimated according to the dynamic equation of the vehicle.
  • this technology mainly includes two forms: laser positioning and visual positioning. Based on the laser sensor and the visual sensor, the environmental information around the vehicle is obtained in real time, and the obtained environmental information is processed and matched with the pre-stored positioning map to determine the vehicle's location. pose. It is understandable that the implementation of this technology needs to build a positioning map in advance, and the positioning map also has different construction methods according to the different sensors used.
  • the positioning map is mainly cloud-like and picture-like.
  • 1Point cloud type Use laser sensors to collect point cloud data, and then filter the point cloud data to remove noise. Finally, the processed point cloud data are spliced and superimposed together to form a point cloud map as shown in Figure 1.
  • 2Picture format Perform grid processing on the point cloud data spliced in 1, and encode each grid as a pixel, so as to convert the stitched point cloud data into the positioning of the picture format as shown in Figure 2 map.
  • the laser map in the picture format can be a single-channel grayscale picture or a three-channel color picture.
  • the image-based positioning map occupies a small storage space, which can solve the problem of high storage resource overhead of the point cloud map.
  • the positioning map (visual map) is shown in Figure 3, which can include sparse feature map, semi-dense map and dense map three At present, it is mainly saved in the form of point cloud.
  • 2Semi-dense map Save some pixels in the original image, such as pixels with gradients, into the map, so the number of pixels is medium.
  • 3Dense map Save all the pixels in the original image to the map, so the number of pixels is the largest.
  • Signal-based positioning technology is currently the most widely used positioning technology due to its global positioning capability.
  • the signal-based positioning technology cannot provide accurate positioning when the vehicle is driving in cities, tunnels and other road conditions, so it cannot meet the needs of full-scene autonomous driving.
  • the current research direction in the field of autonomous driving is to integrate signal-based positioning technology, dead reckoning-based positioning technology, and feature matching-based positioning technology to make up for the shortcomings of signal-based positioning technology.
  • the laser positioning technology using laser feature matching supplements or replaces the signal-based positioning technology and the visual positioning technology using visual feature matching supplements or replaces the signal-based positioning technology.
  • the technical routes are independent of each other, the map data of laser positioning and visual positioning are stored separately, and the algorithms are independent of each other, resulting in excessive data storage and calculation overhead, high requirements for the vehicle's control unit ECU and other hardware systems, and low efficiency.
  • the embodiments of the present application provide a positioning method.
  • the technical solutions of the embodiments of the present application can be applied to various vehicles using automatic driving technology or positioning technology, including but not limited to various vehicles: such as vehicles (cars), ships, trains, subways, airplanes, etc., and Various robots, such as service robots, transport robots, automated guided vehicles (AGVs), unmanned ground vehicles (UGVs), etc., as well as various construction machinery, such as tunnel boring machines.
  • vehicles such as vehicles (cars), ships, trains, subways, airplanes, etc.
  • Various robots such as service robots, transport robots, automated guided vehicles (AGVs), unmanned ground vehicles (UGVs), etc.
  • various construction machinery such as tunnel boring machines.
  • the vehicle is configured with the following modules: a lidar LiDAR 110 , a camera 120 , a GNSS/inertial navigation system (INS) combined module 130 , and a control unit 140 . in:
  • the lidar 110 is used to collect the distance information of the elements around the environment (for example: vehicles, obstacles, pedestrians, street signs, etc.) from the vehicle.
  • the lidar 110 can perform omnidirectional scanning of 360 degrees around the environment, or can only scan environmental information within a partial range (eg, 180 degrees) in front of the front of the vehicle.
  • the camera 120 is used to collect image information around the vehicle.
  • the camera 120 may have one (that is, a monocular camera) or multiple (that is, a multi-eye camera), and may capture a 360-degree panoramic image of the surroundings, or may only capture an image of a partial range in front of the front of the vehicle.
  • the GNSS/INS combination module 130 may include devices such as a GNSS receiver and an inertial measurement unit IMU, and is used to implement fusion positioning based on satellite signals and IMU.
  • the control unit 140 may be the core computing unit of the electronic system of the entire autonomous vehicle, such as: a mobile data center (mobile data center, MDC), an electronic control unit (electronic control unit, ECU), etc., for processing other various modules to generate data, and generate vehicle control information based on the processing results.
  • a mobile data center mobile data center, MDC
  • an electronic control unit electronic control unit, ECU
  • the vehicle can also be configured with the following modules:
  • the ultrasonic sensor 150 is used for short-distance ranging, for example, it is turned on when assisting in parking, so as to provide short-distance warning information and the like.
  • Millimeter-wave radar 160 used for long-distance ranging, due to the strong anti-jamming ability of millimeter-wave and the ability to penetrate fog, smoke and dust, the millimeter-wave radar can work around the clock, for example, to assist in obstacle detection in bad weather Object ranging.
  • the hardware environment illustrated by the embodiments of the present application does not constitute a specific limitation on the technical solutions of the embodiments of the present application.
  • the hardware environment implemented by the technical solutions of the embodiments of the present application may include more or less components than shown in the figure, or combine some components, or separate some components, or different Component placement.
  • the illustrated components may be implemented in hardware, software, or a combination of software and hardware.
  • a positioning map for laser positioning (hereinafter referred to as a laser map) and a positioning map for visual positioning (hereinafter referred to as a visual positioning map)
  • the map data in the map) is fused into a positioning map, so that when the laser positioning and visual positioning technologies are applied at the same time, the data required for these two positioning methods can be obtained from the fused positioning map, thereby reducing the amount of data storage.
  • Overhead and computational overhead improve positioning efficiency.
  • the basic idea of obtaining the positioning map in the embodiment of the present application is to integrate and add the visual semantic information of the road markings to the laser map, so as to obtain the positioning map.
  • the road signs include information used to assist in determining the position and lane of the vehicle, such as: single yellow line, double yellow line, white dotted line, white solid line, straight sign, left turn sign, right turn sign, U-turn sign, etc.
  • the embodiment of the present application preferably uses a laser map in a picture format. Laser maps are composed of grayscale images with single color channel or color images with multiple color channels.
  • the laser map is implemented as a three-channel color image, including, for example, an R channel (red channel), a G channel (green channel), and a B channel (blue channel) ), and some laser map feature information can also be added to each channel.
  • the R channel can be added with laser features, such as line features, corner features, gradient features, height features, etc. of various elements in the map
  • the G channel can be added with the brightness information of the laser map
  • the B channel can add relative height information, such as the height of the pixel relative to the ground, etc.
  • the laser map may also include more channels, such as an alpha channel, etc., which is not limited in this embodiment of the present application.
  • the laser map may also be implemented by other color modes, such as RYYB format, etc. In this case, the laser map may correspondingly include four channels.
  • each channel in the pixel of the laser map may contain a certain number of bits of information bits, such as 8 bits (8 bits of information), 16 bits, 32 bits, and so on. Therefore, feature information such as the above-mentioned laser feature, brightness information, and relative height information can be encoded and represented in the bits of the pixel.
  • the brightness information can be represented by 1 bit encoding, for example, a bit value of 1 indicates that there is brightness information, and 0 indicates that there is no brightness information.
  • the visual semantic information usually located in the visual map is fused into the laser map.
  • the pixels that will contain visual semantic information in the laser map can be determined according to the corresponding positions of the content of road signs and road traffic signs in the laser map, and then the visual semantic information is encoded and stored in one or some of these pixels. in the bit information of these channels.
  • the visual semantic information may be represented in an 8-bit binary encoding format, occupying 8 bits of the channel where it is located.
  • FIG. 6 is an example diagram of an 8-bit encoding format for visual semantic information.
  • the 8-bit encoding format can be composed of at least one of three parts: marker bit, type encoding and luminance encoding, wherein the marker bit can be used to indicate the type of road marking; the type encoding is used to indicate the road marking content; brightness information coding can be used to represent the brightness information of the picture.
  • the flag bit occupies 1 bit as shown in FIG. 6 , for example, the first bit of the 8-bit encoding, or other bits.
  • the flag bit can have two values, 0 and 1, representing at most two types. For example, if the visual semantic information is divided into text elements and graphic reticle elements, then 0 can represent text elements, 1 can represent graphic reticle elements, or 1 represents text elements, and 0 represents graphic markers. Line element. For example, if the visual semantic information is divided into pavement marking elements and other marking elements, then 0 may represent pavement marking elements, 1 may represent other marking elements, or 1 may represent pavement marking elements, and 0 may represent other marking elements. Logo class element.
  • the flag bit can occupy more than 1 bit, such as 2 bits, 3 bits, etc., so that more types of visual semantic information can be represented, for example, 2 bits can represent at most 4 types, 3 bits can represent up to 8 kinds.
  • 2 bits can represent at most 4 types
  • 3 bits can represent up to 8 kinds.
  • those skilled in the art can determine the length of the flag bit according to actual classification requirements, and the embodiment of the present application does not specifically limit the length of the flag bit.
  • the type code occupies 6 bits as shown in FIG. 6 , for example, 6 consecutive bits following the flag bit, or other bits.
  • the value range of the type code can be from 000000 to 111111, which can represent up to 26 content types. Exemplarily: 000000 can represent road, 000001 can represent road signs, etc; road signs etc.
  • the type encoding is a necessary part of the 8-bit encoding of the visual semantic information.
  • the type code can occupy more than 6 bits, such as 7 bits, so that more content types can be represented; it can also be less than 6 bits, such as 5 bits, 4 bits, etc., so that in In the case of being able to represent all the required road signs and/or road traffic signs, reduce the bit length of the type code to reduce the data overhead and allow more bit information to be reserved in the 8-bit code to represent other information.
  • those skilled in the art can determine the length of the type code according to the number of road signs and/or road traffic signs to be distinguished, and the length of the type code is not specifically limited in this embodiment of the present application.
  • the luma encoding occupies 1 bit as shown in FIG. 6 , such as the last bit of the 8-bit encoding, or other bits.
  • the flag bit can have two values of 0 and 1, 0 means no brightness information, 1 means there is brightness information. Taking the lane line as an example, its brightness is higher than that of the road surface, so its brightness information can be 1; taking the road marking as an example, it may include the white line part and the road surface part, then the brightness information of the white line part is It can be 1, and the brightness information of the road surface can be 0.
  • the luminance information may occupy more than 1 bit, such as 2 bits, 3 bits, etc., so that the luminance can be represented in more subdivisions.
  • those skilled in the art can determine the length of the brightness information according to the actual classification requirements, and the embodiments of the present application do not specifically limit the length of the brightness information.
  • the binary encoding of the visual semantic information may only include a part of the flag bit, the type encoding and the luminance encoding, for example, only the type encoding and the luminance encoding, or only the flag bit and the type encoding, or only the type encoding.
  • Type encoding may also be other encoding formats other than 8-bit encoding formats, such as encoding greater than 8 bits, such as 16-bit encoding, or encoding less than 8 bits, such as 4-bit encoding, etc., this application implements The example does not limit this.
  • the encoded visual semantic information can be stored in the G channel of the pixel.
  • the 8 bits of information can sequentially include flag bits, type encoding and luma encoding.
  • FIG. 7 is a schematic diagram of 8-bit encoding of visual semantic information corresponding to pixels of different regions on the positioning map.
  • the pixels in area 1 correspond to the road surface, and the flag bit is 0, the type code is 000000, and the brightness code is 0, so the visual semantic information is 00000000;
  • the pixels in area 2 correspond to straight signs , its flag bit is 1, the type code is 000011, and the brightness code is 1, so the visual semantic information is 10000111;
  • the pixel in area 3 corresponds to the white solid line mark, its flag bit is 1, the type code is 000010, and the brightness code is 1. So the visual semantic information is 10000101.
  • the visual semantic information originally located in the visual map is encoded and stored in the pixel channel of the laser map, so as to realize the fusion of the laser map and the visual map, and obtain a visual map feature and a laser map at the same time.
  • the location map of the feature reduces the storage overhead of the map.
  • FIG. 8 is a hardware framework diagram of a positioning system for implementing a positioning method provided by an embodiment of the present application.
  • the positioning system may include a control unit 140 , a GNSS/INS combination module 130 , a wheel speedometer 170 , an odometer 180 , a lidar 110 , a camera 120 , a memory 190 , and the like.
  • the GNSS/INS combination module, wheel speedometer, odometer, lidar, camera and other modules are used to collect data respectively, and send the data to the control unit for processing. data, program instructions for execution by the control unit, and data generated by the control unit during data processing.
  • the step flow of the positioning method provided by the embodiment of the present application will be described in detail. It can be understood that, in addition to the vehicle, the method of the embodiment of the present application
  • the positioning target can also be ships, trains and other vehicles, various robots, and construction machinery.
  • FIG. 9 is a flowchart of a positioning method provided by an embodiment of the present application
  • FIG. 10 is a block diagram of data flow involved in the positioning method. As shown in FIG. 9 and FIG. 10 , the positioning method can be implemented through the following steps S101-S109:
  • Step S101 determine the initial pose of the vehicle, and generate N sampling points C 1 -C N around the initial pose, where N is a positive integer.
  • the initial posture may include the initial position and initial posture of the vehicle.
  • the control unit can obtain the data collected by the GNSS/INS combination module. Then, the control unit may determine the initial position of the vehicle according to the antenna signal of the GNSS. Generally speaking, the initial position of the vehicle may be a global position. In addition, the control unit can also determine the initial attitude of the vehicle according to information such as the angular velocity and acceleration of the vehicle measured by the inertial measurement unit IMU of the INS module. Generally speaking, the initial attitude of the vehicle can be determined from the initial heading angle, pitch angle and roll angle of the vehicle. One or more parameters of , since the heading angle is mainly used in vehicle positioning and navigation, the initial attitude of the vehicle can also only include the heading angle.
  • control unit may further generate N sampling points C 1 within a certain range near the vehicle and within a certain range near the heading angle of the vehicle with the initial position of the vehicle as the center ⁇ CN .
  • Figure 11 exemplarily provides a scheme for generating sample points.
  • the 1000 sampling points may be generated in a uniform distribution manner, so that the distribution of the 1000 sampling points in the distribution area thereof is relatively uniform.
  • the initial pose of the sampling point is also determined according to the initial pose of the vehicle.
  • the 1000 sampling points may also be generated in a non-uniform manner, such as a normal distribution, which is not limited in this embodiment of the present application.
  • control unit may perform steps S102 to S108 for each sampling point respectively.
  • Step S102 extracting the first laser feature and at least one visual semantic information from the positioning map according to the current predicted pose of the vehicle.
  • the control unit can periodically position the vehicle, and the positioning behavior of each period can be called a positioning frame.
  • the positioning frame corresponding to the current time t can be called the current frame
  • the previous positioning frame of the current frame is called the first historical frame
  • the first historical frame is called the first historical frame.
  • the time of one historical frame is recorded as the first historical time t-1.
  • the obtained vehicle pose at the first historical time t-1 is referred to as the first historical pose.
  • the first historical pose and the odometer parameters it can be predicted that the vehicle is at the current time.
  • the current predicted pose P t of t It should be noted that, if the first historical moment t-1 is the initial moment, then the first historical pose is the initial pose of the vehicle.
  • this embodiment of the present application may determine the current predicted pose of the vehicle in the following manner:
  • Step a obtain the initial pose of the vehicle.
  • the initial pose may be the pose determined according to the data collected by the GNSS/INS combination module when the positioning method is initialized and executed.
  • Step a is only used to execute when the method is initialized.
  • Step b Determine the relative pose of the vehicle between the current time t and the first historical time t-1 according to the odometer data.
  • the embodiment of the present application uses the mileage coordinate system to obtain the relative pose.
  • Figure 12 is a schematic diagram of an odometer coordinate system.
  • the odometer coordinate system can take the initial pose of the vehicle as the origin Odom, and take the forward direction of the vehicle body in the initial pose as the X-axis direction, so as to be perpendicular to the forward direction of the vehicle body and point to the vehicle The direction of the left side of the body is taken as the Y-axis direction.
  • the local pose of the vehicle in the odometer coordinate system can be calculated by the odometer according to the measurement data of the wheel speedometer and the inertial measurement unit IMU.
  • the odometry can adopt the following kinematic models:
  • S represents the movement distance of the vehicle relative to the initial pose
  • V represents the movement speed of the vehicle
  • ⁇ t represents the movement time of the vehicle
  • ⁇ represents the angular velocity of the vehicle.
  • x 0 is the X-axis coordinate value when the vehicle is moving
  • y 0 is the Y-axis coordinate value when the vehicle is moving
  • yaw is the heading angle of the vehicle.
  • the value of yaw is the odometer coordinate system.
  • the local pose expressed by the parameters of the first local coordinate system at any time when the vehicle is moving can be obtained.
  • the local pose may include ( x 0 , y 0 , yaw).
  • the local pose only represents the pose of the vehicle in the odometer coordinate system, and does not represent the absolute pose of the vehicle in the space environment.
  • the odometer can send the local pose of the vehicle at the current time t and the local pose of the first historical time t-1 to the control unit. Then, the control unit can calculate the relative pose of the vehicle between the current time t and the first historical time t-1 according to the local pose of the vehicle at the current time t and the local pose of the first historical time t-1.
  • the specific calculation method is as formula (5):
  • ⁇ P is the relative pose of the vehicle between the current time t and the first historical time t-1
  • o t is the local pose of the vehicle at the current time t
  • o t-1 is the vehicle at the first historical time t. -1 for the local pose.
  • Step c adding the first historical pose corresponding to the vehicle at the first historical time t-1 and the relative pose to obtain the current predicted pose P t of the vehicle.
  • P t-1 is the first historical pose corresponding to the vehicle at the first historical time t-1.
  • P n(t-1) is the estimated pose corresponding to the sampling point C n at the first historical time t-1.
  • control unit can obtain an area near the current predicted pose P t from the positioning map according to the current predicted pose P t of the vehicle. Extract the first laser feature from the map.
  • the positioning map may be formed by splicing a large number of pictures of preset sizes, and each picture corresponds to a range of a specified size in the spatial environment.
  • each picture of the positioning map is a square picture of equal length and width, and each picture corresponds to a square range of 100 meters long and 100 meters wide.
  • the control unit may obtain the picture where the current predicted pose P t is located and at least one nearby picture from the positioning map as the local positioning map. For example, as shown in Fig. 13 or 14, the control unit can obtain a total of 9 pictures of 3 ⁇ 3 where the current predicted pose P t is located and nearby, if the corresponding range of each picture is 100m ⁇ 100m, then the local positioning map It includes an area of 300m ⁇ 300m near the current predicted pose P t .
  • control unit may extract the first laser feature from the channel of the picture in which the laser feature is stored, for example, extract the first laser feature from the R channel.
  • control unit may extract at least one visual semantic information from the channel storing the visual semantic information in the local location map, for example, decode the G channel data of the picture to extract at least one visual semantic information in the G channel.
  • Step S103 for any sampling point C n , where n is a positive integer, n ⁇ N, according to its corresponding current estimated pose P n , the first laser feature extracted from the self-positioning map is matched with the second laser feature to determine: The first weight of the currently estimated pose P n .
  • the second laser feature can be extracted from the point cloud data collected by the lidar.
  • the control unit may sample the point cloud data collected by the lidar at the current time t to obtain the second laser feature, and the sampling method may be determined according to the specific form of the positioning map. For example: when the positioning map is a sparse feature map, the control unit can perform sparse feature sampling on the point cloud data; when the positioning map is a semi-dense map, the control unit can perform semi-dense feature sampling on the point cloud data; when the positioning map is dense When mapping, the control unit can perform dense feature sampling on the point cloud data. In this way, matching of the first laser feature with the second laser feature is facilitated.
  • the control unit projects the laser features into the coordinate system of the local positioning map according to the current estimated pose P n .
  • the above laser features are based on After the current estimated poses P n of different sampling points C n are projected, they will correspond to different coordinate distributions in the local positioning map.
  • the control unit may match the first laser feature and the second laser feature based on the coordinate distribution of the first laser feature and the second laser in the local positioning map, and calculate the difference between the first laser feature and the second laser feature
  • the matching distance of , and the first weight of the current estimated pose P n is determined according to the matching distance Among them, the matching distance represents the closeness between the actual pose of the vehicle determined based on the laser feature and the current estimated pose Pn of the sampling point Cn , and the higher the closeness is, the higher the first weight is. The larger, the lower the degree of proximity, the first weight the smaller.
  • the matching distance may be a cosine distance or an Euler distance, and the embodiments of the present application do not limit the algorithm used to obtain the matching distance.
  • the value range of the matching distance can be [0, 1]. The larger the value, the difference between the actual pose of the vehicle determined based on the laser feature and the current estimated pose P n of the sampling point C n The lower the closeness between the two, the smaller the value, which means the higher the closeness between the actual pose of the vehicle determined based on the laser feature and the current estimated pose Pn of the sampling point Cn .
  • Step S104 for any sampling point C n , according to the current estimated pose P n , match at least one visual semantic information with at least one road semantic information to determine the second weight of the current estimated pose P n .
  • the above-mentioned at least one pavement semantic information is extracted from the image data collected by the camera.
  • the control unit may first perform preprocessing on the image data collected by the camera, such as noise removal, cropping, grayscale processing, and the like.
  • the control unit can use the pre-trained deep neural network to perform pixel-level semantic segmentation on the preprocessed image to extract at least one pavement semantic information from the image.
  • the pavement semantic information can be pixel-level information, and each pavement
  • the semantic information may include: a pixel block containing at least one road marking, the number of pixels in the pixel block, the type and probability of the road marking to which each pixel belongs, and the like.
  • the pixel block should contain at least all pixels of the road marking.
  • the pixel block may be a regular shape such as a rectangle, a circle, etc., or other shapes, preferably a regular shape to facilitate data processing.
  • the pixel block when it is guaranteed that the pixel block includes all the pixels of the road surface marking, the pixel block preferably contains as few pixels as possible that are not the road surface marking.
  • the deep neural network used in the embodiments of the present application may be, for example, a convolutional neural network (CNN), a long short-term memory (LSTM), a recurrent neural network (RNN), or Other neural networks can also be a combination of various neural networks.
  • the deep neural network uses the training corpus as input in the training phase, the training corpus can be the road image collected in advance, and the road signs in the road image are marked at the pixel level; the output of the deep neural network in the training phase is the annotation result of the training corpus , such as the type of pavement markings being marked, etc.
  • the input of the neural network in the use stage is the image collected by the camera, and the output is the pixel-level information of the road signs contained in the image.
  • the specific method of using the deep neural network for information extraction is not the focus of the discussion in this embodiment of the present application, and due to space limitations, it will not be repeated here.
  • the pavement semantic information actually belongs to the visual semantic information, the difference is that it is extracted from the image collected by the camera, rather than stored in the positioning map.
  • step S104 can be implemented by the following steps S201-S204 as shown in FIG. 15 :
  • Step S201 at least one valid road semantic information is determined from the at least one road semantic information, and a set of valid road semantic information is established, wherein the number of pixels of each valid road semantic information is within a preset range.
  • step S201 can be specifically implemented by the following steps S301-S303:
  • Step S301 respectively calculating the pixel area of each road surface semantic information.
  • step S105 the control unit acquires a plurality of pavement semantic information from the image collected by the camera, for example, including pavement semantic information L0 , pavement semantic information L1 and pavement semantic information L2 .
  • Step S302 determining the effective road semantic information according to the pixel area.
  • the embodiment of the present application may set a pixel area lower threshold value T 1 and a pixel area upper threshold value T 2 for determining the effective road semantic information, and the control unit uses the lower threshold value T 1 and the upper threshold value T 2 to be respectively associated with each
  • the pixel area S (for example: S 0 , S 1 , S 2 , etc.) of the road semantic information is compared.
  • T 1 ⁇ S ⁇ T 2 the road semantic information is valid semantic information.
  • S ⁇ T 1 or S> the pavement semantic information is invalid semantic information.
  • the lower threshold T1 and the upper threshold T2 may be preset values or dynamically generated values.
  • the control unit can count the pixel areas of all road semantic information within a period of time to obtain the distribution range of the pixel areas, and then calculate the pixel area in the pixel area. A certain range is selected from the distribution range as the range of valid road semantic information, and then the lower threshold T1 and the upper threshold T2 are determined .
  • Step S303 establishing an effective pavement semantic information set for the effective pavement semantic information.
  • the quantity of valid road semantic information included in the valid road semantic information set is also different.
  • the effective pavement semantic information set is an empty set; when it is determined in step S302 that a part of pavement semantic information is valid pavement semantic information, the effective pavement semantic information set is pavement semantic information A subset of the semantic information set, when it is determined in step S302 that all the pavement semantic information is valid pavement semantic information, the effective pavement semantic information set is the same as the pavement semantic information set.
  • any valid pavement semantic information in the effective pavement semantic information set may be in the following form:
  • a i represents the ith valid pavement semantic information in the effective pavement semantic information set
  • M represents the number of pixels contained in a i
  • Ka i represents the semantic type value corresponding to a i (such as the type value of pavement signs)
  • p 1 -p M respectively represent the probability that the 1st to M-th pixels in a i belong to Kai
  • p 1 -p M can be obtained from the output results of the deep neural network.
  • steps S301 to S303 are exemplary implementations of step S201.
  • Step S202 unifying the coordinate system of the effective pavement semantic information and the visual semantic information.
  • control unit may project the at least one valid road semantic information into the coordinate system of the local positioning map according to the current estimated pose P n .
  • the control unit may project the at least one valid road semantic information into the coordinate system of the local positioning map according to the current estimated pose P n .
  • the local positioning map may use a known coordinate system such as the GNSS coordinate system, or may independently have its own coordinate system.
  • the control unit may take the center point of the local positioning map as the origin, and take For the X-axis and Y-axis, establish the coordinate system of the local positioning map.
  • the control unit can project the effective road semantic information from the camera coordinate system to the coordinate system of the local positioning map by means of matrix transformation.
  • the transformation matrix can be, for example, a matrix with a size of 4 ⁇ 4, and its mathematical meaning represents the spatial transformation process of one translation and one rotation, that is to say, any pixel point in the valid road semantic information can undergo one translation and one rotation.
  • the rotation is projected into the coordinate system of the local positioning map. Projection transformation of spatial points between different coordinate systems is a common method in the field of navigation and positioning, and will not be repeated here.
  • Step S203 determining a semantic association relationship between at least one valid pavement semantic information and at least one visual semantic information.
  • step S203 For any valid pavement semantic information a i (a i ⁇ A) in the effective pavement semantic information set, find a semantically related to a i from the visual semantic information set B of the local positioning map.
  • step S203 as shown in FIG. 18 can be implemented by the following steps S401-S404:
  • Step S401 calculating the semantic weight of the effective pavement semantic information a i .
  • M represents the number of pixels included in a i
  • p m represents the probability that the mth pixel in a i belongs to Kai
  • Kai represents the semantic type value corresponding to a i .
  • Step S402 calculating the semantic weight of the visual semantic information b j .
  • G represents the number of pixels contained in b j
  • p g represents the gth pixel in b j belongs to The probability
  • Step S403 according to the difference between the semantic weight of the effective pavement semantic information a i and the semantic weight of the visual semantic information b j , determine the semantic relevance of the effective pavement semantic information a i and the visual semantic information b j .
  • the semantic relevance ⁇ w is the absolute value of the difference between the effective pavement semantic information a i and any visual semantic information b j , that is, the following formula is used:
  • Step S404 when the semantic correlation degree is less than the preset first threshold, it is determined that the effective road semantic information a i and the visual semantic information b j have semantic correlation.
  • any valid pavement semantic information a i and any visual semantic information b j if their semantic relevance ⁇ w is less than the preset first threshold ⁇ , then the effective pavement semantic information a i and any visual semantic information b j have Semantic correlation; if its semantic correlation degree ⁇ w is greater than or equal to the preset first threshold ⁇ , the effective road semantic information a i and any visual semantic information b j do not have semantic correlation.
  • step S401 to step S404 by performing the operations from step S401 to step S404 multiple times, at least one pair of valid pavement semantic information and visual semantic information with semantic association can finally be obtained.
  • valid pavement semantic information and visual semantic information with semantic association Information may be referred to as a semantically related group.
  • steps S401 to S404 are exemplary implementations of step S203.
  • step S204 semantic matching is performed on each pair of semantically associated valid pavement semantic information and visual semantic information, and a second weight is determined according to the semantic matching result.
  • step S204 can be implemented by the following steps S501-S503:
  • Step S501 respectively calculating the matching distance between each pair of semantically associated effective pavement semantic information and visual semantic information.
  • the matching distance may be the Euclidean distance (ie Euclidean distance) of the effective pavement semantic information and visual semantic information, or cosine distance, etc.
  • the embodiment is not specifically limited.
  • FIG. 20 shows the positions of the valid road semantic information a i and the visual semantic information b j in the coordinate system of the local localization map. Then the Euclidean distance is the straight-line distance between the effective pavement semantic information a i and the visual semantic information b j in the coordinate system of the local positioning map.
  • the cosine distance is the cosine value of the angle ⁇ ij between the effective road semantic information a i and the visual semantic information b j and the origin of the coordinate system.
  • Step S502 the weighted summation of each matching distance is performed to obtain a total matching distance.
  • step S502 can be implemented through the following steps S601-S602 as shown in FIG. 21 .
  • Step S601 calculating the weight of each semantic association group.
  • K represents the number of semantic association groups
  • w′ k represents the weight of the k-th semantic association group
  • ⁇ w k represents the semantic association degree of the k-th semantic association group.
  • Step S602 Determine the total matching distance according to the weights and matching distances of all semantic association groups.
  • the weight of each semantic association group and the matching distance can be multiplied separately, and then all the multiplication results are summed to obtain the total matching distance E, that is, the following formula is used:
  • K represents the number of semantic association groups
  • w′ k represents the weight of the k-th semantic association group
  • E k represents the matching distance of the k-th semantic association group.
  • Step S503 determine the second weight of the current estimated pose P n according to the total matching distance
  • the total matching distance represents the closeness between the actual pose of the vehicle determined based on the visual features and the current estimated pose Pn of the sampling point Cn , and the higher the closeness, the second weight. The higher, the lower the degree of proximity, the second weight lower.
  • the corresponding second weight The larger the value of the total matching distance, the lower the proximity between the actual vehicle pose determined based on the visual feature and the current estimated pose P n of the sampling point C n , so the corresponding second weight smaller.
  • the matching distance is a cosine distance
  • the smaller the value of the total matching distance the higher the degree of closeness between the actual pose of the sampling point C n determined based on the visual feature and the current predicted pose P n , so the corresponding second weight
  • the larger the value of the total matching distance the lower the proximity between the actual pose of the sampling point C n determined based on the visual feature and the current predicted pose P n , so the corresponding second weight smaller.
  • the control unit can use any algorithm to determine the second weight This embodiment of the present application does not limit this.
  • second weight The value range of , for example, may be [0, 1], or other ranges, which are not limited in this embodiment of the present application.
  • the second weight It can be obtained by complementing the total matching distance, taking the reciprocal, and normalizing the numerical range.
  • steps S501 to S504 are exemplary implementations of step S204.
  • Step S105 Calculate the current estimated pose P 1 according to the current estimated pose P 1 -PN of the N sampling points C 1 to CN and the first weight and the second weight - The weighted average of P N , and the weighted average is used as the current pose of the vehicle.
  • step S105 can be implemented by the following steps S701-S703 as shown in FIG. 22 :
  • Step S701 using the first weights of the current estimated poses P 1 -PN to weight the current estimated poses P 1 -PN to obtain a first weighted average.
  • the first weighted average can be obtained by the following formula:
  • P l represents the first weighted average value
  • N is the number of sampling points
  • P n represents the current estimated pose of the nth sampling point
  • Step S702 using the second weight of the current estimated pose P 1 -PN to weight the current estimated pose P 1 -PN to obtain a second weighted average.
  • the second weighted average can be obtained by the following formula:
  • P c represents the second weighted average
  • N is the number of sampling points
  • P n represents the current estimated pose of the nth sampling point
  • step S701 and step S702 are only used to describe the method steps, and do not represent the sequence of steps.
  • the control unit may execute step S701 and step S702 in parallel, or may execute step S701 and step S702 in sequence.
  • the first weighted average value and the second weighted average value are both poses.
  • Step S703 the first weighted average value and the second weighted average value are weighted and averaged to obtain a weighted average value, and the weighted average value is used as the current pose of the vehicle.
  • the positioning result of the vehicle can be obtained by the following formula:
  • P is the current pose of the vehicle, that is, the actual pose of the vehicle output by the positioning system at this time;
  • P l represents the first weighted average value
  • P c represents the second weighted average value;
  • ⁇ l represents the first weighted average value.
  • the values of ⁇ l and ⁇ c may be determined according to actual requirements, and the larger the value of ⁇ l or ⁇ c , the higher the weight of the first weighted average or the second weighted average.
  • the technician wants to use the laser feature to dominate the positioning result he can increase the value of ⁇ l , for example, the value of ⁇ l is 0.7, 0.8, etc.; if the technician wants to use the visual feature to dominate the positioning result, he can increase the value of ⁇ l
  • the value of large ⁇ c for example, the value of ⁇ c is 0.7, 0.8, etc.; if the technician wants the laser feature and the visual feature to play an equal role in the positioning result, the value of ⁇ l and ⁇ c can be both 0.5.
  • the positioning system has completed a complete positioning process.
  • the positioning process is also continuously performed.
  • the distribution of sampling points may diverge.
  • the divergence condition of the sampling points can be set by those skilled in the art, for example: when the second weight of the currently estimated pose corresponding to a preset ratio of sampling points is lower than the preset threshold, the sampling points are considered divergent; or, When the first weight of the currently estimated pose corresponding to a preset ratio of sampling points is lower than the preset threshold, the sampling points are considered to be divergent.
  • the control unit may reselect the same number of sampling points as before.
  • the embodiment of the present application also does not limit the method of reselection of sampling points.
  • the control unit may also reselect the sampling points periodically, for example, every 100 positioning frames is regarded as a reselection period, and the sampling points are reselected.
  • the positioning method provided by the embodiment of the present application can respectively determine the first weight of the currently estimated pose based on the matching of laser features, and determine the current estimated pose based on the matching of visual semantic information and road semantic information (that is, the matching of visual features).
  • the second weight and then weighted and averaged the current estimated pose according to the first weight and the second weight to obtain the positioning result of the vehicle, thus realizing the combination of the laser positioning technology based on laser feature matching and the visual feature matching based laser positioning technology.
  • the visual positioning technology is integrated to improve the positioning efficiency.
  • the positioning method provided by the embodiments of the present application encodes and stores laser features and visual semantic information in the same positioning map, which realizes the fusion of map data and reduces the data overhead and calculation overhead generated in the positioning process.
  • the positioning system may include corresponding hardware structures and/or software modules for performing each function.
  • the present application can be implemented in hardware or a combination of hardware and computer software with the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein. Whether a function is performed by hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Skilled artisans may implement the described functionality using different methods for each particular application, but such implementations should not be considered beyond the scope of this application.
  • the positioning system may implement corresponding functions through the hardware structure shown in FIG. 8 , the positioning system may be installed in the vehicle, and the installation manner of some components may be as shown in FIG. 4 .
  • the GNSS/INS combination module 130 is used to determine the initial pose of the vehicle;
  • the memory 190 is used to store the positioning map, the positioning map includes a plurality of positioning pictures spliced with each other, the positioning picture includes a color channel, and the color channel stores the first coding of laser features and coding of visual semantic information; lidar 110 for collecting point cloud data, point cloud data containing second laser features; camera 120 for collecting image data, image data containing at least one pavement semantic information;
  • control The unit 140 is used to generate N sampling points C 1 to C N around the initial pose, where N is a positive integer;
  • the control unit 140 is also used to extract the first laser feature and the At least one piece of visual semantic information;
  • the control unit 140 is further configured to, for any sampling point C n , n is a positive integer, n
  • the positioning picture includes a first color channel used to store an encoding of visual semantic information.
  • the positioning picture further includes a second color channel for storing the encoding of the first laser characteristic.
  • the coding of visual semantic information includes at least one of flag bit, type coding and brightness coding; the flag bit is used to represent the type of the road marking, the type coding is used to represent the content of the road marking, and the brightness information coding is used for Used to represent the brightness information of the picture.
  • the control unit 140 when the control unit 140 is configured to extract at least one visual semantic information from the positioning map according to the current predicted pose of the vehicle: the control unit 140 is specifically configured to extract the local positioning from the positioning map according to the current predicted pose Map, the local positioning map includes M positioning pictures, the M positioning pictures include the first picture where the current predicted pose is located, and M-1 second pictures near the first picture, where M is a positive integer greater than 1; the control unit 140, which is further used for extracting at least one visual semantic information from the local localization map.
  • the control unit 140 when the control unit 140 is configured to match at least one visual semantic information with at least one road semantic information according to the current estimated pose P n for any sampling point C n to determine the current estimated pose P n
  • the control unit 140 is specifically configured to determine at least one valid pavement semantic information from the at least one pavement semantic information, and the number of pixels of each valid pavement semantic information is within a preset range; the control unit 140 is also configured to use projecting at least one valid pavement semantic information into the coordinate system of the local positioning map according to the current estimated pose P n ; the control unit 140 is further configured to determine the semantic association relationship between the at least one valid pavement semantic information and the at least one visual semantic information; The control unit 140 is further configured to perform semantic matching on each pair of semantically related valid pavement semantic information and visual semantic information, and determine the second weight according to the semantic matching result.
  • the control unit 140 when the control unit 140 is used to determine the semantic relationship between at least one valid pavement semantic information and at least one visual semantic information: the control unit 140 is specifically adapted to calculate the semantic weight of any valid pavement semantic information a i and the semantic weight of any visual semantic information b j ; the control unit 140 is further configured to determine the effective pavement semantic information a according to the difference between the semantic weight of the effective pavement semantic information a i and the semantic weight of the visual semantic information b j The semantic correlation degree between i and the visual semantic information b j ; the control unit 140 is further configured to determine that the effective road semantic information a i and the visual semantic information b j have semantic correlation when the semantic correlation degree is less than the preset first threshold.
  • control unit 140 when the control unit 140 is configured to perform semantic matching on each pair of semantically associated valid pavement semantic information and visual semantic information, and determine the second weight according to the semantic matching result: the control unit 140 is specifically configured to respectively Calculate the matching distance of each pair of semantically associated valid pavement semantic information and visual semantic information; the control unit 140 is also used for the weighted summation of the respective matching distances obtained by calculation to obtain the total matching distance; the control unit 140 is also used for according to The total matching distance determines the second weight.
  • the control unit 140 when the control unit 140 is configured to calculate the current estimated pose P according to the current estimated poses P 1 -PN of the N sampling points C 1 -CN and their first weights and second weights When the weighted average value of 1- PN is used as the current pose of the vehicle: the control unit 140 is specifically configured to use the first weight of the current estimated pose P 1 -PN to determine the current estimated pose P 1 -P N is weighted and averaged to obtain a first weighted average value; the control unit 140 is further configured to use the second weight of the current estimated pose P 1 -PN to weight the current estimated pose P 1 -PN , to obtain a second weighted average value; the control unit 140 is further configured to perform a weighted average of the first weighted average value and the second weighted average value to obtain a weighted average value, and use the weighted average value as the current pose of the vehicle.
  • the pavement semantic information includes: a pixel block containing at least one pavement marker, the number of pixels in the pixel block, and the type of pavement marker to which each pixel belongs.
  • the positioning system further includes an odometer 180; the control unit 140 is further configured to determine the relative pose of the vehicle between the current time t and the first historical time t-1 according to the data of the odometer 180; the control unit 140 is further configured to add the predicted pose corresponding to the sampling point C n at the first historical time t-1 and the relative pose to obtain the current predicted pose.
  • control unit 140 is further configured to regenerate N sampling points C 1 -C N when there is a preset ratio of the first weight or the second weight is lower than the second threshold.
  • the positioning system may implement corresponding functions through the software modules shown in FIG. 23 .
  • the positioning system may include a sampling point generation module 810 , an extraction module 820 , a first matching module 830 , a second matching module 840 , and a solving module 850 .
  • the functions of the above modules are described in detail below:
  • the sampling point generating module 810 is used for determining the initial pose of the vehicle, and generating N sampling points C 1 -C N around the initial pose, where N is a positive integer;
  • the extraction module 820 is configured to extract the first laser feature and at least one visual semantic information from the positioning map according to the current predicted pose of the vehicle.
  • the first matching module 830 is configured to, for any sampling point C n , where n is a positive integer, n ⁇ N, according to its corresponding current estimated pose P n , extract the first laser feature and the second laser feature of the self-positioning map Matching is performed to determine the first weight of the current estimated pose Pn , and the second laser feature is extracted from the point cloud data collected by the lidar.
  • the second matching module 840 is configured to, for any sampling point C n , match at least one visual semantic information with at least one pavement semantic information according to the current estimated pose P n to determine the second weight of the current estimated pose P n value, at least one pavement semantic information is extracted from the image data collected by the camera.
  • the solving module 850 is configured to calculate the weight of the current estimated pose P 1 -PN according to the current estimated pose P 1 -PN of the N sampling points C 1 -CN and its first weight and second weight The average value is calculated by weighting as the current pose of the vehicle.
  • the positioning map includes a plurality of positioning pictures that are spliced with each other, the positioning pictures include a color channel, and the encoding of the first laser feature and the encoding of the visual semantic information are stored in the color channel.
  • the positioning system provided by the embodiment of the present application can determine the first weight of the currently estimated pose based on the matching of laser features, and determine the current estimated pose based on the matching of visual semantic information and road semantic information (ie, matching of visual features).
  • the second weight and then calculate the weighted average of the current estimated pose according to the first weight and the second weight, and use the weighted average as the current pose of the vehicle, thereby realizing the laser positioning based on laser feature matching.
  • the technology is integrated with the visual positioning technology based on visual feature matching, which improves the positioning efficiency.
  • the technical solutions provided by the embodiments of the present application encode and store laser features and visual semantic information in the same positioning map, which realizes the fusion of map data and reduces the data overhead and computing overhead generated in the positioning process.
  • the embodiments of the present application further provide a vehicle, which may include the positioning system provided by the foregoing embodiments, and the user executes the positioning methods provided by the foregoing embodiments.
  • Embodiments of the present application further provide a computer-readable storage medium, where instructions are stored in the computer-readable storage medium, when the computer-readable storage medium runs on a computer, the computer executes the methods of the above aspects.
  • Embodiments of the present application also provide a computer program product containing instructions, which, when run on a computer, cause the computer to execute the methods of the above aspects.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Remote Sensing (AREA)
  • General Physics & Mathematics (AREA)
  • Electromagnetism (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Theoretical Computer Science (AREA)
  • Traffic Control Systems (AREA)
  • Navigation (AREA)

Abstract

A positioning method, a positioning system, and a vehicle. The positioning method comprises: generating N sampling points C1-CN around an initial posture; respectively determining, on the basis of matching of laser features, first weights of the current estimation postures corresponding to the sampling points; determining, on the basis of matching of visual semantic information and pavement semantic information, second weights of the current estimation postures corresponding to the sampling points; and then performing weighted averaging on the current estimation postures according to the first weights and the second weights to obtain an actual posture of the vehicle. Accordingly, a laser positioning technique based on a laser feature and a visual positioning technique based on a feature are fused, and positioning efficiency is improved. Moreover, by coding and storing the laser feature and the visual semantic information in the same positioning map, fusion of map data is achieved, and data overhead and calculation overhead generated in a positioning process are reduced. The positioning system corresponds to the positioning method. The vehicle comprises the positioning system.

Description

一种定位方法、定位***和车辆A positioning method, positioning system and vehicle
本申请要求于2020年08月28日提交到国家知识产权局、申请号为202010884916.3、发明名称为“一种定位方法、定位***和车辆”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。This application claims the priority of the Chinese patent application with the application number 202010884916.3 and the invention titled "A Positioning Method, Positioning System and Vehicle", which was submitted to the State Intellectual Property Office on August 28, 2020, the entire contents of which are incorporated by reference in this application.
技术领域technical field
本申请涉及自动驾驶技术领域，尤其涉及一种定位方法、定位***和车辆。The present application relates to the technical field of automatic driving, and in particular, to a positioning method, a positioning system and a vehicle.
背景技术Background technique
自动驾驶需要解决驾驶的三个核心问题：在哪里？(车辆定位)；去哪里？(确定目的地)；怎么去？(路径规划)。其中，定位技术主要用于解决“在哪里？”的问题，是实现自动驾驶必不可少的关键技术之一。Autonomous driving needs to address three core questions of driving: where? (vehicle location); where to go? (determine the destination); how to get there? (route plan). Among them, positioning technology is mainly used to solve the problem of "where?", and is one of the key technologies that are essential to realize autonomous driving.
基于信号的定位技术使用卫星信号或5G信号实现车辆的定位，因此具备全局定位的能力，是目前最广泛使用的定位技术。然而，由于GNSS的卫星信号容易受到高楼、山脉等遮挡的影响，使得车辆在行驶在城市、隧道等路况时，基于信号的定位技术无法提供准确的定位，因此不能满足实现全场景自动驾驶的需求。为解决这一问题，目前自动驾驶领域的研究方向是将基于信号的定位技术、基于航位推算的定位技术、基于特征匹配的定位技术进行融合，以弥补基于信号的定位技术的不足。Signal-based positioning technology uses satellite signals or 5G signals to achieve vehicle positioning, so it has the ability to global positioning and is currently the most widely used positioning technology. However, since the GNSS satellite signal is easily affected by the occlusion of tall buildings and mountains, the signal-based positioning technology cannot provide accurate positioning when the vehicle is driving in cities, tunnels and other road conditions, so it cannot meet the needs of full-scene autonomous driving. . To solve this problem, the current research direction in the field of autonomous driving is to integrate signal-based positioning technology, dead reckoning-based positioning technology, and feature matching-based positioning technology to make up for the shortcomings of signal-based positioning technology.
目前，对基于环境特征匹配的定位技术来说，基于激光特征匹配的激光定位技术补充或替代基于信号的定位技术，或基于视觉特征匹配的视觉定位技术补充或替代基于信号的定位技术，是两条独立的技术路线，激光定位和视觉定位所依赖的地图数据分别存储、算法独立运行，导致其数据存储开销和计算开销过大，对车辆的控制单元ECU等硬件***要求高，并且效率低下。At present, for the positioning technology based on environmental feature matching, the laser positioning technology based on laser feature matching supplements or replaces the signal-based positioning technology, or the visual positioning technology based on visual feature matching supplements or replaces the signal-based positioning technology. An independent technical route, the map data that laser positioning and visual positioning rely on are stored separately, and the algorithms run independently, resulting in excessive data storage overhead and computational overhead, high requirements for the vehicle's control unit ECU and other hardware systems, and low efficiency.
发明内容SUMMARY OF THE INVENTION
本申请实施例提供了一种定位方法、定位***和车辆，能够将基于激光特征匹配的激光定位技术和基于视觉特征匹配的视觉定位技术进行融合，以提高定位效率，并且能够减少定位过程中产生的数据和计算开销。The embodiments of the present application provide a positioning method, a positioning system, and a vehicle, which can integrate a laser positioning technology based on laser feature matching and a visual positioning technology based on visual feature matching, so as to improve positioning efficiency and reduce the occurrence of data and computational overhead.
第一方面，本申请实施例提供了一种定位方法，该方法包括：确定车辆的初始位姿，在初始位姿周围生成N个采样点C 1～C N，N为正整数；根据车辆的当前预测位姿P从定位地图中提取第一激光特征和至少一个视觉语义信息；对于任意采样点C n，n为正整数，n≤N，根据其对应的当前估计位姿P n，将提取自定位地图的第一激光特征与第二激光特征进行匹配，以确定当前估计位姿P n的第一权值，第二激光特征是从激光雷达采集的点云数据中提取的；以及，对于任意采样点C n，根据当前估计位姿P n，将至少一个视觉语义信息与至少一个路面语义信息进行匹配，以确定当前估计位姿P n的第二权值，至少一个路面语义信息是从摄像头采集的图像数据中提取的；根据N个采样点C 1～C N的当前估计位姿P 1-P N及其第一权值和第二权值，计算当前估计位姿P 1-P N的加权平均值，以加权求平均值作为车辆的当前位姿；其中，定位地图包括相互拼接的多张定位图片，定位图片包括色彩通道，第一激光特征的编码和视 觉语义信息的编码存储在色彩通道中。 In a first aspect, an embodiment of the present application provides a positioning method, the method includes: determining an initial pose of a vehicle, and generating N sampling points C 1 to C N around the initial pose, where N is a positive integer; The current predicted pose P extracts the first laser feature and at least one visual semantic information from the positioning map; for any sampling point C n , n is a positive integer, n≤N, according to its corresponding current estimated pose P n , the extracted The first laser feature of the self-localization map is matched with the second laser feature to determine the first weight of the current estimated pose P n , and the second laser feature is extracted from the point cloud data collected by the lidar; and, for At any sampling point C n , according to the current estimated pose P n , at least one visual semantic information is matched with at least one pavement semantic information to determine the second weight of the current estimated pose P n , and the at least one pavement semantic information is derived from It is extracted from the image data collected by the camera; according to the current estimated pose P 1 -PN of N sampling points C 1 -CN and its first weight and second weight, the current estimated pose P 1 - P is calculated The weighted average of N is taken as the current pose of the vehicle; wherein, the positioning map includes multiple positioning pictures spliced with each other, the positioning pictures include color channels, the encoding of the first laser feature and the encoding and storage of visual semantic information in the color channel.
本申请实施例提供的技术方案，能够分别基于激光特征的匹配确定当前估计位姿的第一权值，基于视觉语义信息和路面语义信息的匹配(即视觉特征的匹配)确定当前估计位姿的第二权值，然后根据的第一权值和第二权值计算当前估计位姿的加权平均值，以加权求平均值作为车辆的当前位姿，从而实现了将基于激光特征匹配的激光定位技术和基于视觉特征匹配的视觉定位技术进行融合，提高了定位效率。并且，本申请实施例提供的技术方案将激光特征和视觉语义信息编码存储在同一张定位地图中，实现了地图数据的融合，降低了定位过程中产生的数据开销和计算开销。The technical solutions provided by the embodiments of the present application can respectively determine the first weight of the currently estimated pose based on the matching of laser features, and determine the current estimated pose based on the matching of visual semantic information and road semantic information (ie, matching of visual features). The second weight, and then calculate the weighted average of the current estimated pose according to the first weight and the second weight, and use the weighted average as the current pose of the vehicle, thereby realizing the laser positioning based on laser feature matching. The technology is integrated with the visual positioning technology based on visual feature matching, which improves the positioning efficiency. In addition, the technical solutions provided by the embodiments of the present application encode and store laser features and visual semantic information in the same positioning map, which realizes the fusion of map data and reduces the data overhead and computing overhead generated in the positioning process.
在一种可选择的实现方式中，定位图片包括第一色彩通道，第一色彩通道用于存储视觉语义信息的编码。这样，本申请实施例提供的方法就可以对第一色彩通道进行解码以获取视觉语义信息。In an optional implementation manner, the positioning picture includes a first color channel, and the first color channel is used to store coding of visual semantic information. In this way, the method provided by the embodiment of the present application can decode the first color channel to obtain visual semantic information.
在一种可选择的实现方式中，定位图片还包括第二色彩通道，第二色彩通道用于存储第一激光特征的编码。这样，本申请实施例提供的方法就可以对第二色彩通道进行解码以获取第一激光特征。In an optional implementation, the positioning picture further includes a second color channel, where the second color channel is used to store the code of the first laser feature. In this way, the method provided by the embodiment of the present application can decode the second color channel to obtain the first laser feature.
在一种可选择的实现方式中，视觉语义信息的编码包括标志位、类型编码和亮度编码中的至少一种；标志位用于表示路面标志的类型，类型编码用于表示路面标志的内容，亮度信息编码用于表示图片的亮度信息。这样，通过视觉语义信息的编码就可以确定视觉语义信息包含什么样的路面标志，例如白色虚线、白色实线、直行标志等，以利于与路面语义信息进行匹配。In an optional implementation manner, the encoding of the visual semantic information includes at least one of a marker bit, a type encoding, and a brightness encoding; the marker bit is used to represent the type of the road marking, and the type encoding is used to represent the content of the road marking, Luma information encoding is used to represent the luma information of a picture. In this way, through the coding of the visual semantic information, it is possible to determine what kind of road signs the visual semantic information contains, such as white dotted lines, white solid lines, straight signs, etc., so as to facilitate matching with the road semantic information.
在一种可选择的实现方式中，至少一个视觉语义信息通过以下步骤提取：根据当前预测位姿从定位地图中提取局部定位地图，局部定位地图包含M张定位图片，M张定位图片包含当前预测位姿所在的第一图片，以及第一图片附近的M-1张第二图片，M为大于1的正整数；从局部定位地图中提取至少一个视觉语义信息。这样，视觉语义信息就可以包含当前预测位姿周围的路面标志等信息，以便于与摄像头采集到的车辆周围的图像数据中的路面语义信息进行匹配，以确定第二权值。In an optional implementation, at least one piece of visual semantic information is extracted by the following steps: extracting a local positioning map from the positioning map according to the current predicted pose, the local positioning map includes M positioning pictures, and the M positioning pictures include the current prediction The first picture where the pose is located, and M-1 second pictures near the first picture, where M is a positive integer greater than 1; at least one visual semantic information is extracted from the local positioning map. In this way, the visual semantic information can include information such as road signs around the current predicted pose, so as to be matched with the road semantic information in the image data around the vehicle collected by the camera to determine the second weight.
在一种可选择的实现方式中，对于任意采样点C n，根据当前估计位姿P n，将至少一个视觉语义信息与至少一个路面语义信息进行匹配，以确定当前估计位姿P n的第二权值，包括：从至少一个路面语义信息中确定至少一个有效路面语义信息，每一个有效路面语义信息的像素数量在预设范围内；根据当前估计位姿P n将至少一个有效路面语义信息投影到局部定位地图的坐标系中；确定至少一个有效路面语义信息和至少一个视觉语义信息的语义关联关系；对每一对语义关联的有效路面语义信息和视觉语义信息进行语义匹配，根据语义匹配结果确定第二权值。这样，通过确定有效路面语义信息可以滤除一些不完整或者尺寸过大的误检视觉语义信息，提高语义匹配效率，减小语义匹配的计算量。 In an optional implementation manner, for any sampling point C n , according to the current estimated pose P n , at least one piece of visual semantic information is matched with at least one piece of road semantic information, so as to determine the first position of the current estimated pose P n Two weights, including: determining at least one valid pavement semantic information from at least one pavement semantic information, and the number of pixels of each valid pavement semantic information is within a preset range; Projecting into the coordinate system of the local positioning map; determining the semantic relationship between at least one valid pavement semantic information and at least one visual semantic information; performing semantic matching on each pair of semantically related valid pavement semantic information and visual semantic information, according to the semantic matching As a result, the second weight is determined. In this way, some incomplete or oversized falsely detected visual semantic information can be filtered out by determining the effective pavement semantic information, so as to improve the efficiency of semantic matching and reduce the calculation amount of semantic matching.
在一种可选择的实现方式中，确定至少一个有效路面语义信息和至少一个视觉语义信息的语义关联关系，包括：计算任意有效路面语义信息a i的语义权值和任意视觉语义信息b j的语义权值；根据有效路面语义信息a i的语义权值和视觉语义信息b j的语义权值的差值，确定有效路面语义信息a i和视觉语义信息b j的语义关联度；当语义关联度小于预设第一阈值时，确定有效路面语义信息a i和视觉语义信息b j具有语义关联。这样，语义关联度小于第一阈值时，说明有效路面语义信息a i的语义权值和视觉语义信息b j的语义权值比较接近，进而说明有效路面语义信息a i和视觉语义信息b j具有语义关联。 In an optional implementation manner, determining the semantic relationship between at least one valid pavement semantic information and at least one visual semantic information includes: calculating the semantic weight of any valid pavement semantic information a i and the arbitrary visual semantic information b j Semantic weight; according to the difference between the semantic weight of the effective pavement semantic information a i and the semantic weight of the visual semantic information b j , the semantic correlation degree of the effective pavement semantic information a i and the visual semantic information b j is determined; when the semantic correlation When the degree is less than the preset first threshold, it is determined that the effective pavement semantic information a i and the visual semantic information b j have a semantic relationship. In this way, when the semantic relevance is less than the first threshold, it means that the semantic weight of the effective pavement semantic information a i and the semantic weight of the visual semantic information b j are relatively close, which further means that the effective pavement semantic information a i and the visual semantic information b j have Semantic association.
在一种可选择的实现方式中，对每一对语义关联的有效路面语义信息和视觉语义信息进行语义匹配，根据语义匹配结果确定第二权值，包括：分别计算每一对语义关联的有效路面语义信息和视觉语义信息的匹配距离；将计算得到的各个匹配距离的加权求和得到总匹配距离；根据总匹配距离确定第二权值。这样，第二权值能够体现出当前估计位姿P n与车辆真实位姿的接近程度。 In an optional implementation manner, semantic matching is performed on each pair of semantically associated valid pavement semantic information and visual semantic information, and the second weight is determined according to the semantic matching result, including: separately calculating the valid pavement semantic information for each pair of semantic associations. The matching distance between the pavement semantic information and the visual semantic information; the weighted summation of the calculated matching distances is used to obtain the total matching distance; the second weight is determined according to the total matching distance. In this way, the second weight can reflect the closeness of the current estimated pose P n to the real pose of the vehicle.
在一种可选择的实现方式中，根据N个采样点C 1～C N的当前估计位姿P 1-P N及其第一权值和第二权值，计算当前估计位姿P 1-P N的加权平均值，以加权求平均值作为车辆的当前位姿，包括：使用当前估计位姿P 1-P N的第一权值对当前估计位姿P 1-P N加权求平均，得到第一加权平均值；使用当前估计位姿P 1-P N的第二权值对当前估计位姿P 1-P N加权求平均，得到第二加权平均值；对第一加权平均值和第二加权平均值加权求平均，得到加权平均值，以加权求平均值作为车辆的当前位姿。这样，通过使用第一权值和第二权值对当前估计位姿加权求平均，实现了将基于激光特征匹配的激光定位技术和基于视觉特征匹配的视觉定位技术进行融合。 In an optional implementation manner, according to the current estimated poses P 1 -PN of the N sampling points C 1 -CN and their first weights and second weights, the current estimated pose P 1 - The weighted average of P N , and the weighted average value is taken as the current pose of the vehicle, including: using the first weight of the current estimated pose P 1 -P N to weight the current estimated pose P 1 -P N and averaging, Obtain the first weighted average; use the second weight of the current estimated pose P 1 -PN to weight the current estimated pose P 1 -PN to obtain the second weighted average; The second weighted average is weighted and averaged to obtain a weighted average, and the weighted average is used as the current pose of the vehicle. In this way, by using the first weight and the second weight to weight the current estimated pose, the fusion of the laser positioning technology based on laser feature matching and the visual positioning technology based on visual feature matching is realized.
在一种可选择的实现方式中，路面语义信息包括：包含有至少一个路面标志的像素块、像素块的像素数量、每个像素所属的路面标志的类型。这样，通过路面语义信息可以确定路面标志的类型、大小等，以利于与路面语义信息进行匹配。In an optional implementation manner, the pavement semantic information includes: a pixel block containing at least one pavement mark, the number of pixels in the pixel block, and the type of pavement mark to which each pixel belongs. In this way, the type, size, etc. of the pavement markings can be determined through the pavement semantic information, so as to facilitate matching with the pavement semantic information.
在一种可选择的实现方式中，当前预测位姿通过以下步骤确定：根据里程计数据确定车辆在当前时刻t和第一历史时刻t-1之间产生的相对位姿；将采样点C n在第一历史时刻t-1对应的预测位姿与相对位姿相加得到当前预测位姿。 In an optional implementation manner, the current predicted pose is determined by the following steps: determining the relative pose of the vehicle generated between the current time t and the first historical time t -1 according to the odometer data; The predicted pose corresponding to the first historical time t-1 is added to the relative pose to obtain the current predicted pose.
在一种可选择的实现方式中，当有预设比例的第一权值或者第二权值低于第二阈值时，重新生成N个采样点C 1～C N。这样，可以消除采样点随着定位次数的增加而出现的发散现象。 In an optional implementation manner, when there is a preset proportion of the first weight or the second weight is lower than the second threshold, the N sampling points C 1 -C N are regenerated. In this way, the divergence phenomenon of sampling points with the increase of positioning times can be eliminated.
第二方面，本申请实施例提供了一种定位***，包括：安装于车辆的GNSS/INS组合模块、控制单元、存储器、激光雷达和摄像头；GNSS/INS组合模块，用于确定车辆的初始位姿；存储器，用于存储定位地图，定位地图包括相互拼接的多张定位图片，定位图片包括色彩通道，色彩通道存储有第一激光特征的编码和视觉语义信息的编码；激光雷达，用于采集点云数据，点云数据包含第二激光特征；摄像头，用于采集图像数据，图像数据包含至少一个路面语义信息；控制单元，用于在初始位姿周围生成N个采样点C 1～C N，N为正整数；控制单元，还用于根据车辆的当前预测位姿从定位地图中提取第一激光特征和至少一个视觉语义信息；控制单元，还用于对于任意采样点C n，n为正整数，n≤N，根据其对应的当前估计位姿P n，将提取自定位地图的第一激光特征与第二激光特征进行匹配，以确定当前估计位姿P n的第一权值，第二激光特征是从激光雷达采集的点云数据中提取的；控制单元，还用于对于任意采样点C n，根据当前估计位姿P n，将至少一个视觉语义信息与至少一个路面语义信息进行匹配，以确定当前估计位姿P n的第二权值，至少一个路面语义信息是从摄像头采集的图像数据中提取的；控制单元，还用于根据N个采样点C 1～C N的当前估计位姿P 1-P N及其第一权值和第二权值，计算当前估计位姿P 1-P N的加权平均值，以加权求平均值作为车辆的当前位姿。 In a second aspect, an embodiment of the present application provides a positioning system, including: a GNSS/INS combination module, a control unit, a memory, a lidar, and a camera installed on a vehicle; and a GNSS/INS combination module for determining the initial position of the vehicle attitude; memory, used to store the positioning map, the positioning map includes multiple positioning pictures spliced with each other, the positioning picture includes a color channel, and the color channel stores the encoding of the first laser feature and the encoding of visual semantic information; lidar, used for collecting point cloud data, where the point cloud data includes the second laser feature; a camera for collecting image data, where the image data includes at least one pavement semantic information; a control unit for generating N sampling points C 1 to C N around the initial pose , N is a positive integer; the control unit is also used to extract the first laser feature and at least one visual semantic information from the positioning map according to the current predicted pose of the vehicle; the control unit is also used for any sampling point C n , n is A positive integer, n≤N, according to its corresponding current estimated pose P n , the first laser feature extracted from the localization map and the second laser feature are matched to determine the first weight of the current estimated pose P n , The second laser feature is extracted from the point cloud data collected by the lidar; the control unit is further configured to, for any sampling point C n , associate the at least one visual semantic information with the at least one pavement semantic information according to the current estimated pose P n . Matching is performed to determine the second weight of the current estimated pose P n , and at least one road semantic information is extracted from the image data collected by the camera; the control unit is further configured to The current estimated pose P 1 -PN and its first weight and second weight are calculated, and the weighted average of the current estimated pose P 1 -PN is calculated, and the weighted average is used as the current pose of the vehicle.
本申请实施例提供的技术方案，能够分别基于激光特征的匹配确定当前估计位姿的第一权值，基于视觉语义信息和路面语义信息的匹配(即视觉特征的匹配)确定当前估计位姿的第二权值，然后根据的第一权值和第二权值计算当前估计位姿的加权平均值，以加权求平均值作为车辆的当前位姿，从而实现了将基于激光特征匹配的激光定位技术和基于视觉特征匹配的视觉定位技术进行融合，提高了定位效率。并且，本申请实施例提供的技术方案将激光 特征和视觉语义信息编码存储在同一张定位地图中，实现了地图数据的融合，降低了定位过程中产生的数据开销和计算开销。The technical solutions provided by the embodiments of the present application can respectively determine the first weight of the currently estimated pose based on the matching of laser features, and determine the current estimated pose based on the matching of visual semantic information and road semantic information (ie, matching of visual features). The second weight, and then calculate the weighted average of the current estimated pose according to the first weight and the second weight, and use the weighted average as the current pose of the vehicle, thereby realizing the laser positioning based on laser feature matching. The technology is integrated with the visual positioning technology based on visual feature matching, which improves the positioning efficiency. In addition, the technical solutions provided by the embodiments of the present application encode and store laser features and visual semantic information in the same positioning map, which realizes the fusion of map data and reduces the data overhead and computing overhead generated in the positioning process.
在一种可选择的实现方式中，定位图片包括第一色彩通道，第一色彩通道用于存储视觉语义信息的编码。这样，本申请实施例提供的定位***就可以对第一色彩通道进行解码以获取视觉语义信息。In an optional implementation manner, the positioning picture includes a first color channel, and the first color channel is used to store coding of visual semantic information. In this way, the positioning system provided by the embodiment of the present application can decode the first color channel to obtain visual semantic information.
在一种可选择的实现方式中，定位图片还包括第二色彩通道，第二色彩通道用于存储第一激光特征的编码。这样，本申请实施例提供的定位***就可以对第二色彩通道进行解码以获取第一激光特征。In an optional implementation, the positioning picture further includes a second color channel, where the second color channel is used to store the code of the first laser feature. In this way, the positioning system provided in this embodiment of the present application can decode the second color channel to obtain the first laser feature.
在一种可选择的实现方式中，视觉语义信息的编码包括标志位、类型编码和亮度编码中的至少一种；标志位用于表示路面标志的类型，类型编码用于表示路面标志的内容，亮度信息编码用于表示图片的亮度信息。这样，通过视觉语义信息的编码就可以确定视觉语义信息包含什么样的路面标志，例如白色虚线、白色实线、直行标志等，以利于与路面语义信息进行匹配。In an optional implementation manner, the encoding of the visual semantic information includes at least one of a marker bit, a type encoding, and a brightness encoding; the marker bit is used to represent the type of the road marking, and the type encoding is used to represent the content of the road marking, Luma information encoding is used to represent the luma information of a picture. In this way, through the coding of the visual semantic information, it is possible to determine what kind of road signs the visual semantic information contains, such as white dotted lines, white solid lines, straight signs, etc., so as to facilitate matching with the road semantic information.
在一种可选择的实现方式中，当控制单元用于根据车辆的当前预测位姿从定位地图中提取至少一个视觉语义信息时：控制单元，具体用于根据当前预测位姿从定位地图中提取局部定位地图，局部定位地图包含M张定位图片，M张定位图片包含当前预测位姿所在的第一图片，以及第一图片附近的M-1张第二图片，M为大于1的正整数；控制单元，还用于从局部定位地图中提取至少一个视觉语义信息。这样，视觉语义信息就可以包含当前预测位姿周围的路面标志等信息，以便于与摄像头采集到的车辆周围的图像数据中的路面语义信息进行匹配，以确定语义关联关系和第二权值。In an optional implementation manner, when the control unit is used to extract at least one visual semantic information from the positioning map according to the current predicted pose of the vehicle: the control unit is specifically configured to extract from the positioning map according to the current predicted pose Local positioning map, the local positioning map includes M positioning pictures, the M positioning pictures include the first picture where the current predicted pose is located, and M-1 second pictures near the first picture, where M is a positive integer greater than 1; The control unit is further used for extracting at least one visual semantic information from the local localization map. In this way, the visual semantic information can include information such as road signs around the current predicted pose, so as to be matched with the road semantic information in the image data around the vehicle collected by the camera to determine the semantic relationship and the second weight.
在一种可选择的实现方式中，当控制单元用于对于任意采样点C n，根据当前估计位姿P n，将至少一个视觉语义信息与至少一个路面语义信息进行匹配，以确定当前估计位姿P n的第二权值时：控制单元，具体用于从至少一个路面语义信息中确定至少一个有效路面语义信息，每一个有效路面语义信息的像素数量在预设范围内；控制单元，还用于根据当前估计位姿P n将至少一个有效路面语义信息投影到局部定位地图的坐标系中；控制单元，还用于确定至少一个有效路面语义信息和至少一个视觉语义信息的语义关联关系；控制单元，还用于对每一对语义关联的有效路面语义信息和视觉语义信息进行语义匹配，根据语义匹配结果确定第二权值。这样，通过确定有效路面语义信息可以滤除一些不完整或者尺寸过大的误检视觉语义信息，提高语义匹配效率，减小语义匹配的计算量。 In an optional implementation manner, when the control unit is configured to match at least one visual semantic information with at least one pavement semantic information according to the current estimated pose P n for any sampling point C n to determine the current estimated position When the second weight of the pose P n is: the control unit, specifically configured to determine at least one valid road semantic information from the at least one road semantic information, and the number of pixels of each valid road semantic information is within a preset range; the control unit, also is used for projecting at least one valid pavement semantic information into the coordinate system of the local positioning map according to the current estimated pose P n ; the control unit is further adapted to determine the semantic association relationship between at least one valid pavement semantic information and at least one visual semantic information; The control unit is further configured to perform semantic matching on each pair of semantically associated valid pavement semantic information and visual semantic information, and determine the second weight according to the semantic matching result. In this way, some incomplete or oversized falsely detected visual semantic information can be filtered out by determining the effective pavement semantic information, so as to improve the efficiency of semantic matching and reduce the calculation amount of semantic matching.
在一种可选择的实现方式中，当控制单元用于确定至少一个有效路面语义信息和至少一个视觉语义信息的语义关联关系时：控制单元，具体用于计算任意有效路面语义信息a i的语义权值和任意视觉语义信息b j的语义权值；控制单元，还用于根据有效路面语义信息a i的语义权值和视觉语义信息b j的语义权值的差值，确定有效路面语义信息a i和视觉语义信息b j的语义关联度；控制单元，还用于当语义关联度小于预设第一阈值时，确定有效路面语义信息a i和视觉语义信息b j具有语义关联。这样，语义关联度小于第一阈值时，说明有效路面语义信息a i的语义权值和视觉语义信息b j的语义权值比较接近，进而说明有效路面语义信息a i和视觉语义信息b j具有语义关联。 In an optional implementation manner, when the control unit is used to determine the semantic relationship between at least one valid pavement semantic information and at least one visual semantic information: the control unit is specifically adapted to calculate the semantics of any valid pavement semantic information a i weight and the semantic weight of any visual semantic information b j ; the control unit is also used to determine the effective pavement semantic information according to the difference between the semantic weight of the effective pavement semantic information a i and the semantic weight of the visual semantic information b j The semantic correlation degree of a i and the visual semantic information b j ; the control unit is further configured to determine that the effective pavement semantic information a i and the visual semantic information b j have semantic correlation when the semantic correlation degree is less than the preset first threshold. In this way, when the semantic relevance is less than the first threshold, it means that the semantic weight of the effective pavement semantic information a i and the semantic weight of the visual semantic information b j are relatively close, which further means that the effective pavement semantic information a i and the visual semantic information b j have Semantic association.
在一种可选择的实现方式中，当控制单元用于对每一对语义关联的有效路面语义信息和视觉语义信息进行语义匹配，根据语义匹配结果确定第二权值时：控制单元，具体用于分别计算每一对语义关联的有效路面语义信息和视觉语义信息的匹配距离；控制单元，还用于将 计算得到的各个匹配距离的加权求和得到总匹配距离；控制单元，还用于根据总匹配距离确定第二权值。这样，第二权值能够体现出当前预测位姿P n与车辆真实位姿的接近程度。 In an optional implementation manner, when the control unit is used to perform semantic matching on each pair of semantically associated valid pavement semantic information and visual semantic information, and determine the second weight according to the semantic matching result: the control unit, specifically using It is used to calculate the matching distance of each pair of semantically associated effective pavement semantic information and visual semantic information respectively; the control unit is also used to calculate the weighted sum of each matching distance to obtain the total matching distance; the control unit is also used to The total matching distance determines the second weight. In this way, the second weight can reflect the closeness of the current predicted pose P n to the actual pose of the vehicle.
在一种可选择的实现方式中，当控制单元用于根据N个采样点C 1～C N的当前估计位姿P 1-P N及其第一权值和第二权值，计算当前估计位姿P 1-P N的加权平均值，以加权求平均值作为车辆的当前位姿时：控制单元，具体用于使用当前估计位姿P 1-P N的第一权值对当前估计位姿P 1-P N加权求平均，得到第一加权平均值；控制单元，还用于使用当前估计位姿P 1-P N的第二权值对当前估计位姿P 1-P N加权求平均，得到第二加权平均值；控制单元，还用于对第一加权平均值和第二加权平均值加权求平均，得到加权平均值，以加权求平均值作为车辆的当前位姿。这样，通过使用第一权值和第二权值对当前估计位姿加权求平均，实现了将基于激光特征匹配的激光定位技术和基于视觉特征匹配的视觉定位技术进行融合。 In an optional implementation manner, when the control unit is used to calculate the current estimated poses P 1 -PN and their first weights and second weights according to the current estimated poses P 1 -PN of the N sampling points C 1 -CN The weighted average of the poses P 1 - P N , when the weighted average is used as the current pose of the vehicle: the control unit is specifically configured to use the first weight of the current estimated pose P 1 -PN to determine the current estimated pose. The poses P 1 -PN are weighted and averaged to obtain a first weighted average value; the control unit is further configured to use the second weight of the current estimated pose P 1 -PN to weight the current estimated pose P 1 -PN averaging to obtain a second weighted average value; the control unit is further configured to perform a weighted average of the first weighted average value and the second weighted average value to obtain a weighted average value, and use the weighted average value as the current pose of the vehicle. In this way, by using the first weight and the second weight to weight the current estimated pose, the fusion of the laser positioning technology based on laser feature matching and the visual positioning technology based on visual feature matching is realized.
在一种可选择的实现方式中，路面语义信息包括：包含有至少一个路面标志的像素块、像素块的像素数量、每个像素所属的路面标志的类型。这样，通过路面语义信息可以确定路面标志的类型、大小等，以利于与路面语义信息进行匹配。In an optional implementation manner, the pavement semantic information includes: a pixel block containing at least one pavement mark, the number of pixels in the pixel block, and the type of pavement mark to which each pixel belongs. In this way, the type, size, etc. of the pavement markings can be determined through the pavement semantic information, so as to facilitate matching with the pavement semantic information.
在一种可选择的实现方式中，定位***还包括里程计；控制单元，还用于根据里程计数据确定车辆在当前时刻t和第一历史时刻t-1之间产生的相对位姿；控制单元，还用于将采样点C n在第一历史时刻t-1对应的预测位姿与相对位姿相加得到当前预测位姿。 In an optional implementation manner, the positioning system further includes an odometer; the control unit is further configured to determine the relative pose of the vehicle between the current time t and the first historical time t-1 according to the odometer data; control The unit is also used to add the predicted pose corresponding to the sampling point C n at the first historical time t-1 and the relative pose to obtain the current predicted pose.
在一种可选择的实现方式中，控制单元，还用于当有预设比例的第一权值或者第二权值低于第二阈值时，重新生成N个采样点C 1～C N。这样，可以消除采样点随着定位次数的增加而出现的发散现象。 In an optional implementation manner, the control unit is further configured to regenerate the N sampling points C 1 -C N when there is a preset ratio of the first weight or the second weight is lower than the second threshold. In this way, the divergence phenomenon of sampling points with the increase of positioning times can be eliminated.
第三方面，本申请实施例提供了一种车辆，该车辆包括本申请实施例第二方面及其各个实现方式提供的定位***。In a third aspect, an embodiment of the present application provides a vehicle, and the vehicle includes the positioning system provided by the second aspect of the embodiment of the present application and various implementations thereof.
第四方面，本申请实施例还提供一种计算机可读存储介质，计算机可读存储介质中存储有指令，当其在计算机上运行时，使得计算机执行上述各方面及其各个实现方式的方法。In a fourth aspect, embodiments of the present application further provide a computer-readable storage medium, where instructions are stored in the computer-readable storage medium, and when the computer-readable storage medium runs on a computer, the computer executes the methods of the above aspects and their respective implementations.
第五方面，本申请实施例还提供了一种包含指令的计算机程序产品，当其在计算机上运行时，使得计算机执行上述各方面及其各个实现方式的方法。In a fifth aspect, the embodiments of the present application further provide a computer program product including instructions, which, when run on a computer, enables the computer to execute the methods of the above aspects and their respective implementations.
第六方面，本申请实施例还提供了一种芯片***，该芯片***包括处理器，用于支持上述装置或***实现上述方面中所涉及的功能，例如，生成或处理上述方法中所涉及的信息。In a sixth aspect, an embodiment of the present application further provides a chip system, where the chip system includes a processor for supporting the above-mentioned apparatus or system to implement the functions involved in the above-mentioned aspects, for example, generating or processing the functions involved in the above-mentioned method. information.
附图说明Description of drawings
图1是点云式定位地图的示意图；Fig. 1 is the schematic diagram of point cloud type positioning map;
图2是图片式定位地图的示意图；Fig. 2 is the schematic diagram of picture type positioning map;
图3是稀疏特征地图、半稠密地图和稠密地图的像素量示意图；Figure 3 is a schematic diagram of the amount of pixels of a sparse feature map, a semi-dense map and a dense map;
图4是自动驾驶车辆的定位***模块配置图；4 is a configuration diagram of a positioning system module of an autonomous vehicle;
图5是激光地图的示意图；5 is a schematic diagram of a laser map;
图6是本申请实施例提供的视觉语义信息的编码格式图；Fig. 6 is a coding format diagram of visual semantic information provided by an embodiment of the present application;
图7是定位地图上的不同区域的像素对应的视觉语义信息的8比特编码的示意图；7 is a schematic diagram of 8-bit encoding of visual semantic information corresponding to pixels in different regions on the positioning map;
图8是本申请实施例提供用于实现定位方法的定位***硬件框架图；8 is a hardware framework diagram of a positioning system for implementing a positioning method provided by an embodiment of the present application;
图9是本申请实施例提供的定位方法的流程图；9 is a flowchart of a positioning method provided by an embodiment of the present application;
图10是本申请实施例提供的定位方法涉及到的数据流转框图；10 is a block diagram of data flow involved in the positioning method provided by the embodiment of the present application;
图11示例性地提供了生成采样点的方案；Figure 11 exemplarily provides a scheme for generating sampling points;
图12是里程计坐标系的示意图；Figure 12 is a schematic diagram of an odometer coordinate system;
图13是本申请实施例提供的确定第一权值的过程示意图；13 is a schematic diagram of a process for determining a first weight provided by an embodiment of the present application;
图14是本申请实施例提供的确定第二权值的过程示意图；14 is a schematic diagram of a process for determining a second weight provided by an embodiment of the present application;
图15是本申请实施例提供的语义匹配的流程图；15 is a flowchart of semantic matching provided by an embodiment of the present application;
图16是本申请实施例提供的定位方法的步骤S201的流程图；FIG. 16 is a flowchart of step S201 of the positioning method provided by the embodiment of the present application;
图17是本申请实施例提供的计算路面语义信息的像素面积的示意图；17 is a schematic diagram of calculating the pixel area of road semantic information provided by an embodiment of the present application;
图18是本申请实施例提供的定位方法的步骤S203的流程图；FIG. 18 is a flowchart of step S203 of the positioning method provided by the embodiment of the present application;
图19是本申请实施例提供的定位方法的步骤S204的流程图；FIG. 19 is a flowchart of step S204 of the positioning method provided by the embodiment of the present application;
图20是本申请实施例提供的匹配距离的示意图；20 is a schematic diagram of a matching distance provided by an embodiment of the present application;
图21是本申请实施例提供的定位方法的步骤S502的流程图；21 is a flowchart of step S502 of the positioning method provided by the embodiment of the present application;
图22是本申请实施例提供的定位方法的步骤S105的流程图；FIG. 22 is a flowchart of step S105 of the positioning method provided by the embodiment of the present application;
图23是本申请实施例提供的定位***的软件模块框图。FIG. 23 is a block diagram of software modules of a positioning system provided by an embodiment of the present application.
具体实施方式detailed description
自动驾驶(autonomous vehicles或self-driving automobile)，也称无人驾驶、电脑驾驶等。自动驾驶能够以传感器、全球导航卫星***(global navigation satellite system，GNSS)及机器视觉等技术感测其周围环境，并根据感测数据确定自身位置、规划导航路线、更新地图信息、躲避障碍等，最终实现在没有任何人类主动操作或者少有人类主动操作的情况下自动地驾驶车辆。Autonomous vehicles (autonomous vehicles or self-driving automobiles), also known as unmanned driving, computer driving, etc. Autonomous driving can sense its surrounding environment with technologies such as sensors, global navigation satellite system (GNSS) and machine vision, and determine its own position, plan navigation routes, update map information, and avoid obstacles based on the sensing data. The ultimate goal is to drive the vehicle autonomously with little or no human activity.
一般来说，自动驾驶需要解决驾驶的三个核心问题：在哪里？(车辆定位)；去哪里？(确定目的地)；怎么去？(路径规划)。其中，定位技术主要用于解决“在哪里？”的问题，是实现自动驾驶必不可少的关键技术之一。Generally speaking, autonomous driving needs to solve three core problems of driving: where? (vehicle location); where to go? (determine the destination); how to get there? (route plan). Among them, positioning technology is mainly used to solve the problem of "where?", and is one of the key technologies that are essential to realize autonomous driving.
目前，根据所依赖的传感器不同，自动驾驶的定位技术主要可以包括以下三种：At present, depending on the sensors they rely on, the positioning technologies for autonomous driving mainly include the following three types:
1、基于信号的定位技术。1. Signal-based positioning technology.
该技术主要基于卫星信号或5G信号实现车辆的定位。目前主流的方案是在车辆内安装全球卫星导航***(global navigation satellite system，GNSS)接收机，以接收多个GNSS的卫星信号，利用接收的卫星信号计算出车辆在空间环境中的全局位置，并且也可以利用GNSS地面站配合GNSS卫星以提高定位精度。常见的GNSS***例如：北斗卫星导航***(beidou navigation satellite system，BDS)，全球卫星定位***(global positioning system，GPS)等。This technology is mainly based on satellite signals or 5G signals to achieve vehicle positioning. The current mainstream solution is to install a global navigation satellite system (GNSS) receiver in the vehicle to receive multiple GNSS satellite signals, and use the received satellite signals to calculate the global position of the vehicle in the space environment, and It is also possible to use GNSS ground stations to cooperate with GNSS satellites to improve positioning accuracy. Common GNSS systems such as: Beidou navigation satellite system (BDS), global positioning system (global positioning system, GPS) and so on.
2、基于航位推算的定位技术。2. Positioning technology based on dead reckoning.
该技术需要车辆配备惯性测量单元(inertial measurement unit，IMU)和轮速计等传感器。其中，IMU可以测量车辆行驶的角速度、加速度等信息，轮速计则可以测量车轮的转速。基于传感器数据，在确定了车辆的初始位置之后，则可以根据车辆的动力学方程，估计出车辆的当前位姿(位置和姿态)。The technology requires the vehicle to be equipped with sensors such as an inertial measurement unit (IMU) and wheel speedometers. Among them, the IMU can measure the angular velocity and acceleration of the vehicle, and the wheel speedometer can measure the rotational speed of the wheels. Based on the sensor data, after the initial position of the vehicle is determined, the current pose (position and attitude) of the vehicle can be estimated according to the dynamic equation of the vehicle.
3、基于环境特征匹配的定位技术。3. Positioning technology based on environmental feature matching.
目前，该技术主要包括激光定位和视觉定位两种形式，分别基于激光传感器和视觉传感器实时获取车辆周围的环境信息，将获取的环境信息处理后与预先存储的定位地图进行匹配，从而确定车辆的位姿。可以理解的是，该技术的实现需要预先构建定位地图，根据使用的传感器的不同，定位地图也具有不同的构建方式。At present, this technology mainly includes two forms: laser positioning and visual positioning. Based on the laser sensor and the visual sensor, the environmental information around the vehicle is obtained in real time, and the obtained environmental information is processed and matched with the pre-stored positioning map to determine the vehicle's location. pose. It is understandable that the implementation of this technology needs to build a positioning map in advance, and the positioning map also has different construction methods according to the different sensors used.
当采用激光定位时，定位地图主要有点云式和图片式。When using laser positioning, the positioning map is mainly cloud-like and picture-like.
①点云式：利用激光传感器采集点云数据，然后对点云数据进行过滤等操作以去除噪声，最后将处理后的点云数据拼接叠加到一起，形成如图1所示的点云地图。①Point cloud type: Use laser sensors to collect point cloud data, and then filter the point cloud data to remove noise. Finally, the processed point cloud data are spliced and superimposed together to form a point cloud map as shown in Figure 1.
②图片式：将①中拼接后的点云数据进行栅格处理，将每一个栅格对应为一个像素进行编码，以将拼接后的点云数据转化成如图2所示的图片格式的定位地图。其中，图片格式的激光地图可以是单通道的灰度图片，也可以是三通道的彩色图片。图片式的定位地图占用存储空间小，可以解决点云地图存储资源开销大的问题。②Picture format: Perform grid processing on the point cloud data spliced in ①, and encode each grid as a pixel, so as to convert the stitched point cloud data into the positioning of the picture format as shown in Figure 2 map. The laser map in the picture format can be a single-channel grayscale picture or a three-channel color picture. The image-based positioning map occupies a small storage space, which can solve the problem of high storage resource overhead of the point cloud map.
当采用视觉定位时，根据构建时从视觉传感器采集的原始图像中选取像素的方式和数量多少，定位地图(视觉地图)如图3所示，可以包括稀疏特征地图、半稠密地图和稠密地图三种形式，目前也主要以点云形式保存。When visual positioning is used, according to the way and number of pixels selected from the original image collected by the visual sensor during construction, the positioning map (visual map) is shown in Figure 3, which can include sparse feature map, semi-dense map and dense map three At present, it is mainly saved in the form of point cloud.
①稀疏特征地图：将原始图像中的特征点对应的像素保存到地图中，因此像素数量最少。① Sparse feature map: The pixels corresponding to the feature points in the original image are saved to the map, so the number of pixels is the least.
②半稠密地图：将原始图像中的部分像素，例如带有梯度的像素保存到地图中，因此像素数量中等。②Semi-dense map: Save some pixels in the original image, such as pixels with gradients, into the map, so the number of pixels is medium.
③稠密地图：将原始图像中的全部像素保存到地图中，因此像素数量最多。③Dense map: Save all the pixels in the original image to the map, so the number of pixels is the largest.
基于信号的定位技术由于具备全局定位的能力，是目前最广泛使用的定位技术。然而，由于GNSS的卫星信号容易受到高楼、山脉等遮挡的影响，使得车辆在行驶在城市、隧道等路况时，基于信号的定位技术无法提供准确的定位，因此不能满足实现全场景自动驾驶的需求。为解决这一问题，目前自动驾驶领域的研究方向是将基于信号的定位技术、基于航位推算的定位技术、基于特征匹配的定位技术进行融合，以弥补基于信号的定位技术的不足。Signal-based positioning technology is currently the most widely used positioning technology due to its global positioning capability. However, since the GNSS satellite signal is easily affected by the occlusion of tall buildings and mountains, the signal-based positioning technology cannot provide accurate positioning when the vehicle is driving in cities, tunnels and other road conditions, so it cannot meet the needs of full-scene autonomous driving. . To solve this problem, the current research direction in the field of autonomous driving is to integrate signal-based positioning technology, dead reckoning-based positioning technology, and feature matching-based positioning technology to make up for the shortcomings of signal-based positioning technology.
目前，对基于环境特征匹配的定位技术来说，采用激光特征匹配的激光定位技术补充或替代基于信号的定位技术，以及，采用视觉特征匹配的视觉定位技术补充或替代基于信号的定位技术，是目前融合定位的两条独立的技术路线。由于技术路线相互独立，激光定位和视觉定位的地图数据分别存储、算法相互独立，导致其数据存储开销和计算开销过大，对车辆的控制单元ECU等硬件***要求高，并且效率低下。At present, for the positioning technology based on environmental feature matching, the laser positioning technology using laser feature matching supplements or replaces the signal-based positioning technology, and the visual positioning technology using visual feature matching supplements or replaces the signal-based positioning technology. There are currently two independent technical routes for fusion positioning. Since the technical routes are independent of each other, the map data of laser positioning and visual positioning are stored separately, and the algorithms are independent of each other, resulting in excessive data storage and calculation overhead, high requirements for the vehicle's control unit ECU and other hardware systems, and low efficiency.
为解决现有技术中存在的问题，本申请实施例提供了一种定位方法。In order to solve the problems existing in the prior art, the embodiments of the present application provide a positioning method.
本申请实施例的技术方案可以应用在各种采用自动驾驶技术或者定位技术的载具之上，包括但不限于各种交通工具：例如车辆(汽车)、轮船、火车、地铁、飞机等，以及各种机器人，例如：服务机器人、运输机器人、自主导引机器人(automated guided vehicle，AGV)、无人地面车(unmanned ground vehicle，UGV)等，以及各种工程机械，例如：隧道掘进机等。The technical solutions of the embodiments of the present application can be applied to various vehicles using automatic driving technology or positioning technology, including but not limited to various vehicles: such as vehicles (cars), ships, trains, subways, airplanes, etc., and Various robots, such as service robots, transport robots, automated guided vehicles (AGVs), unmanned ground vehicles (UGVs), etc., as well as various construction machinery, such as tunnel boring machines.
下面以车辆为例，对本申请实施例的技术方案所实施的硬件环境进行说明。The hardware environment implemented by the technical solutions of the embodiments of the present application will be described below by taking a vehicle as an example.
如图4所示，该车辆配置有以下模块：激光雷达LiDAR 110、摄像头120、GNSS/惯性导航***(inertial navigation system，INS)组合模块130、控制单元140。其中：As shown in FIG. 4 , the vehicle is configured with the following modules: a lidar LiDAR 110 , a camera 120 , a GNSS/inertial navigation system (INS) combined module 130 , and a control unit 140 . in:
激光雷达110，用于采集环境周围的元素(例如：车辆、障碍物、行人、路牌等)距离本车辆的距离信息。激光雷达110可以对环境周围进行360度的全向扫描，也可以只扫描车头前方部分范围(例如180度)内的环境信息。The lidar 110 is used to collect the distance information of the elements around the environment (for example: vehicles, obstacles, pedestrians, street signs, etc.) from the vehicle. The lidar 110 can perform omnidirectional scanning of 360 degrees around the environment, or can only scan environmental information within a partial range (eg, 180 degrees) in front of the front of the vehicle.
摄像头120，用于采集车辆周围的图像信息。摄像头120可以具有一个(即单目摄像头)或者多个(即多目摄像头)，可以对周围进行360度的全景图像采集，也可以只采集车头前方部分范围的图像。The camera 120 is used to collect image information around the vehicle. The camera 120 may have one (that is, a monocular camera) or multiple (that is, a multi-eye camera), and may capture a 360-degree panoramic image of the surroundings, or may only capture an image of a partial range in front of the front of the vehicle.
GNSS/INS组合模块130，可以包括GNSS接收机和惯性测量单元IMU等器件，用于实现基于卫星信号和IMU的融合定位。The GNSS/INS combination module 130 may include devices such as a GNSS receiver and an inertial measurement unit IMU, and is used to implement fusion positioning based on satellite signals and IMU.
控制单元140，可以是整个自动驾驶车辆的电子***的核心计算单元，例如：移动数据 中心(mobile data center，MDC)、电子控制单元(electronic control unit，ECU)等，用于处理其他各个模块产生的数据，并基于处理结果生成车辆的控制信息。The control unit 140 may be the core computing unit of the electronic system of the entire autonomous vehicle, such as: a mobile data center (mobile data center, MDC), an electronic control unit (electronic control unit, ECU), etc., for processing other various modules to generate data, and generate vehicle control information based on the processing results.
此外，为实现车辆的其他辅助驾驶功能，该车辆还可以配置以下模块：In addition, in order to realize other assisted driving functions of the vehicle, the vehicle can also be configured with the following modules:
超声波传感器150，用于短距离测距，例如在辅助泊车时开启，以提供短距离告警信息等。The ultrasonic sensor 150 is used for short-distance ranging, for example, it is turned on when assisting in parking, so as to provide short-distance warning information and the like.
毫米波雷达160，用于长距离测距，由于毫米波抗干扰能力强，穿透雾、烟和尘土等能力强，因此毫米波雷达可以全天候工作，例如用于在恶劣天气情况下辅助对障碍物测距。Millimeter-wave radar 160, used for long-distance ranging, due to the strong anti-jamming ability of millimeter-wave and the ability to penetrate fog, smoke and dust, the millimeter-wave radar can work around the clock, for example, to assist in obstacle detection in bad weather Object ranging.
可以理解的是，本申请实施例示意的硬件环境并不构成对本申请实施例的技术方案的具体限定。在本申请另一些实施例中，本申请实施例的技术方案所实施的硬件环境可以包括比图示更多或更少的部件，或者组合某些部件，或者拆分某些部件，或者不同的部件布置。图示的部件可以以硬件，软件或软件和硬件的组合实现。It can be understood that the hardware environment illustrated by the embodiments of the present application does not constitute a specific limitation on the technical solutions of the embodiments of the present application. In other embodiments of the present application, the hardware environment implemented by the technical solutions of the embodiments of the present application may include more or less components than shown in the figure, or combine some components, or separate some components, or different Component placement. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.
本申请实施例中，为实现激光定位与视觉定位技术的融合，减少存储开销和计算开销，基本思路是：将激光定位的定位地图(以下简称激光地图)与视觉定位的定位地图(以下简称视觉地图)中的地图数据融合进一张定位地图中，这样在同时应用激光定位与视觉定位技术时，就可以从融合后的定位地图中获取这两种定位方式需要的数据，从而减少数据存储的开销和计算开销，提高定位效率。In the embodiments of the present application, in order to realize the fusion of laser positioning and visual positioning technologies and reduce storage overhead and computing overhead, the basic idea is: a positioning map for laser positioning (hereinafter referred to as a laser map) and a positioning map for visual positioning (hereinafter referred to as a visual positioning map) The map data in the map) is fused into a positioning map, so that when the laser positioning and visual positioning technologies are applied at the same time, the data required for these two positioning methods can be obtained from the fused positioning map, thereby reducing the amount of data storage. Overhead and computational overhead, improve positioning efficiency.
下面结合一些示例具体阐述本申请实施例是如何得到定位地图的。The following describes in detail how the embodiments of the present application obtain the positioning map with reference to some examples.
本申请实施例得到定位地图的基本思路是：将路面标志的视觉语义信息整合添加到激光地图中，以得到定位地图。其中，路面标志包括用于辅助确定车辆的位置和所在车道等信息，例如：单黄线、双黄线、白色虚线、白色实线、直行标志、左转标志、右转标志、调头标志等。为减小存储开销，本申请实施例优选使用图片格式的激光地图。激光地图由可以单色彩通道的灰度图片构成，也可以由多色彩通道的彩色图片构成。The basic idea of obtaining the positioning map in the embodiment of the present application is to integrate and add the visual semantic information of the road markings to the laser map, so as to obtain the positioning map. Among them, the road signs include information used to assist in determining the position and lane of the vehicle, such as: single yellow line, double yellow line, white dotted line, white solid line, straight sign, left turn sign, right turn sign, U-turn sign, etc. To reduce storage overhead, the embodiment of the present application preferably uses a laser map in a picture format. Laser maps are composed of grayscale images with single color channel or color images with multiple color channels.
在一个实施例中，如图5所示，激光地图以三通道的彩色图片实现，例如包括一个R通道(red红色通道)、一个G通道(green绿色通道)和一个B通道(blue蓝色通道)，每个通道中还可以添加一些激光地图的特征信息。作为一种可选择的实现方式，R通道中可以添加有激光特征，例如地图中各种元素的线特征、角点特征、梯度特征、高度特征等；G通道可以添加有激光地图的亮度信息；B通道可以添加相对高度信息，例如像素相对地面的高度等。In one embodiment, as shown in FIG. 5, the laser map is implemented as a three-channel color image, including, for example, an R channel (red channel), a G channel (green channel), and a B channel (blue channel) ), and some laser map feature information can also be added to each channel. As an optional implementation, the R channel can be added with laser features, such as line features, corner features, gradient features, height features, etc. of various elements in the map; the G channel can be added with the brightness information of the laser map; The B channel can add relative height information, such as the height of the pixel relative to the ground, etc.
另外，除包含上述RGB通道以外，激光地图还可以包括更多的通道，例如alpha通道等，本申请实施例不做限定。在一些其他的实现方式中，激光地图还可以通过其他的色彩模式实现，例如RYYB格式等，此时，激光地图可以相应地包括四个通道。In addition, in addition to the above-mentioned RGB channels, the laser map may also include more channels, such as an alpha channel, etc., which is not limited in this embodiment of the present application. In some other implementation manners, the laser map may also be implemented by other color modes, such as RYYB format, etc. In this case, the laser map may correspondingly include four channels.
在一个实施例中，激光地图的像素中的每一个通道都可以包含一定数量的比特信息bit，例如8bit(8个比特信息)、16bit、32bit等。因此，如上述激光特征、亮度信息和相对高度信息等特征信息可以在像素的bit位中进行编码表示。例如亮度信息就可以通过1个bit编码来表示，如以比特值1表示有亮度信息，0表示没有亮度信息。In one embodiment, each channel in the pixel of the laser map may contain a certain number of bits of information bits, such as 8 bits (8 bits of information), 16 bits, 32 bits, and so on. Therefore, feature information such as the above-mentioned laser feature, brightness information, and relative height information can be encoded and represented in the bits of the pixel. For example, the brightness information can be represented by 1 bit encoding, for example, a bit value of 1 indicates that there is brightness information, and 0 indicates that there is no brightness information.
本申请实施例为了将视觉地图和激光地图融合在一张地图中，将通常位于视觉地图中的视觉语义信息融合进激光地图中。具体实现中，可以根据路面标志、道路交通标志内容在激光地图中对应的位置，确定激光地图中将会包含视觉语义信息的像素，然后将视觉语义信息编码后存储到这些像素的某个或者某些通道的比特信息中。In this embodiment of the present application, in order to fuse the visual map and the laser map into one map, the visual semantic information usually located in the visual map is fused into the laser map. In the specific implementation, the pixels that will contain visual semantic information in the laser map can be determined according to the corresponding positions of the content of road signs and road traffic signs in the laser map, and then the visual semantic information is encoded and stored in one or some of these pixels. in the bit information of these channels.
在一个实施例中，视觉语义信息可以采用8位二进制编码格式表示，占据其所在通道的8个比特长度。图6是视觉语义信息采用8位编码格式的一个示例图。如图6所示，8位编码 格式可以由标志位、类型编码和亮度编码三个部分中的至少一种组成，其中，标志位可以用于表示路面标志的类型；类型编码用于表示路面标志的内容；亮度信息编码可以用于表示图片的亮度信息。In one embodiment, the visual semantic information may be represented in an 8-bit binary encoding format, occupying 8 bits of the channel where it is located. FIG. 6 is an example diagram of an 8-bit encoding format for visual semantic information. As shown in Figure 6, the 8-bit encoding format can be composed of at least one of three parts: marker bit, type encoding and luminance encoding, wherein the marker bit can be used to indicate the type of road marking; the type encoding is used to indicate the road marking content; brightness information coding can be used to represent the brightness information of the picture.
在一种实现方式中，标志位如图6所示占据1个比特，例如8位编码的第一个比特，或者其他比特。此时，标志位可以有0和1两个取值，最多表示两个种类。示例地，如果将视觉语义信息划分位文字类元素和图形标线类元素，那么，0可以表示文字类元素，1可以表示图形标线类元素，或者，1表示文字类元素，0表示图形标线类元素。示例地，如果将视觉语义信息划分为路面标志类元素和其他标志类元素，那么，0可以表示路面标志类元素，1可以表示其他标志类元素，或者，1表示路面标志类元素，0表示其他标志类元素。In an implementation manner, the flag bit occupies 1 bit as shown in FIG. 6 , for example, the first bit of the 8-bit encoding, or other bits. At this time, the flag bit can have two values, 0 and 1, representing at most two types. For example, if the visual semantic information is divided into text elements and graphic reticle elements, then 0 can represent text elements, 1 can represent graphic reticle elements, or 1 represents text elements, and 0 represents graphic markers. Line element. For example, if the visual semantic information is divided into pavement marking elements and other marking elements, then 0 may represent pavement marking elements, 1 may represent other marking elements, or 1 may represent pavement marking elements, and 0 may represent other marking elements. Logo class element.
在一些其他的实现方式中，标志位可以占据大于1个比特，例如2个比特、3个比特等，从而可以表示更多的视觉语义信息的种类，例如2个比特最多可以表示4个种类，3个比特最多可以表示8个种类。在具体实践中，本领域技术人员可以根据实际分类的需求确定标志位的长度，本申请实施例对标志位的长度不做具体限定。In some other implementations, the flag bit can occupy more than 1 bit, such as 2 bits, 3 bits, etc., so that more types of visual semantic information can be represented, for example, 2 bits can represent at most 4 types, 3 bits can represent up to 8 kinds. In specific practice, those skilled in the art can determine the length of the flag bit according to actual classification requirements, and the embodiment of the present application does not specifically limit the length of the flag bit.
在一种实现方式中，类型编码如图6所示占据6个比特，例如标志位后面的连续6个比特，或者其他的比特。此时类型编码的取值范围可以从000000至111111，最多能够表示2 6种内容类型。示例地：000000可以表示路面、000001可以表示路面标志等；更具体地：000001可以表示白色虚线、000010表示白色实线、000011表示直行标志、000100表示左转标志、000101表示警告标志、000110表示指路标志等。本申请实施例中，类型编码是视觉语义信息的8位编码中的必要部分。 In an implementation manner, the type code occupies 6 bits as shown in FIG. 6 , for example, 6 consecutive bits following the flag bit, or other bits. At this time, the value range of the type code can be from 000000 to 111111, which can represent up to 26 content types. Exemplarily: 000000 can represent road, 000001 can represent road signs, etc; road signs etc. In this embodiment of the present application, the type encoding is a necessary part of the 8-bit encoding of the visual semantic information.
在一些其他的实现方式中，类型编码可以占据大于6个比特，例如7个比特，从而可以表示更多的内容类型；也可以小于6个比特，例如5个比特、4个比特等，从而在能够表示所需的全部路面标志和/或道路交通标志的情况下，减小类型编码的比特长度，以减小数据开销，并使得8位编码中能够保留出更多的比特信息来表示其他的信息。在具体实践中，本领域技术人员可以根据需要区分的路面标志和/或道路交通标志的数量确定类型编码的长度，本申请实施例对类型编码的长度不做具体限定。In some other implementations, the type code can occupy more than 6 bits, such as 7 bits, so that more content types can be represented; it can also be less than 6 bits, such as 5 bits, 4 bits, etc., so that in In the case of being able to represent all the required road signs and/or road traffic signs, reduce the bit length of the type code to reduce the data overhead and allow more bit information to be reserved in the 8-bit code to represent other information. In specific practice, those skilled in the art can determine the length of the type code according to the number of road signs and/or road traffic signs to be distinguished, and the length of the type code is not specifically limited in this embodiment of the present application.
在一种实现方式中，亮度编码如图6所示占据1个比特，例如8位编码的最后一个比特，或者其他比特。此时，标志位可以有0和1两个取值，0表示没有亮度信息，1表示有亮度信息。以车道线为例，其相比于路面的亮度更高，因此其亮度信息可以是1；以路面标志为例，其可能包括喷绘的白线部分和路面部分，那么白线部分的亮度信息即可以是1，路面部分的的亮度信息即可以是0。In one implementation, the luma encoding occupies 1 bit as shown in FIG. 6 , such as the last bit of the 8-bit encoding, or other bits. At this time, the flag bit can have two values of 0 and 1, 0 means no brightness information, 1 means there is brightness information. Taking the lane line as an example, its brightness is higher than that of the road surface, so its brightness information can be 1; taking the road marking as an example, it may include the white line part and the road surface part, then the brightness information of the white line part is It can be 1, and the brightness information of the road surface can be 0.
在一些其他的实现方式中，亮度信息可以占据大于1个比特，例如2个比特、3个比特等，从而可以更细分地表示亮度。在具体实践中，本领域技术人员可以根据实际分类的需求确定亮度信息的长度，本申请实施例对亮度信息的长度不做具体限定。In some other implementations, the luminance information may occupy more than 1 bit, such as 2 bits, 3 bits, etc., so that the luminance can be represented in more subdivisions. In specific practice, those skilled in the art can determine the length of the brightness information according to the actual classification requirements, and the embodiments of the present application do not specifically limit the length of the brightness information.
在一个实施例中，视觉语义信息的二进制编码可以仅包含标志位、类型编码和亮度编码中的一部分，例如仅包含类型编码和亮度编码，或者，仅包含标志位和类型编码，或者，仅包含类型编码。并且，视觉语义信息的二进制编码还可以是除8位编码格式以外的其他编码格式，例如大于8位的编码，如16位编码，或者小于8位的编码，如4位编码等，本申请实施例对此不做限定。In one embodiment, the binary encoding of the visual semantic information may only include a part of the flag bit, the type encoding and the luminance encoding, for example, only the type encoding and the luminance encoding, or only the flag bit and the type encoding, or only the type encoding. Type encoding. In addition, the binary encoding of the visual semantic information may also be other encoding formats other than 8-bit encoding formats, such as encoding greater than 8 bits, such as 16-bit encoding, or encoding less than 8 bits, such as 4-bit encoding, etc., this application implements The example does not limit this.
在一个实施例中，编码后的视觉语义信息可以存储在像素的G通道中，这样，当像素的G通道包含8个比特信息时，这8个比特信息从前到后可以依次包括标志位、类型编码和亮 度编码。In one embodiment, the encoded visual semantic information can be stored in the G channel of the pixel. In this way, when the G channel of the pixel contains 8 bits of information, the 8 bits of information can sequentially include flag bits, type encoding and luma encoding.
图7是定位地图上的不同区域的像素对应的视觉语义信息的8比特编码的示意图。如图6所示，根据上述示例的编码规则，区域①的像素对应路面，其标志位为0，类型编码为000000，亮度编码为0，因此视觉语义信息为00000000；区域②的像素对应直行标志，其标志位为1，类型编码为000011，亮度编码为1，因此视觉语义信息为10000111；区域③的像素对应白色实线标志，其标志位为1，类型编码为000010，亮度编码为1，因此视觉语义信息为10000101。FIG. 7 is a schematic diagram of 8-bit encoding of visual semantic information corresponding to pixels of different regions on the positioning map. As shown in Figure 6, according to the coding rules of the above example, the pixels in area ① correspond to the road surface, and the flag bit is 0, the type code is 000000, and the brightness code is 0, so the visual semantic information is 00000000; the pixels in area ② correspond to straight signs , its flag bit is 1, the type code is 000011, and the brightness code is 1, so the visual semantic information is 10000111; the pixel in area ③ corresponds to the white solid line mark, its flag bit is 1, the type code is 000010, and the brightness code is 1. So the visual semantic information is 10000101.
可以理解的是，本申请实施例将原本位于视觉地图中的视觉语义信息编码存储到激光地图的像素的通道中，实现了激光地图与视觉地图的融合，得到了同时包含视觉地图特征和激光地图特征的定位地图，减少了地图的存储开销。It can be understood that, in the embodiment of the present application, the visual semantic information originally located in the visual map is encoded and stored in the pixel channel of the laser map, so as to realize the fusion of the laser map and the visual map, and obtain a visual map feature and a laser map at the same time. The location map of the feature reduces the storage overhead of the map.
下面对本申请实施例提供的定位方法的技术方案进行具体说明。The technical solutions of the positioning method provided by the embodiments of the present application will be specifically described below.
图8是本申请实施例提供用于实现定位方法的定位***硬件框架图。如图8所示，该定位***可以包括控制单元140、GNSS/INS组合模块130、轮速计170、里程计180、激光雷达110、摄像头120和存储器190等。其中，GNSS/INS组合模块、轮速计、里程计、激光雷达、摄像头等模块用于各自采集数据，将数据发送给控制单元进行处理，存储器可以用于存储定位地图、存储上述各个模块采集到的数据、存储用于控制单元执行的程序指令以及存储控制单元在数据处理过程中产生的数据等。FIG. 8 is a hardware framework diagram of a positioning system for implementing a positioning method provided by an embodiment of the present application. As shown in FIG. 8 , the positioning system may include a control unit 140 , a GNSS/INS combination module 130 , a wheel speedometer 170 , an odometer 180 , a lidar 110 , a camera 120 , a memory 190 , and the like. Among them, the GNSS/INS combination module, wheel speedometer, odometer, lidar, camera and other modules are used to collect data respectively, and send the data to the control unit for processing. data, program instructions for execution by the control unit, and data generated by the control unit during data processing.
下面基于图8所示的硬件结构，以被定位的目标为车辆为例，对本申请实施例提供的定位方法的步骤流程进行具体说明，可以理解的是，除车辆以外，本申请实施例的方法的定位目标还可以是轮船、火车等其他交通工具、各种机器人、以及工程机械等。Based on the hardware structure shown in FIG. 8 , taking the target to be positioned as a vehicle as an example, the step flow of the positioning method provided by the embodiment of the present application will be described in detail. It can be understood that, in addition to the vehicle, the method of the embodiment of the present application The positioning target can also be ships, trains and other vehicles, various robots, and construction machinery.
图9是本申请实施例提供的定位方法的流程图，图10是该定位方法涉及到的数据流转框图。如图9和图10所示，该定位方法可以通过以下步骤S101-步骤S109实现：FIG. 9 is a flowchart of a positioning method provided by an embodiment of the present application, and FIG. 10 is a block diagram of data flow involved in the positioning method. As shown in FIG. 9 and FIG. 10 , the positioning method can be implemented through the following steps S101-S109:
步骤S101，确定车辆的初始位姿，在初始位姿周围生成N个采样点C 1～C N，N为正整数。 Step S101 , determine the initial pose of the vehicle, and generate N sampling points C 1 -C N around the initial pose, where N is a positive integer.
其中，初始位姿可以包括车辆的初始位置和初始姿态。Wherein, the initial posture may include the initial position and initial posture of the vehicle.
具体实现中，控制单元可以获取GNSS/INS组合模块采集的数据。然后，控制单元可以根据GNSS的天线信号确定车辆的初始位置，一般来说，车辆的初始位置可以是一个全局位置。另外，控制单元还可以根据INS模块的惯性测量单元IMU测量的车辆的角速度和加速度等信息确定车辆的初始姿态，一般来说车辆的初始姿态可以由车辆初始的航向角、俯仰角和滚转角中的一个或者多个参数组成，由于车辆定位和导航中主要使用航向角，因此车辆的初始姿态也可以仅包含航向角。In the specific implementation, the control unit can obtain the data collected by the GNSS/INS combination module. Then, the control unit may determine the initial position of the vehicle according to the antenna signal of the GNSS. Generally speaking, the initial position of the vehicle may be a global position. In addition, the control unit can also determine the initial attitude of the vehicle according to information such as the angular velocity and acceleration of the vehicle measured by the inertial measurement unit IMU of the INS module. Generally speaking, the initial attitude of the vehicle can be determined from the initial heading angle, pitch angle and roll angle of the vehicle. One or more parameters of , since the heading angle is mainly used in vehicle positioning and navigation, the initial attitude of the vehicle can also only include the heading angle.
在一个实施例中，控制单元在确定了车辆的初始位姿之后，还可以以车辆初始位置为中心，在车辆附近的一定范围内以及车辆航向角附近的一定范围内生成N个采样点C 1～C N。 In one embodiment, after determining the initial pose of the vehicle, the control unit may further generate N sampling points C 1 within a certain range near the vehicle and within a certain range near the heading angle of the vehicle with the initial position of the vehicle as the center ~ CN .
图11示例性地提供了生成采样点的方案。如图11所示，控制单元可以以车辆初始位置为中心，确定半径为R＝5米的圆形范围，以及，以航向角yaw的指向(车辆前进方向)为中心，选取左右偏离2°的扇形范围，即yaw±2°，然后圆形范围和扇形范围的重合区域(即图11中的灰色阴影区域)内生成离散的N＝1000个采样点。在一些实现方式中，这1000个采样点可以采用均匀分布的方式生成，使得这1000个采样点在其分布区域内分布比较均匀。可以理解的是，在采样点选定之后，采样点的初始位姿也随即根据车辆的初始位姿确定。在另一些实现方式中，这1000个采样点也可以采用非均匀的方式生成，例如正态分布等，本申请实施例对此不做限定。Figure 11 exemplarily provides a scheme for generating sample points. As shown in FIG. 11 , the control unit can determine a circular range with a radius of R=5 meters with the initial position of the vehicle as the center, and, with the direction of the heading angle yaw (the vehicle’s forward direction) as the center, select a left and right deviation of 2° The fan-shaped range, that is, yaw±2°, and then discrete N=1000 sampling points are generated in the overlapping area of the circular range and the fan-shaped range (ie, the gray shaded area in FIG. 11 ). In some implementations, the 1000 sampling points may be generated in a uniform distribution manner, so that the distribution of the 1000 sampling points in the distribution area thereof is relatively uniform. It can be understood that after the sampling point is selected, the initial pose of the sampling point is also determined according to the initial pose of the vehicle. In other implementation manners, the 1000 sampling points may also be generated in a non-uniform manner, such as a normal distribution, which is not limited in this embodiment of the present application.
需要补充说明的是，本申请实施例围绕车辆的初始位姿选取大量的采样点，可以实现对 车辆位姿的多重采样，多重计算，并结合采样点滤波技术提高定位精度。具体实现中，控制单元可以对每个采样点分别执行步骤S102-步骤S108。It should be supplemented that the embodiment of the present application selects a large number of sampling points around the initial pose of the vehicle, which can realize multiple sampling and multiple calculations of the vehicle pose, and improve the positioning accuracy in combination with the sampling point filtering technology. In a specific implementation, the control unit may perform steps S102 to S108 for each sampling point respectively.
步骤S102，根据车辆的当前预测位姿从定位地图中提取第一激光特征和至少一个视觉语义信息。Step S102, extracting the first laser feature and at least one visual semantic information from the positioning map according to the current predicted pose of the vehicle.
这里需要说明的是，由于车辆在行驶过程中的位姿是不断变化的，因此各种定位方法都被要求能够对车辆进行实时定位，以利于自动驾驶***实现实时路径规划和导航功能。为实现实时定位的目的，控制单元可以周期性地对车辆进行定位，每个周期的定位行为可以称作一个定位帧。具体描述时，为了区分不同时刻的定位帧，如果设置当前时刻为t，那么当前时刻t对应的定位帧可以称作当前帧，将当前帧的前一个定位帧称作第一历史帧，将第一历史帧的时刻记作第一历史时刻t-1。It should be noted here that since the pose of the vehicle is constantly changing during the driving process, various positioning methods are required to be able to locate the vehicle in real time, so as to facilitate the automatic driving system to realize real-time path planning and navigation functions. For the purpose of real-time positioning, the control unit can periodically position the vehicle, and the positioning behavior of each period can be called a positioning frame. In the specific description, in order to distinguish the positioning frames at different times, if the current time is set as t, the positioning frame corresponding to the current time t can be called the current frame, the previous positioning frame of the current frame is called the first historical frame, and the first historical frame is called the first historical frame. The time of one historical frame is recorded as the first historical time t-1.
可以理解的是，在车辆的行驶过程中，控制单元每次对车辆每进行一次定位，都会得到一个车辆的预测位姿。为便于描述，本申请实施例将第一历史时刻t-1的得到的车辆的位姿称作第一历史位姿，那么根据第一历史位姿和里程计参数，可以预测得到车辆在当前时刻t的当前预测位姿P t。需要说明的是，如果第一历史时刻t-1是初始时刻，那么第一历史位姿为车辆的初始位姿。 It can be understood that, during the driving process of the vehicle, each time the control unit locates the vehicle, a predicted pose of the vehicle will be obtained. For ease of description, in the embodiment of the present application, the obtained vehicle pose at the first historical time t-1 is referred to as the first historical pose. Then, according to the first historical pose and the odometer parameters, it can be predicted that the vehicle is at the current time. The current predicted pose P t of t . It should be noted that, if the first historical moment t-1 is the initial moment, then the first historical pose is the initial pose of the vehicle.
基于上述定义，本申请实施例可以采用以下方式确定车辆的当前预测位姿：Based on the above definition, this embodiment of the present application may determine the current predicted pose of the vehicle in the following manner:
步骤a，获取车辆的初始位姿。Step a, obtain the initial pose of the vehicle.
其中，如前文，初始位姿可以是定位方法初始化执行时根据GNSS/INS组合模块采集的数据确定的位姿。步骤a仅用于在该方法初始化时执行。Among them, as mentioned above, the initial pose may be the pose determined according to the data collected by the GNSS/INS combination module when the positioning method is initialized and executed. Step a is only used to execute when the method is initialized.
步骤b，根据里程计数据确定车辆在当前时刻t和第一历史时刻t-1之间产生的相对位姿。Step b: Determine the relative pose of the vehicle between the current time t and the first historical time t-1 according to the odometer data.
在一种实现方式中，本申请实施例利用里程坐标系得到相对位姿。In an implementation manner, the embodiment of the present application uses the mileage coordinate system to obtain the relative pose.
图12是里程计坐标系的示意图。如图12所示，里程计坐标系可以以车辆的初始位姿作为原点Odom，以车辆在初始位姿下的车体正前方向作为X轴方向，以垂直于车体正前方向并且指向车体左侧的方向作为Y轴方向。Figure 12 is a schematic diagram of an odometer coordinate system. As shown in Figure 12, the odometer coordinate system can take the initial pose of the vehicle as the origin Odom, and take the forward direction of the vehicle body in the initial pose as the X-axis direction, so as to be perpendicular to the forward direction of the vehicle body and point to the vehicle The direction of the left side of the body is taken as the Y-axis direction.
车辆在里程计坐标系中的局部位姿，可以由里程计根据轮速计和惯性测量单元IMU的测量数据计算得到。在计算的局部位姿时，里程计可以采用以下运动学模型：The local pose of the vehicle in the odometer coordinate system can be calculated by the odometer according to the measurement data of the wheel speedometer and the inertial measurement unit IMU. When calculating the local pose, the odometry can adopt the following kinematic models:
S＝V*Δt     (1)S=V*Δt (1)
Figure PCTCN2021082388-appb-000001
Figure PCTCN2021082388-appb-000001
其中，S表示车辆相对于初始位姿的运动里程，V表示车辆的运动速度，Δt表示车辆的运动时间；
Figure PCTCN2021082388-appb-000002
表示车辆相对于初始位姿的航向角的变化值，ω表示车辆的角速度。 Among them, S represents the movement distance of the vehicle relative to the initial pose, V represents the movement speed of the vehicle, and Δt represents the movement time of the vehicle;
Figure PCTCN2021082388-appb-000002
Represents the change value of the heading angle of the vehicle relative to the initial pose, and ω represents the angular velocity of the vehicle.
根据上述运动模型(1)(2)可以得到：According to the above motion model (1) (2), we can get:
x 0＝V*cos(yaw)*Δt  (3) x 0 =V*cos(yaw)*Δt (3)
y 0＝V*sin(yaw)*Δt  (4) y 0 =V*sin(yaw)*Δt (4)
其中，x 0为车辆运动时的X轴坐标值，y 0为车辆运动时的Y轴坐标值，yaw表示车辆的航向角，在里程计坐标系中yaw的取值为
Figure PCTCN2021082388-appb-000003
Among them, x 0 is the X-axis coordinate value when the vehicle is moving, y 0 is the Y-axis coordinate value when the vehicle is moving, and yaw is the heading angle of the vehicle. In the odometer coordinate system, the value of yaw is
Figure PCTCN2021082388-appb-000003
根据上述运动模型(1)(2)和公式(3)(4)即可得到车辆运动时，在任意时刻的以第一局部坐标系的参数表示的局部位姿，例如局部位姿可以包括(x 0，y 0，yaw)。 According to the above motion model (1) (2) and formula (3) (4), the local pose expressed by the parameters of the first local coordinate system at any time when the vehicle is moving can be obtained. For example, the local pose may include ( x 0 , y 0 , yaw).
这里需要说明的是，局部位姿中仅表示车辆在里程计坐标系中的位姿，并不表示车辆在空间环境中的绝对位姿。It should be noted here that the local pose only represents the pose of the vehicle in the odometer coordinate system, and does not represent the absolute pose of the vehicle in the space environment.
需要补充说明的是，除了采用里程计坐标系之外，本申请实施例还可以采用其他的坐标 系得到相对位姿，例如GNSS坐标系、IMU坐标系、车辆后轮轴地面投影坐标系等，本申请实施例对此不做限定。It should be added that, in addition to the odometer coordinate system, other coordinate systems can also be used in this embodiment of the present application to obtain the relative pose, such as the GNSS coordinate system, the IMU coordinate system, and the ground projection coordinate system of the rear wheel axle of the vehicle. The application embodiments do not limit this.
基于里程计坐标系，里程计可以将车辆在当前时刻t的局部位姿和第一历史时刻t-1的局部位姿发送给控制单元。那么，控制单元就可以根据车辆在当前时刻t的局部位姿和第一历史时刻t-1的局部位姿计算车辆在当前时刻t和第一历史时刻t-1之间产生的相对位姿。具体计算方法如公式(5)：Based on the odometer coordinate system, the odometer can send the local pose of the vehicle at the current time t and the local pose of the first historical time t-1 to the control unit. Then, the control unit can calculate the relative pose of the vehicle between the current time t and the first historical time t-1 according to the local pose of the vehicle at the current time t and the local pose of the first historical time t-1. The specific calculation method is as formula (5):
ΔP＝o t-o t-1   (5) ΔP=o t -o t-1 (5)
其中，ΔP为车辆在当前时刻t和第一历史时刻t-1之间产生的相对位姿，o t为车辆在当前时刻t的局部位姿，o t-1为车辆在第一历史时刻t-1的局部位姿。 Among them, ΔP is the relative pose of the vehicle between the current time t and the first historical time t-1, o t is the local pose of the vehicle at the current time t, and o t-1 is the vehicle at the first historical time t. -1 for the local pose.
步骤c，将车辆在第一历史时刻t-1对应的第一历史位姿与相对位姿相加得到车辆的当前预测位姿P t。 Step c, adding the first historical pose corresponding to the vehicle at the first historical time t-1 and the relative pose to obtain the current predicted pose P t of the vehicle.
如以下公式(6)As the following formula (6)
P t＝P t-1+ΔP  (6) P t =P t-1 +ΔP (6)
其中，P t-1为车辆在第一历史时刻t-1对应的第一历史位姿。 Among them, P t-1 is the first historical pose corresponding to the vehicle at the first historical time t-1.
可以理解的是，采样点C n的当前估计位姿P n也通过上述步骤c进行估计，即： It can be understood that the current estimated pose P n of the sampling point C n is also estimated through the above step c, namely:
P n＝P n(t-1)+ΔP P n =P n(t-1) +ΔP
其中，P n(t-1)为采样点C n在第一历史时刻t-1对应的估计位姿。 Among them, P n(t-1) is the estimated pose corresponding to the sampling point C n at the first historical time t-1.
进一步地，控制单元可以根据车辆的当前预测位姿P t，从定位地图中获取当前预测位姿P t附近的一块区域，为便于描述，可以将该区域称作局部定位地图，然后从局部定位地图中提取第一激光特征。 Further, the control unit can obtain an area near the current predicted pose P t from the positioning map according to the current predicted pose P t of the vehicle. Extract the first laser feature from the map.
在一个实施例中，定位地图可以由大量的预设尺寸的图片拼接构成，每一张图片对应空间环境中指定大小的范围。示例地，定位地图的每一张图片均为长宽相等的正方形图片，每一张图片对应长100米、宽100米的正方形范围。In one embodiment, the positioning map may be formed by splicing a large number of pictures of preset sizes, and each picture corresponds to a range of a specified size in the spatial environment. For example, each picture of the positioning map is a square picture of equal length and width, and each picture corresponds to a square range of 100 meters long and 100 meters wide.
在一个实施例中，当定位地图由大量图片构成时，控制单元可以从定位地图中获取当前预测位姿P t所在的图片和附近的至少一张图片，作为局部定位地图。示例地，如图13或14所示，控制单元可以获取当前预测位姿P t所在的和附近的3×3共计9张图片，如果每张图片对应的范围是100m×100m，那么局部定位地图就包括了当前预测位姿P t附近300m×300m的区域。 In one embodiment, when the positioning map is composed of a large number of pictures, the control unit may obtain the picture where the current predicted pose P t is located and at least one nearby picture from the positioning map as the local positioning map. For example, as shown in Fig. 13 or 14, the control unit can obtain a total of 9 pictures of 3 × 3 where the current predicted pose P t is located and nearby, if the corresponding range of each picture is 100m × 100m, then the local positioning map It includes an area of 300m×300m near the current predicted pose P t .
基于上述提取的定位地图的图片，控制单元可以从图片的存储有激光特征的通道中提取第一激光特征，例如从R通道中提取第一激光特征。Based on the above extracted picture of the positioning map, the control unit may extract the first laser feature from the channel of the picture in which the laser feature is stored, for example, extract the first laser feature from the R channel.
另外，控制单元可以局部定位地图的从存储有视觉语义信息的通道中提取至少一个视觉语义信息，例如对图片的G通道数据进行解码，以提取G通道中的至少一个视觉语义信息。In addition, the control unit may extract at least one visual semantic information from the channel storing the visual semantic information in the local location map, for example, decode the G channel data of the picture to extract at least one visual semantic information in the G channel.
步骤S103，对于任意采样点C n，n为正整数，n≤N，根据其对应的当前估计位姿P n，将提取自定位地图的第一激光特征与第二激光特征进行匹配，以确定当前估计位姿P n的第一权值。 Step S103 , for any sampling point C n , where n is a positive integer, n≤N, according to its corresponding current estimated pose P n , the first laser feature extracted from the self-positioning map is matched with the second laser feature to determine: The first weight of the currently estimated pose P n .
其中，第二激光特征可以从激光雷达采集的点云数据中提取。The second laser feature can be extracted from the point cloud data collected by the lidar.
具体实现中，控制单元可以对激光雷达在当前时刻t采集到的点云数据进行采样，以得到第二激光特征，其采样方式可以根据定位地图的具体形式确定。例如：当定位地图为稀疏特征地图时，控制单元可以对点云数据进行稀疏特征采样；当定位地图为半稠密地图时，控制单元可以对点云数据进行半稠密特征采样；当定位地图为稠密地图时，控制单元可以对点 云数据进行稠密特征采样。这样，便于第一激光特征与第二激光特征进行匹配。In specific implementation, the control unit may sample the point cloud data collected by the lidar at the current time t to obtain the second laser feature, and the sampling method may be determined according to the specific form of the positioning map. For example: when the positioning map is a sparse feature map, the control unit can perform sparse feature sampling on the point cloud data; when the positioning map is a semi-dense map, the control unit can perform semi-dense feature sampling on the point cloud data; when the positioning map is dense When mapping, the control unit can perform dense feature sampling on the point cloud data. In this way, matching of the first laser feature with the second laser feature is facilitated.
接下来，控制单元根据当前估计位姿P n将激光特征投影到局部定位地图的坐标系中，对于不同的采样点C n来说，由于其当前估计位姿P n不同，因此上述激光特征根据不同采样点C n的当前估计位姿P n进行投影之后，会在局部定位地图中对应不同的坐标分布。 Next, the control unit projects the laser features into the coordinate system of the local positioning map according to the current estimated pose P n . For different sampling points C n , since their current estimated poses P n are different, the above laser features are based on After the current estimated poses P n of different sampling points C n are projected, they will correspond to different coordinate distributions in the local positioning map.
接下来，控制单元可以基于第一激光特征与第二激光在局部定位地图中的坐标分布，将第一激光特征与第二激光特征进行匹配，计算出第一激光特征与第二激光特征之间的匹配距离，并且根据匹配距离确定出当前估计位姿P n的第一权值
Figure PCTCN2021082388-appb-000004
其中，匹配距离代表了基于激光特征确定的车辆实际位姿与采样点C n的当前估计位姿P n之间的接近程度，而接近程度越高，第一权值
Figure PCTCN2021082388-appb-000005
就越大，接近程度越低，第一权值
Figure PCTCN2021082388-appb-000006
就越小。 Next, the control unit may match the first laser feature and the second laser feature based on the coordinate distribution of the first laser feature and the second laser in the local positioning map, and calculate the difference between the first laser feature and the second laser feature The matching distance of , and the first weight of the current estimated pose P n is determined according to the matching distance
Figure PCTCN2021082388-appb-000004
Among them, the matching distance represents the closeness between the actual pose of the vehicle determined based on the laser feature and the current estimated pose Pn of the sampling point Cn , and the higher the closeness is, the higher the first weight is.
Figure PCTCN2021082388-appb-000005
The larger, the lower the degree of proximity, the first weight
Figure PCTCN2021082388-appb-000006
the smaller.
在一些实施例中，匹配距离可以是余弦距离，也可以是欧拉距离，本申请实施例对获得匹配距离所采用的算法不做限定。例如：当匹配距离是余弦距离时，匹配距离的数值范围可以为[0，1]，数值越大，表示基于激光特征确定的车辆实际位姿与采样点C n的当前估计位姿P n之间的接近程度越低，数值越小，表示基于激光特征确定的车辆实际位姿与采样点C n的当前估计位姿P n之间的接近程度越高。 In some embodiments, the matching distance may be a cosine distance or an Euler distance, and the embodiments of the present application do not limit the algorithm used to obtain the matching distance. For example: when the matching distance is a cosine distance, the value range of the matching distance can be [0, 1]. The larger the value, the difference between the actual pose of the vehicle determined based on the laser feature and the current estimated pose P n of the sampling point C n The lower the closeness between the two, the smaller the value, which means the higher the closeness between the actual pose of the vehicle determined based on the laser feature and the current estimated pose Pn of the sampling point Cn .
步骤S104，对于任意采样点C n，根据当前估计位姿P n，将至少一个视觉语义信息与至少一个路面语义信息进行匹配，以确定当前估计位姿P n的第二权值。 Step S104 , for any sampling point C n , according to the current estimated pose P n , match at least one visual semantic information with at least one road semantic information to determine the second weight of the current estimated pose P n .
其中，上述至少一个路面语义信息是从摄像头采集的图像数据中提取的。Wherein, the above-mentioned at least one pavement semantic information is extracted from the image data collected by the camera.
具体实现中，如图14所示，控制单元可以首先对摄像头采集的图像数据进行预处理，例如去除噪声、裁切、灰度处理等等。接下来，控制单元可以使用预先训练好的深度神经网络对预处理之后的图像进行像素级的语义分割，以从图像中提取至少一个路面语义信息，路面语义信息可以是像素级信息，每个路面语义信息可以包括：包含有至少一个路面标志的像素块、像素块中的像素的数量、每个像素所属的路面标志的类型和概率等。其中，该像素块应该至少包含路面标志的全部像素，在一些实施例中，像素块可以是矩形、圆形等规则形状，或者其他形状，优选为规则形状，以便于数据处理。另外，在保证像素块包含路面标志的全部像素的情况下，像素块优选尽量少地包含非路面标志的像素。In a specific implementation, as shown in FIG. 14 , the control unit may first perform preprocessing on the image data collected by the camera, such as noise removal, cropping, grayscale processing, and the like. Next, the control unit can use the pre-trained deep neural network to perform pixel-level semantic segmentation on the preprocessed image to extract at least one pavement semantic information from the image. The pavement semantic information can be pixel-level information, and each pavement The semantic information may include: a pixel block containing at least one road marking, the number of pixels in the pixel block, the type and probability of the road marking to which each pixel belongs, and the like. The pixel block should contain at least all pixels of the road marking. In some embodiments, the pixel block may be a regular shape such as a rectangle, a circle, etc., or other shapes, preferably a regular shape to facilitate data processing. In addition, when it is guaranteed that the pixel block includes all the pixels of the road surface marking, the pixel block preferably contains as few pixels as possible that are not the road surface marking.
本申请实施例使用的深度神经网络例如可以是：卷积神经网络(convolutional neural network，CNN)、长短期记忆网络(long short-term memory，LSTM)、循环神经网络(recurrent neural network，RNN)或者其他神经网络，也可以是多种神经网络的组合。深度神经网络在训练阶段使用训练语料作为输入，该训练语料可以是事先采集的路面图片，并且路面图片中的路面标志被进行像素级标注；深度神经网络在训练阶段的输出是训练语料的标注结果，例如被标注的路面标志的类型等。神经网络在使用阶段的输入为摄像头采集到的图像，输出则为图像中包含的路面标志的像素级信息。使用深度神经网络进行信息提取的具体方法不是本申请实施例的重点讨论内容，因篇幅所限，此处就不再赘述。The deep neural network used in the embodiments of the present application may be, for example, a convolutional neural network (CNN), a long short-term memory (LSTM), a recurrent neural network (RNN), or Other neural networks can also be a combination of various neural networks. The deep neural network uses the training corpus as input in the training phase, the training corpus can be the road image collected in advance, and the road signs in the road image are marked at the pixel level; the output of the deep neural network in the training phase is the annotation result of the training corpus , such as the type of pavement markings being marked, etc. The input of the neural network in the use stage is the image collected by the camera, and the output is the pixel-level information of the road signs contained in the image. The specific method of using the deep neural network for information extraction is not the focus of the discussion in this embodiment of the present application, and due to space limitations, it will not be repeated here.
另外，需要补充说明的是，路面语义信息实际上也属于视觉语义信息，区别之处在于：是从摄像头采集的图像中提取的，而不是存储在定位地图中。In addition, it needs to be added that the pavement semantic information actually belongs to the visual semantic information, the difference is that it is extracted from the image collected by the camera, rather than stored in the positioning map.
基于上述提取的路面语义信息和视觉语义信息，步骤S104可以通过如图15所示的以下步骤S201-步骤S204实现：Based on the above-mentioned extracted pavement semantic information and visual semantic information, step S104 can be implemented by the following steps S201-S204 as shown in FIG. 15 :
步骤S201，从至少一个路面语义信息中确定至少一个有效路面语义信息，建立有效路面语义信息集合，其中，每个有效路面语义信息的像素数量在预设范围内。Step S201, at least one valid road semantic information is determined from the at least one road semantic information, and a set of valid road semantic information is established, wherein the number of pixels of each valid road semantic information is within a preset range.
在一个实施例中，步骤S201如图16所示具体可以通过以下步骤S301-步骤S303实现：In one embodiment, as shown in FIG. 16 , step S201 can be specifically implemented by the following steps S301-S303:
步骤S301，分别计算每一个路面语义信息的像素面积。Step S301, respectively calculating the pixel area of each road surface semantic information.
具体实现中，路面语义信息的像素面积可以是像素块的像素数量。以矩形的像素块为例，假设像素块的分辨率大小为W像素×H像素，W和H分别为像素块水平方向和垂直方向的像素数量，那么该像素块的像素面积S＝W×H。In a specific implementation, the pixel area of the pavement semantic information may be the number of pixels of the pixel block. Taking a rectangular pixel block as an example, assuming that the resolution size of the pixel block is W pixels×H pixels, and W and H are the number of pixels in the horizontal and vertical directions of the pixel block, respectively, then the pixel area of the pixel block S=W×H .
示例地，如图17所示，在步骤S105中，控制单元从摄像头采集的图像中获取到了多个路面语义信息，例如包含：路面语义信息L0、路面语义信息L1和路面语义信息L2。那么，在步骤S301中，则可以分别计算L0、L1、L2的像素块的像素量，以得到L0的像素面积为S 0＝W0×H0、L1的像素面积为S 1＝W1×H1、L2的像素面积为S 2＝W2×H2。 For example, as shown in FIG. 17 , in step S105 , the control unit acquires a plurality of pavement semantic information from the image collected by the camera, for example, including pavement semantic information L0 , pavement semantic information L1 and pavement semantic information L2 . Then, in step S301, the pixel quantities of the pixel blocks of L0, L1, and L2 can be calculated respectively, so as to obtain the pixel area of L0 as S 0 =W0×H0, and the pixel area of L1 as S 1 =W1×H1,L2 The pixel area is S 2 =W2×H2.
步骤S302，根据像素面积确定有效路面语义信息。Step S302, determining the effective road semantic information according to the pixel area.
具体实现中，本申请实施例可以设置用于确定有效路面语义信息的一个像素面积下限阈值T 1和一个像素面积上限阈值T 2，控制单元使用下限阈值T 1和上限阈值T 2分别与每个路面语义信息的像素面积S(例如：S 0、S 1、S 2等)进行比较，当T 1<S<T 2时，路面语义信息即为有效语义信息，当S<T 1或者S>T 2时，路面语义信息即为无效语义信息。其中，下限阈值T 1和上限阈值T 2可以是预先设置的值，也可以是动态生成的值。 In specific implementation, the embodiment of the present application may set a pixel area lower threshold value T 1 and a pixel area upper threshold value T 2 for determining the effective road semantic information, and the control unit uses the lower threshold value T 1 and the upper threshold value T 2 to be respectively associated with each The pixel area S (for example: S 0 , S 1 , S 2 , etc.) of the road semantic information is compared. When T 1 <S<T 2 , the road semantic information is valid semantic information. When S<T 1 or S> At T 2 , the pavement semantic information is invalid semantic information. The lower threshold T1 and the upper threshold T2 may be preset values or dynamically generated values.
在一个实施例中，当下限阈值T 1和上限阈值T 2为动态生成的值时，控制单元可以统计一段时间内所有路面语义信息的像素面积，得到像素面积的分布范围，然后在像素面积的分布范围中选取一定的范围作为有效路面语义信息的范围，进而确定下限阈值T 1和上限阈值T 2。 In one embodiment, when the lower limit threshold T1 and the upper limit threshold T2 are dynamically generated values, the control unit can count the pixel areas of all road semantic information within a period of time to obtain the distribution range of the pixel areas, and then calculate the pixel area in the pixel area. A certain range is selected from the distribution range as the range of valid road semantic information, and then the lower threshold T1 and the upper threshold T2 are determined .
步骤S303，对有效路面语义信息建立有效路面语义信息集合。Step S303, establishing an effective pavement semantic information set for the effective pavement semantic information.
其中，根据步骤S302确定的有效语义路面信息结果的不同，有效路面语义信息集合中包含的有效路面语义信息的数量也不同。例如：当步骤S302确定路面语义信息中不包含有效路面语义信息集合时，有效路面语义信息集合为空集合；当步骤S302确定一部分路面语义信息是有效路面语义信息时，有效路面语义信息集合是路面语义信息集合的子集，当步骤S302确定所有的路面语义信息都是有效路面语义信息时，有效路面语义信息集合与路面语义信息集合相同。Wherein, according to different results of the valid semantic road surface information determined in step S302, the quantity of valid road semantic information included in the valid road semantic information set is also different. For example: when it is determined in step S302 that the pavement semantic information does not contain the effective pavement semantic information set, the effective pavement semantic information set is an empty set; when it is determined in step S302 that a part of pavement semantic information is valid pavement semantic information, the effective pavement semantic information set is pavement semantic information A subset of the semantic information set, when it is determined in step S302 that all the pavement semantic information is valid pavement semantic information, the effective pavement semantic information set is the same as the pavement semantic information set.
示例地，有效路面语义信息集合中的任意一个有效路面语义信息可以是如下形式：For example, any valid pavement semantic information in the effective pavement semantic information set may be in the following form:
a i＝[M，p 1～p M，Ka i] a i =[M, p 1 to p M , Ka i ]
其中，a i表示有效路面语义信息集合中的第i个有效路面语义信息，M表示a i包含的像素点的数量，Ka i表示a i对应的语义类型值(例如路面标志的类型值)，不同的语义类型具有不同的类型值，p 1～p M分别表示a i中第1至第M个像素点属于Ka i的概率，p 1～p M可由深度神经网络的输出结果得到。 Among them, a i represents the ith valid pavement semantic information in the effective pavement semantic information set, M represents the number of pixels contained in a i , and Ka i represents the semantic type value corresponding to a i (such as the type value of pavement signs), Different semantic types have different type values, p 1 -p M respectively represent the probability that the 1st to M-th pixels in a i belong to Kai , and p 1 -p M can be obtained from the output results of the deep neural network.
以上步骤S301-步骤S303即为步骤S201示例性的可实现方式。The above steps S301 to S303 are exemplary implementations of step S201.
步骤S202，统一有效路面语义信息与视觉语义信息的坐标系。Step S202, unifying the coordinate system of the effective pavement semantic information and the visual semantic information.
具体实现中，控制单元可以根据当前估计位姿P n将上述至少一个有效路面语义信息投影到局部定位地图的坐标系中，对于不同的采样点C n来说，由于其当前估计位姿P n不同，因此上述至少一个有效路面语义信息根据不同采样点C n的当前估计位姿P n进行投影之后，会在局部定位地图中对应不同的坐标分布。 In specific implementation, the control unit may project the at least one valid road semantic information into the coordinate system of the local positioning map according to the current estimated pose P n . For different sampling points C n , due to the current estimated pose P n Therefore, after the at least one valid pavement semantic information is projected according to the current estimated pose P n of different sampling points C n , it will correspond to different coordinate distributions in the local positioning map.
在一个实施例中，局部定位地图可以使用GNSS坐标系等已知的坐标系，也可以单独具有自己的坐标系，例如控制单元可以以局部定位地图的中心点为原点，以水平方向和垂直方向为X轴和Y轴，建立局部定位地图的坐标系。In one embodiment, the local positioning map may use a known coordinate system such as the GNSS coordinate system, or may independently have its own coordinate system. For example, the control unit may take the center point of the local positioning map as the origin, and take For the X-axis and Y-axis, establish the coordinate system of the local positioning map.
进一步地，在确定了局部定位地图的坐标系之后，控制单元可以通过矩阵变换的方式将有效路面语义信息从摄像头坐标系投影到局部定位地图的坐标系。其中，变换矩阵例如可以是一个大小为4×4的矩阵，其数学意义表示一次平移和一次旋转的空间变换过程，也就是说，有效路面语义信息中的任意像素点均可以经过一次平移和一次旋转投影到局部定位地图的坐标系中。空间点在不同坐标系之间的投影变换是导航定位领域的常见手段，此处不在赘述。Further, after determining the coordinate system of the local positioning map, the control unit can project the effective road semantic information from the camera coordinate system to the coordinate system of the local positioning map by means of matrix transformation. Among them, the transformation matrix can be, for example, a matrix with a size of 4×4, and its mathematical meaning represents the spatial transformation process of one translation and one rotation, that is to say, any pixel point in the valid road semantic information can undergo one translation and one rotation. The rotation is projected into the coordinate system of the local positioning map. Projection transformation of spatial points between different coordinate systems is a common method in the field of navigation and positioning, and will not be repeated here.
步骤S203，确定至少一个有效路面语义信息和至少一个视觉语义信息的语义关联关系。Step S203, determining a semantic association relationship between at least one valid pavement semantic information and at least one visual semantic information.
步骤S203所要实现的目的是：对于有效路面语义信息集合中的任意一个有效路面语义信息a i(a i∈A)，从局部定位地图的视觉语义信息集合B中找到一个与a i语义关联的视觉语义信息b j(b j∈B)。 The purpose to be achieved in step S203 is: for any valid pavement semantic information a i (a i ∈ A) in the effective pavement semantic information set, find a semantically related to a i from the visual semantic information set B of the local positioning map. Visual semantic information b j (b j ∈ B).
在一个实施例中，步骤S203如图18所示可以通过以下步骤S401-步骤S404实现：In one embodiment, step S203 as shown in FIG. 18 can be implemented by the following steps S401-S404:
步骤S401，计算有效路面语义信息a i的语义权值。 Step S401, calculating the semantic weight of the effective pavement semantic information a i .
具体实现中，可以采用以下公式：In the specific implementation, the following formula can be used:
Figure PCTCN2021082388-appb-000007
Figure PCTCN2021082388-appb-000007
其中，
Figure PCTCN2021082388-appb-000008
表示a i的语义权值，M表示a i包含的像素点的数量，p m表示a i中第m个像素点属于Ka i的概率，Ka i表示a i对应的语义类型值。 in,
Figure PCTCN2021082388-appb-000008
Represents the semantic weight of a i , M represents the number of pixels included in a i , p m represents the probability that the mth pixel in a i belongs to Kai , and Kai represents the semantic type value corresponding to a i .
步骤S402，计算视觉语义信息b j的语义权值。 Step S402, calculating the semantic weight of the visual semantic information b j .
具体实现中，可以采用以下公式：In the specific implementation, the following formula can be used:
Figure PCTCN2021082388-appb-000009
Figure PCTCN2021082388-appb-000009
其中，
Figure PCTCN2021082388-appb-000010
表示b j的语义权值，G表示b j包含的像素点的数量，p g表示b j中第g个像素点属于
Figure PCTCN2021082388-appb-000011
的概率，
Figure PCTCN2021082388-appb-000012
表示b j对应的语义类型值。 in,
Figure PCTCN2021082388-appb-000010
Represents the semantic weight of b j , G represents the number of pixels contained in b j , and p g represents the gth pixel in b j belongs to
Figure PCTCN2021082388-appb-000011
The probability,
Figure PCTCN2021082388-appb-000012
Represents the semantic type value corresponding to b j .
这里需要补充说明的是，在局部地图中，一个像素点是否属于路面标志或者道路交通标志是已知的，因此对于b j中第g个像素点来说，其p g值只有两种0和1两种可能，如果该像素点属于路面标志或者道路交通标志，那么其p g＝1，否则p g＝0。 What needs to be added here is that in the local map, it is known whether a pixel belongs to a road sign or a road traffic sign, so for the gth pixel in b j , its p g value has only two kinds of 0 and 0. 1 Two possibilities, if the pixel belongs to a road sign or a road traffic sign, then its p g =1, otherwise p g =0.
步骤S403，根据有效路面语义信息a i的语义权值和视觉语义信息b j的语义权值的差值，确定有效路面语义信息a i和视觉语义信息b j的语义关联度。 Step S403, according to the difference between the semantic weight of the effective pavement semantic information a i and the semantic weight of the visual semantic information b j , determine the semantic relevance of the effective pavement semantic information a i and the visual semantic information b j .
具体实现中，对于任意有效路面语义信息a i和任意视觉语义信息b j，语义关联度Δw是有效路面语义信息a i和任意视觉语义信息b j的差值的绝对值，即采用以下公式： In the specific implementation, for any effective pavement semantic information a i and any visual semantic information b j , the semantic relevance Δw is the absolute value of the difference between the effective pavement semantic information a i and any visual semantic information b j , that is, the following formula is used:
Figure PCTCN2021082388-appb-000013
Figure PCTCN2021082388-appb-000013
步骤S404，当语义关联度小于预设第一阈值时，确定有效路面语义信息a i和视觉语义信息b j具有语义关联。 Step S404, when the semantic correlation degree is less than the preset first threshold, it is determined that the effective road semantic information a i and the visual semantic information b j have semantic correlation.
具体实现中，对于任意有效路面语义信息a i和任意视觉语义信息b j，如果其语义关联度Δw小于预设的第一阈值σ，则有效路面语义信息a i和任意视觉语义信息b j具有语义关联；如果其语义关联度Δw大于或者等于预设的第一阈值σ，则有效路面语义信息a i和任意视觉语义信息b j不具有语义关联。 In specific implementation, for any valid pavement semantic information a i and any visual semantic information b j , if their semantic relevance Δw is less than the preset first threshold σ, then the effective pavement semantic information a i and any visual semantic information b j have Semantic correlation; if its semantic correlation degree Δw is greater than or equal to the preset first threshold σ, the effective road semantic information a i and any visual semantic information b j do not have semantic correlation.
本申请实施例通过多次执行步骤S401-步骤S404的操作，最终可以得到至少一对具有语义关联的有效路面语义信息和视觉语义信息，为便于描述，具有语义关联的有效路面语义信息和视觉语义信息可以被称作语义关联组。In this embodiment of the present application, by performing the operations from step S401 to step S404 multiple times, at least one pair of valid pavement semantic information and visual semantic information with semantic association can finally be obtained. For convenience of description, valid pavement semantic information and visual semantic information with semantic association Information may be referred to as a semantically related group.
以上步骤S401-步骤S404即为步骤S203示例性的可实现方式。The above steps S401 to S404 are exemplary implementations of step S203.
步骤S204，对每一对语义关联的有效路面语义信息和视觉语义信息进行语义匹配，根据语义匹配结果确定第二权值。In step S204, semantic matching is performed on each pair of semantically associated valid pavement semantic information and visual semantic information, and a second weight is determined according to the semantic matching result.
具体实现中，步骤S204如图19所示可以通过以下步骤S501-步骤S503实现：In the specific implementation, as shown in FIG. 19 , step S204 can be implemented by the following steps S501-S503:
步骤S501，分别计算每一对语义关联的有效路面语义信息和视觉语义信息的匹配距离。Step S501, respectively calculating the matching distance between each pair of semantically associated effective pavement semantic information and visual semantic information.
其中，对于每一对具有语义关联的有效路面语义信息和视觉语义信息，匹配距离可以是有效路面语义信息和视觉语义信息的欧几里得距离(即欧式距离)，或者余弦距离等，本申请实施例不做具体限定。Among them, for each pair of valid pavement semantic information and visual semantic information with semantic association, the matching distance may be the Euclidean distance (ie Euclidean distance) of the effective pavement semantic information and visual semantic information, or cosine distance, etc. The embodiment is not specifically limited.
下面结合图20对欧几里得距离和余弦距离的数学意义进行说明。图20在局部定位地图的坐标系中示出了有效路面语义信息a i和视觉语义信息b j的位置。那么欧几里得距离是有效路面语义信息a i和视觉语义信息b j在局部定位地图的坐标系中的直线距离。余弦距离是有效路面语义信息a i和视觉语义信息b j与坐标系原点连线夹角α ij的余弦值。 The mathematical meaning of the Euclidean distance and the cosine distance will be described below with reference to FIG. 20 . FIG. 20 shows the positions of the valid road semantic information a i and the visual semantic information b j in the coordinate system of the local localization map. Then the Euclidean distance is the straight-line distance between the effective pavement semantic information a i and the visual semantic information b j in the coordinate system of the local positioning map. The cosine distance is the cosine value of the angle α ij between the effective road semantic information a i and the visual semantic information b j and the origin of the coordinate system.
步骤S502，将各个匹配距离的加权求和得到总匹配距离。Step S502, the weighted summation of each matching distance is performed to obtain a total matching distance.
为得到总匹配距离，步骤S502如图21所示可以通过以下步骤S601-步骤S602实现。To obtain the total matching distance, step S502 can be implemented through the following steps S601-S602 as shown in FIG. 21 .
步骤S601，计算每个语义关联组的权重。Step S601, calculating the weight of each semantic association group.
具体可以采用以下公式：Specifically, the following formula can be used:
Figure PCTCN2021082388-appb-000014
Figure PCTCN2021082388-appb-000014
其中，K表示语义关联组的数量，w′ k表示第k个语义关联组的权重，Δw k表示第k个语义关联组的语义关联度。 Among them, K represents the number of semantic association groups, w′ k represents the weight of the k-th semantic association group, and Δw k represents the semantic association degree of the k-th semantic association group.
步骤S602，根据所有语义关联组的权重和匹配距离确定总匹配距离。Step S602: Determine the total matching distance according to the weights and matching distances of all semantic association groups.
其中，可以分别将每个语义关联组的权重和匹配距离相乘，然后对所有的相乘结果求和，得到总匹配距离E，即采用以下公式：Among them, the weight of each semantic association group and the matching distance can be multiplied separately, and then all the multiplication results are summed to obtain the total matching distance E, that is, the following formula is used:
Figure PCTCN2021082388-appb-000015
Figure PCTCN2021082388-appb-000015
其中，K表示语义关联组的数量，w′ k表示第k个语义关联组的权重，E k表示第k个语义关联组的匹配距离。 Among them, K represents the number of semantic association groups, w′ k represents the weight of the k-th semantic association group, and E k represents the matching distance of the k-th semantic association group.
步骤S503，根据总匹配距离确定当前估计位姿P n的第二权值
Figure PCTCN2021082388-appb-000016
Step S503, determine the second weight of the current estimated pose P n according to the total matching distance
Figure PCTCN2021082388-appb-000016
总匹配距离代表了基于视觉特征确定的车辆实际位姿与采样点C n的当前估计位姿P n之间的接近程度，而接近程度越高，第二权值
Figure PCTCN2021082388-appb-000017
就越高，接近程度越低，第二权值
Figure PCTCN2021082388-appb-000018
就越低。 The total matching distance represents the closeness between the actual pose of the vehicle determined based on the visual features and the current estimated pose Pn of the sampling point Cn , and the higher the closeness, the second weight.
Figure PCTCN2021082388-appb-000017
The higher, the lower the degree of proximity, the second weight
Figure PCTCN2021082388-appb-000018
lower.
在一个实施例中，当匹配距离是欧几里得距离时，总匹配距离的数值越小，表示基于视觉特征确定的车辆实际位姿与采样点C n的当前估计位姿P n之间的接近程度越高，因此相应的第二权值
Figure PCTCN2021082388-appb-000019
越大；总匹配距离的数值越大，表示基于视觉特征确定的车辆实际位姿与采样点C n的当前估计位姿P n之间的接近程度越低，因此相应的第二权值
Figure PCTCN2021082388-appb-000020
越小。 In one embodiment, when the matching distance is the Euclidean distance, the smaller the value of the total matching distance is, the smaller the value of the total matching distance indicates the difference between the actual pose of the vehicle determined based on the visual feature and the current estimated pose P n of the sampling point C n The higher the degree of proximity, the corresponding second weight
Figure PCTCN2021082388-appb-000019
The larger the value of the total matching distance, the lower the proximity between the actual vehicle pose determined based on the visual feature and the current estimated pose P n of the sampling point C n , so the corresponding second weight
Figure PCTCN2021082388-appb-000020
smaller.
在一个实施例中，当匹配距离是余弦距离时，总匹配距离的数值越小，表示基于视觉特征确定的采样点C n的实际位姿与当前预测位姿P n之间的接近程度越高，因此相应的第二权值
Figure PCTCN2021082388-appb-000021
越大；总匹配距离的数值越大，表示基于视觉特征确定的采样点C n的实际位姿与当前预测位姿P n之间的接近程度越低，因此相应的第二权值
Figure PCTCN2021082388-appb-000022
越小。 In one embodiment, when the matching distance is a cosine distance, the smaller the value of the total matching distance, the higher the degree of closeness between the actual pose of the sampling point C n determined based on the visual feature and the current predicted pose P n , so the corresponding second weight
Figure PCTCN2021082388-appb-000021
The larger the value of the total matching distance, the lower the proximity between the actual pose of the sampling point C n determined based on the visual feature and the current predicted pose P n , so the corresponding second weight
Figure PCTCN2021082388-appb-000022
smaller.
基于上述总匹配距离与第二权值
Figure PCTCN2021082388-appb-000023
之间的数值变换关系，控制单元可以采用任意的算法确定第二权值
Figure PCTCN2021082388-appb-000024
本申请实施例对此不做限定。第二权值
Figure PCTCN2021082388-appb-000025
的取值范围例如可以是[0，1]，或者其他范围，本申请实施例对此不做限定。示例地，第二权值
Figure PCTCN2021082388-appb-000026
可以通过对总匹配距离取补、取倒数、数值范围归化等方式得到。 Based on the above total matching distance and the second weight
Figure PCTCN2021082388-appb-000023
The numerical transformation relationship between , the control unit can use any algorithm to determine the second weight
Figure PCTCN2021082388-appb-000024
This embodiment of the present application does not limit this. second weight
Figure PCTCN2021082388-appb-000025
The value range of , for example, may be [0, 1], or other ranges, which are not limited in this embodiment of the present application. Illustratively, the second weight
Figure PCTCN2021082388-appb-000026
It can be obtained by complementing the total matching distance, taking the reciprocal, and normalizing the numerical range.
以上步骤S501-步骤S504即为步骤S204示例性的可实现方式。The above steps S501 to S504 are exemplary implementations of step S204.
步骤S105，根据所述N个采样点C 1～C N的当前估计位姿P 1-P N及其所述第一权值和所述第二权值，计算所述当前估计位姿P 1-P N的加权平均值，以所述加权求平均值作为所述车辆的 当前位姿。 Step S105: Calculate the current estimated pose P 1 according to the current estimated pose P 1 -PN of the N sampling points C 1 to CN and the first weight and the second weight - The weighted average of P N , and the weighted average is used as the current pose of the vehicle.
具体实现中，步骤S105如图22所示可以通过以下步骤S701-步骤S703实现：In the specific implementation, step S105 can be implemented by the following steps S701-S703 as shown in FIG. 22 :
步骤S701，使用当前估计位姿P 1-P N的第一权值对当前估计位姿P 1-P N加权求平均，得到第一加权平均值。 Step S701, using the first weights of the current estimated poses P 1 -PN to weight the current estimated poses P 1 -PN to obtain a first weighted average.
在一个实施例中，第一加权平均值可以通过以下公式得到：In one embodiment, the first weighted average can be obtained by the following formula:
Figure PCTCN2021082388-appb-000027
Figure PCTCN2021082388-appb-000027
其中，P l表示第一加权平均值，N为采样点的数量；P n表示第n个采样点的当前估计位姿；
Figure PCTCN2021082388-appb-000028
为第n个采样点的当前估计位姿对应的第一权值，
Figure PCTCN2021082388-appb-000029
为采样点C 1～C N的当前估计位姿P 1-P N对应的第一权值
Figure PCTCN2021082388-appb-000030
之和。 Among them, P l represents the first weighted average value, N is the number of sampling points; P n represents the current estimated pose of the nth sampling point;
Figure PCTCN2021082388-appb-000028
is the first weight corresponding to the current estimated pose of the nth sampling point,
Figure PCTCN2021082388-appb-000029
is the first weight corresponding to the currently estimated pose P 1 - P N of the sampling points C 1 -CN
Figure PCTCN2021082388-appb-000030
Sum.
步骤S702，使用当前估计位姿P 1-P N的第二权值对当前估计位姿P 1-P N加权求平均，得到第二加权平均值。 Step S702, using the second weight of the current estimated pose P 1 -PN to weight the current estimated pose P 1 -PN to obtain a second weighted average.
在一个实施例中，第二加权平均值可以通过以下公式得到：In one embodiment, the second weighted average can be obtained by the following formula:
Figure PCTCN2021082388-appb-000031
Figure PCTCN2021082388-appb-000031
其中，P c表示第二加权平均值，N为采样点的数量；P n表示第n个采样点的当前估计位姿；
Figure PCTCN2021082388-appb-000032
为第n个采样点的当前估计位姿对应的第二权值，
Figure PCTCN2021082388-appb-000033
为采样点C 1～C N的当前估计位姿P 1-P N对应的第二权值
Figure PCTCN2021082388-appb-000034
之和。 Among them, P c represents the second weighted average, N is the number of sampling points; P n represents the current estimated pose of the nth sampling point;
Figure PCTCN2021082388-appb-000032
is the second weight corresponding to the current estimated pose of the nth sampling point,
Figure PCTCN2021082388-appb-000033
is the second weight corresponding to the current estimated pose P 1 - P N of the sampling points C 1 -CN
Figure PCTCN2021082388-appb-000034
Sum.
其中，步骤S701和步骤S702仅用于描述方法步骤，不代表步骤的先后顺序，一般来说，控制单元可以并行执行步骤S701和步骤S702，也可以先后执行S701和步骤S702。另外，本申请实施例中，第一加权平均值和第二加权平均值均为位姿。Wherein, step S701 and step S702 are only used to describe the method steps, and do not represent the sequence of steps. Generally speaking, the control unit may execute step S701 and step S702 in parallel, or may execute step S701 and step S702 in sequence. In addition, in this embodiment of the present application, the first weighted average value and the second weighted average value are both poses.
步骤S703，对第一加权平均值和第二加权平均值加权求平均，得到加权平均值，以加权求平均值作为车辆的当前位姿。Step S703, the first weighted average value and the second weighted average value are weighted and averaged to obtain a weighted average value, and the weighted average value is used as the current pose of the vehicle.
具体实现中，车辆的定位结果可以通过以下公式得到：In the specific implementation, the positioning result of the vehicle can be obtained by the following formula:
P＝α l·P l+α c·P c      (14) P = α l ·P l +α c ·P c (14)
其中，P为车辆的当前位姿，即定位***本次定位输出的车辆实际位姿；P l表示第一加权平均值，P c表示第二加权平均值；α l表示第一加权平均值的权值，α c表示第二加权平均值的权值，α l+α c＝1。其中，α l和α c的取值可以根据实际需求确定，α l或α c的数值越大，表示第一加权平均值或第二加权平均值的权重越高。具体实践时，如果技术人员希望以激光特征主导定位结果，则可以增大α l的取值，例如α l取值为0.7、0.8等；如果技术人员希望以视觉特征主导定位结果，则可以增大α c的取值，例如α c取值为0.7、0.8等；如果技术人员希望激光特征和视觉特征对定位结果起到均等的作用，则α l和α c的取值可以均为0.5。 Among them, P is the current pose of the vehicle, that is, the actual pose of the vehicle output by the positioning system at this time; P l represents the first weighted average value, P c represents the second weighted average value; α l represents the first weighted average value. Weight, α c represents the weight of the second weighted average value, α l +α c =1. The values of α l and α c may be determined according to actual requirements, and the larger the value of α l or α c , the higher the weight of the first weighted average or the second weighted average. In specific practice, if the technician wants to use the laser feature to dominate the positioning result, he can increase the value of α l , for example, the value of α l is 0.7, 0.8, etc.; if the technician wants to use the visual feature to dominate the positioning result, he can increase the value of α l The value of large α c , for example, the value of α c is 0.7, 0.8, etc.; if the technician wants the laser feature and the visual feature to play an equal role in the positioning result, the value of α l and α c can be both 0.5.
可以理解的是，在步骤S101-步骤S105执行完成之后，定位***即完成了一次完整的定位过程。在车辆的行驶过程中，由于车辆的位姿时刻变化，定位过程也是不断进行的。It can be understood that, after the execution of steps S101 to S105 is completed, the positioning system has completed a complete positioning process. During the driving process of the vehicle, since the pose of the vehicle changes constantly, the positioning process is also continuously performed.
可以理解的是，受里程计误差或者其他因素的影响，随着定位次数的增加，采样点的分布可能会出现发散的现象。其中，采样点的发散条件可以由本领域技术人员自行设置，例如：当有预设比例的采样点对应的当前估计位姿的第二权值低于预设阈值时，认为采样点发散；或者，当有预设比例的采样点对应的当前估计位姿的第一权值低于预设阈值时，认为采样点发散。当采样点发散时，控制单元可以重新选与之前数量相同的采样点，本申请实施例对重 选采样点的方式同样不做限定，例如：在第一权值和/或者第二权值较高的一个或者多个当前估计位姿周围选取一定比例(例如：90％)的采样点，在GNSS/INS组合模块输出的位姿周围选取剩余比例(例如：10％)的采样点。另外，控制单元还可以周期性地重选采样点，例如每隔100个定位帧作为一个重选周期，重选采样点。It is understandable that, due to the influence of odometer errors or other factors, with the increase of the number of positioning times, the distribution of sampling points may diverge. Wherein, the divergence condition of the sampling points can be set by those skilled in the art, for example: when the second weight of the currently estimated pose corresponding to a preset ratio of sampling points is lower than the preset threshold, the sampling points are considered divergent; or, When the first weight of the currently estimated pose corresponding to a preset ratio of sampling points is lower than the preset threshold, the sampling points are considered to be divergent. When the sampling points diverge, the control unit may reselect the same number of sampling points as before. The embodiment of the present application also does not limit the method of reselection of sampling points. For example, when the first weight and/or the second weight are compared A certain proportion (for example: 90%) of sampling points are selected around the high one or more currently estimated poses, and the remaining proportion (for example: 10%) of sampling points are selected around the poses output by the GNSS/INS combination module. In addition, the control unit may also reselect the sampling points periodically, for example, every 100 positioning frames is regarded as a reselection period, and the sampling points are reselected.
本申请实施例提供的定位方法，能够分别基于激光特征的匹配确定当前估计位姿的第一权值，基于视觉语义信息和路面语义信息的匹配(即视觉特征的匹配)确定当前估计位姿的第二权值，然后根据的第一权值和第二权值对当前估计位姿加权求平均，得到车辆的定位结果，从而实现了将基于激光特征匹配的激光定位技术和基于视觉特征匹配的视觉定位技术进行融合，提高了定位效率。并且，本申请实施例提供的定位方法将激光特征和视觉语义信息编码存储在同一张定位地图中，实现了地图数据的融合，降低了定位过程中产生的数据开销和计算开销。The positioning method provided by the embodiment of the present application can respectively determine the first weight of the currently estimated pose based on the matching of laser features, and determine the current estimated pose based on the matching of visual semantic information and road semantic information (that is, the matching of visual features). The second weight, and then weighted and averaged the current estimated pose according to the first weight and the second weight to obtain the positioning result of the vehicle, thus realizing the combination of the laser positioning technology based on laser feature matching and the visual feature matching based laser positioning technology. The visual positioning technology is integrated to improve the positioning efficiency. Moreover, the positioning method provided by the embodiments of the present application encodes and stores laser features and visual semantic information in the same positioning map, which realizes the fusion of map data and reduces the data overhead and calculation overhead generated in the positioning process.
上述本申请提供的实施例对定位方法的各方案进行了介绍。可以理解的是，定位***为了实现上述功能，可以包含执行各个功能相应的硬件结构和/或软件模块。本领域技术人员应该很容易意识到，结合本文中所公开的实施例描述的各示例的单元及算法步骤，本申请能够以硬件或硬件和计算机软件的结合形式来实现。某个功能究竟以硬件还是计算机软件驱动硬件的方式来执行，取决于技术方案的特定应用和设计约束条件。专业技术人员可以对每个特定的应用来使用不同方法来实现所描述的功能，但是这种实现不应认为超出本申请的范围。The above-mentioned embodiments provided in this application introduce various solutions of the positioning method. It can be understood that, in order to realize the above-mentioned functions, the positioning system may include corresponding hardware structures and/or software modules for performing each function. Those skilled in the art should easily realize that the present application can be implemented in hardware or a combination of hardware and computer software with the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein. Whether a function is performed by hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Skilled artisans may implement the described functionality using different methods for each particular application, but such implementations should not be considered beyond the scope of this application.
在一个实施例中，定位***可以通过如图8所示的硬件结构实现相应的功能，该定位***可以安装在车辆中，一些器件的安装方式可以如图4所示。其中，GNSS/INS组合模块130，用于确定车辆的初始位姿；存储器190，用于存储定位地图，定位地图包括相互拼接的多张定位图片，定位图片包括色彩通道，色彩通道存储有第一激光特征的编码和视觉语义信息的编码；激光雷达110，用于采集点云数据，点云数据包含第二激光特征；摄像头120，用于采集图像数据，图像数据包含至少一个路面语义信息；控制单元140，用于在初始位姿周围生成N个采样点C 1～C N，N为正整数；控制单元140，还用于根据车辆的当前预测位姿从定位地图中提取第一激光特征和至少一个视觉语义信息；控制单元140，还用于对于任意采样点C n，n为正整数，n≤N，根据其对应的当前估计位姿P n，将提取自定位地图的第一激光特征与第二激光特征进行匹配，以确定当前估计位姿P n的第一权值，第二激光特征是从激光雷达采集的点云数据中提取的；控制单元140，还用于对于任意采样点C n，根据当前估计位姿P n，将至少一个视觉语义信息与至少一个路面语义信息进行匹配，以确定当前估计位姿P n的第二权值，至少一个路面语义信息是从摄像头采集的图像数据中提取的；控制单元140，还用于根据N个采样点C 1～C N的当前估计位姿P 1-P N及其第一权值和第二权值，计算当前估计位姿P 1-P N的加权平均值，以加权求平均值作为车辆的当前位姿。 In one embodiment, the positioning system may implement corresponding functions through the hardware structure shown in FIG. 8 , the positioning system may be installed in the vehicle, and the installation manner of some components may be as shown in FIG. 4 . Among them, the GNSS/INS combination module 130 is used to determine the initial pose of the vehicle; the memory 190 is used to store the positioning map, the positioning map includes a plurality of positioning pictures spliced with each other, the positioning picture includes a color channel, and the color channel stores the first coding of laser features and coding of visual semantic information; lidar 110 for collecting point cloud data, point cloud data containing second laser features; camera 120 for collecting image data, image data containing at least one pavement semantic information; control The unit 140 is used to generate N sampling points C 1 to C N around the initial pose, where N is a positive integer; the control unit 140 is also used to extract the first laser feature and the At least one piece of visual semantic information; the control unit 140 is further configured to, for any sampling point C n , n is a positive integer, n≤N, according to its corresponding current estimated pose P n , extract the first laser feature of the self-positioning map Matching with the second laser feature to determine the first weight of the current estimated pose P n , the second laser feature is extracted from the point cloud data collected by the lidar; the control unit 140 is also used for any sampling point C n , according to the current estimated pose P n , at least one visual semantic information is matched with at least one pavement semantic information to determine the second weight of the current estimated pose P n , and the at least one pavement semantic information is collected from the camera extracted from the image data; the control unit 140 is further configured to calculate the current estimated pose according to the current estimated poses P 1 -PN of the N sampling points C 1 to CN and their first weights and second weights The weighted average of P 1 - P N , and the weighted average is taken as the current pose of the vehicle.
在一个实施例中，定位图片包括第一色彩通道，第一色彩通道用于存储视觉语义信息的编码。In one embodiment, the positioning picture includes a first color channel used to store an encoding of visual semantic information.
在一个实施例中，定位图片还包括第二色彩通道，第二色彩通道用于存储第一激光特征的编码。In one embodiment, the positioning picture further includes a second color channel for storing the encoding of the first laser characteristic.
在一个实施例中，视觉语义信息的编码包括标志位、类型编码和亮度编码中的至少一种；标志位用于表示路面标志的类型，类型编码用于表示路面标志的内容，亮度信息编码用于表示图片的亮度信息。In one embodiment, the coding of visual semantic information includes at least one of flag bit, type coding and brightness coding; the flag bit is used to represent the type of the road marking, the type coding is used to represent the content of the road marking, and the brightness information coding is used for Used to represent the brightness information of the picture.
在一个实施例中，当控制单元140用于根据车辆的当前预测位姿从定位地图中提取至少 一个视觉语义信息时：控制单元140，具体用于根据当前预测位姿从定位地图中提取局部定位地图，局部定位地图包含M张定位图片，M张定位图片包含当前预测位姿所在的第一图片，以及第一图片附近的M-1张第二图片，M为大于1的正整数；控制单元140，还用于从局部定位地图中提取至少一个视觉语义信息。In one embodiment, when the control unit 140 is configured to extract at least one visual semantic information from the positioning map according to the current predicted pose of the vehicle: the control unit 140 is specifically configured to extract the local positioning from the positioning map according to the current predicted pose Map, the local positioning map includes M positioning pictures, the M positioning pictures include the first picture where the current predicted pose is located, and M-1 second pictures near the first picture, where M is a positive integer greater than 1; the control unit 140, which is further used for extracting at least one visual semantic information from the local localization map.
在一个实施例中，当控制单元140用于对于任意采样点C n，根据当前估计位姿P n，将至少一个视觉语义信息与至少一个路面语义信息进行匹配，以确定当前估计位姿P n的第二权值时：控制单元140，具体用于从至少一个路面语义信息中确定至少一个有效路面语义信息，每一个有效路面语义信息的像素数量在预设范围内；控制单元140，还用于根据当前估计位姿P n将至少一个有效路面语义信息投影到局部定位地图的坐标系中；控制单元140，还用于确定至少一个有效路面语义信息和至少一个视觉语义信息的语义关联关系；控制单元140，还用于对每一对语义关联的有效路面语义信息和视觉语义信息进行语义匹配，根据语义匹配结果确定第二权值。 In one embodiment, when the control unit 140 is configured to match at least one visual semantic information with at least one road semantic information according to the current estimated pose P n for any sampling point C n to determine the current estimated pose P n When the second weight is: the control unit 140 is specifically configured to determine at least one valid pavement semantic information from the at least one pavement semantic information, and the number of pixels of each valid pavement semantic information is within a preset range; the control unit 140 is also configured to use projecting at least one valid pavement semantic information into the coordinate system of the local positioning map according to the current estimated pose P n ; the control unit 140 is further configured to determine the semantic association relationship between the at least one valid pavement semantic information and the at least one visual semantic information; The control unit 140 is further configured to perform semantic matching on each pair of semantically related valid pavement semantic information and visual semantic information, and determine the second weight according to the semantic matching result.
在一个实施例中，当控制单元140用于确定至少一个有效路面语义信息和至少一个视觉语义信息的语义关联关系时：控制单元140，具体用于计算任意有效路面语义信息a i的语义权值和任意视觉语义信息b j的语义权值；控制单元140，还用于根据有效路面语义信息a i的语义权值和视觉语义信息b j的语义权值的差值，确定有效路面语义信息a i和视觉语义信息b j的语义关联度；控制单元140，还用于当语义关联度小于预设第一阈值时，确定有效路面语义信息a i和视觉语义信息b j具有语义关联。 In one embodiment, when the control unit 140 is used to determine the semantic relationship between at least one valid pavement semantic information and at least one visual semantic information: the control unit 140 is specifically adapted to calculate the semantic weight of any valid pavement semantic information a i and the semantic weight of any visual semantic information b j ; the control unit 140 is further configured to determine the effective pavement semantic information a according to the difference between the semantic weight of the effective pavement semantic information a i and the semantic weight of the visual semantic information b j The semantic correlation degree between i and the visual semantic information b j ; the control unit 140 is further configured to determine that the effective road semantic information a i and the visual semantic information b j have semantic correlation when the semantic correlation degree is less than the preset first threshold.
在一个实施例中，当控制单元140用于对每一对语义关联的有效路面语义信息和视觉语义信息进行语义匹配，根据语义匹配结果确定第二权值时：控制单元140，具体用于分别计算每一对语义关联的有效路面语义信息和视觉语义信息的匹配距离；控制单元140，还用于将计算得到的各个匹配距离的加权求和得到总匹配距离；控制单元140，还用于根据总匹配距离确定第二权值。In one embodiment, when the control unit 140 is configured to perform semantic matching on each pair of semantically associated valid pavement semantic information and visual semantic information, and determine the second weight according to the semantic matching result: the control unit 140 is specifically configured to respectively Calculate the matching distance of each pair of semantically associated valid pavement semantic information and visual semantic information; the control unit 140 is also used for the weighted summation of the respective matching distances obtained by calculation to obtain the total matching distance; the control unit 140 is also used for according to The total matching distance determines the second weight.
在一个实施例中，当控制单元140用于根据N个采样点C 1～C N的当前估计位姿P 1-P N及其第一权值和第二权值，计算当前估计位姿P 1-P N的加权平均值，以加权求平均值作为车辆的当前位姿时：控制单元140，具体用于使用当前估计位姿P 1-P N的第一权值对当前估计位姿P 1-P N加权求平均，得到第一加权平均值；控制单元140，还用于使用当前估计位姿P 1-P N的第二权值对当前估计位姿P 1-P N加权求平均，得到第二加权平均值；控制单元140，还用于对第一加权平均值和第二加权平均值加权求平均，得到加权平均值，以加权求平均值作为车辆的当前位姿。 In one embodiment, when the control unit 140 is configured to calculate the current estimated pose P according to the current estimated poses P 1 -PN of the N sampling points C 1 -CN and their first weights and second weights When the weighted average value of 1- PN is used as the current pose of the vehicle: the control unit 140 is specifically configured to use the first weight of the current estimated pose P 1 -PN to determine the current estimated pose P 1 -P N is weighted and averaged to obtain a first weighted average value; the control unit 140 is further configured to use the second weight of the current estimated pose P 1 -PN to weight the current estimated pose P 1 -PN , to obtain a second weighted average value; the control unit 140 is further configured to perform a weighted average of the first weighted average value and the second weighted average value to obtain a weighted average value, and use the weighted average value as the current pose of the vehicle.
在一个实施例中，路面语义信息包括：包含有至少一个路面标志的像素块、像素块的像素数量、每个像素所属的路面标志的类型。In one embodiment, the pavement semantic information includes: a pixel block containing at least one pavement marker, the number of pixels in the pixel block, and the type of pavement marker to which each pixel belongs.
在一个实施例中，定位***还包括里程计180；控制单元140，还用于根据里程计180数据确定车辆在当前时刻t和第一历史时刻t-1之间产生的相对位姿；控制单元140，还用于将采样点C n在第一历史时刻t-1对应的预测位姿与相对位姿相加得到当前预测位姿。 In one embodiment, the positioning system further includes an odometer 180; the control unit 140 is further configured to determine the relative pose of the vehicle between the current time t and the first historical time t-1 according to the data of the odometer 180; the control unit 140 is further configured to add the predicted pose corresponding to the sampling point C n at the first historical time t-1 and the relative pose to obtain the current predicted pose.
在一个实施例中，控制单元140，还用于当有预设比例的第一权值或者第二权值低于第二阈值时，重新生成N个采样点C 1～C N。 In one embodiment, the control unit 140 is further configured to regenerate N sampling points C 1 -C N when there is a preset ratio of the first weight or the second weight is lower than the second threshold.
在另一个实施例中，定位***可以通过图23所示的软件模块实现相应的功能。如图23所示，定位***可以包括采样点生成模块810、提取模块820、第一匹配模块830、第二匹配模块840、求解模块850。下面对上述模块的功能进行具体说明：In another embodiment, the positioning system may implement corresponding functions through the software modules shown in FIG. 23 . As shown in FIG. 23 , the positioning system may include a sampling point generation module 810 , an extraction module 820 , a first matching module 830 , a second matching module 840 , and a solving module 850 . The functions of the above modules are described in detail below:
采样点生成模块810，用于确定车辆的初始位姿，在初始位姿周围生成N个采样点C 1～C N，N为正整数； The sampling point generating module 810 is used for determining the initial pose of the vehicle, and generating N sampling points C 1 -C N around the initial pose, where N is a positive integer;
提取模块820，用于根据车辆的当前预测位姿从定位地图中提取第一激光特征和至少一个视觉语义信息。The extraction module 820 is configured to extract the first laser feature and at least one visual semantic information from the positioning map according to the current predicted pose of the vehicle.
第一匹配模块830，用于对于任意采样点C n，n为正整数，n≤N，根据其对应的当前估计位姿P n，将提取自定位地图的第一激光特征与第二激光特征进行匹配，以确定当前估计位姿P n的第一权值，第二激光特征是从激光雷达采集的点云数据中提取的。 The first matching module 830 is configured to, for any sampling point C n , where n is a positive integer, n≤N, according to its corresponding current estimated pose P n , extract the first laser feature and the second laser feature of the self-positioning map Matching is performed to determine the first weight of the current estimated pose Pn , and the second laser feature is extracted from the point cloud data collected by the lidar.
第二匹配模块840，用于对于任意采样点C n，根据当前估计位姿P n，将至少一个视觉语义信息与至少一个路面语义信息进行匹配，以确定当前估计位姿P n的第二权值，至少一个路面语义信息是从摄像头采集的图像数据中提取的。 The second matching module 840 is configured to, for any sampling point C n , match at least one visual semantic information with at least one pavement semantic information according to the current estimated pose P n to determine the second weight of the current estimated pose P n value, at least one pavement semantic information is extracted from the image data collected by the camera.
求解模块850，用于根据N个采样点C 1～C N的当前估计位姿P 1-P N及其第一权值和第二权值，计算当前估计位姿P 1-P N的加权平均值，以加权求平均值作为车辆的当前位姿。 The solving module 850 is configured to calculate the weight of the current estimated pose P 1 -PN according to the current estimated pose P 1 -PN of the N sampling points C 1 -CN and its first weight and second weight The average value is calculated by weighting as the current pose of the vehicle.
其中，定位地图包括相互拼接的多张定位图片，定位图片包括色彩通道，第一激光特征的编码和视觉语义信息的编码存储在色彩通道中。The positioning map includes a plurality of positioning pictures that are spliced with each other, the positioning pictures include a color channel, and the encoding of the first laser feature and the encoding of the visual semantic information are stored in the color channel.
本申请实施例提供的定位***，能够分别基于激光特征的匹配确定当前估计位姿的第一权值，基于视觉语义信息和路面语义信息的匹配(即视觉特征的匹配)确定当前估计位姿的第二权值，然后根据的第一权值和第二权值计算当前估计位姿的加权平均值，以加权求平均值作为车辆的当前位姿，从而实现了将基于激光特征匹配的激光定位技术和基于视觉特征匹配的视觉定位技术进行融合，提高了定位效率。并且，本申请实施例提供的技术方案将激光特征和视觉语义信息编码存储在同一张定位地图中，实现了地图数据的融合，降低了定位过程中产生的数据开销和计算开销。The positioning system provided by the embodiment of the present application can determine the first weight of the currently estimated pose based on the matching of laser features, and determine the current estimated pose based on the matching of visual semantic information and road semantic information (ie, matching of visual features). The second weight, and then calculate the weighted average of the current estimated pose according to the first weight and the second weight, and use the weighted average as the current pose of the vehicle, thereby realizing the laser positioning based on laser feature matching. The technology is integrated with the visual positioning technology based on visual feature matching, which improves the positioning efficiency. In addition, the technical solutions provided by the embodiments of the present application encode and store laser features and visual semantic information in the same positioning map, which realizes the fusion of map data and reduces the data overhead and computing overhead generated in the positioning process.
本申请实施例还提供了一种车辆，该车辆可以包含前述各实施例提供的定位***，并且用户执行前述各个实施例提供的定位方法。The embodiments of the present application further provide a vehicle, which may include the positioning system provided by the foregoing embodiments, and the user executes the positioning methods provided by the foregoing embodiments.
本申请实施例还提供一种计算机可读存储介质，该计算机可读存储介质中存储有指令，当其在计算机上运行时，使得计算机执行上述各方面的方法。Embodiments of the present application further provide a computer-readable storage medium, where instructions are stored in the computer-readable storage medium, when the computer-readable storage medium runs on a computer, the computer executes the methods of the above aspects.
本申请实施例还提供了一种包含指令的计算机程序产品，当其在计算机上运行时，使得计算机执行上述各方面的方法。Embodiments of the present application also provide a computer program product containing instructions, which, when run on a computer, cause the computer to execute the methods of the above aspects.
以上的具体实施方式，对本申请实施例的目的、技术方案和有益效果进行了进一步详细说明，所应理解的是，以上仅为本申请实施例的具体实施方式而已，并不用于限定本申请实施例的保护范围，凡在本申请实施例的技术方案的基础之上，所做的任何修改、等同替换、改进等，均应包括在本申请实施例的保护范围之内。The above specific embodiments further describe in detail the purposes, technical solutions and beneficial effects of the embodiments of the present application. It should be understood that the above are only specific implementations of the embodiments of the present application, and are not intended to limit the implementation of the present application. Any modification, equivalent replacement, improvement, etc. made on the basis of the technical solutions of the embodiments of the present application shall be included in the protection scope of the embodiments of the present application.

Claims (27)

  1. 一种定位方法，其特征在于，包括：A positioning method, comprising:
    确定车辆的初始位姿，在所述初始位姿周围生成N个采样点C 1～C N，N为正整数； Determine the initial pose of the vehicle, and generate N sampling points C 1 to C N around the initial pose, where N is a positive integer;
    根据所述车辆的当前预测位姿从定位地图中提取第一激光特征和至少一个视觉语义信息；对于任意采样点C n，n为正整数，n≤N，根据其对应的当前估计位姿P n，将提取自定位地图的第一激光特征与第二激光特征进行匹配，以确定所述当前估计位姿P n的第一权值，所述第二激光特征是从激光雷达采集的点云数据中提取的；以及， Extract the first laser feature and at least one visual semantic information from the positioning map according to the current predicted pose of the vehicle; for any sampling point C n , n is a positive integer, n≤N, according to its corresponding current estimated pose P n , the first laser feature extracted from the localization map is matched with the second laser feature to determine the first weight of the current estimated pose P n , and the second laser feature is the point cloud collected from the lidar extracted from the data; and,
    对于任意采样点C n，根据所述当前估计位姿P n，将所述至少一个视觉语义信息与至少一个路面语义信息进行匹配，以确定所述当前估计位姿P n的第二权值，所述至少一个路面语义信息是从摄像头采集的图像数据中提取的； For any sampling point C n , according to the current estimated pose P n , the at least one visual semantic information and at least one road semantic information are matched to determine the second weight of the current estimated pose P n , The at least one pavement semantic information is extracted from image data collected by a camera;
    根据所述N个采样点C 1～C N的当前估计位姿P 1-P N及其所述第一权值和所述第二权值，计算所述当前估计位姿P 1-P N的加权平均值，以所述加权求平均值作为所述车辆的当前位姿； According to the current estimated poses P 1 -PN of the N sampling points C 1 -CN and the first weight and the second weight, the current estimated pose P 1 -PN is calculated The weighted average of , and the weighted average is taken as the current pose of the vehicle;
    其中，所述定位地图包括相互拼接的多张定位图片，所述定位图片包括色彩通道，所述第一激光特征的编码和所述视觉语义信息的编码存储在所述色彩通道中。The positioning map includes a plurality of positioning pictures spliced with each other, the positioning pictures include a color channel, and the encoding of the first laser feature and the encoding of the visual semantic information are stored in the color channel.
  2. 根据权利要求1所述的方法，其特征在于，所述定位图片包括第一色彩通道，所述第一色彩通道用于存储所述视觉语义信息的编码。The method according to claim 1, wherein the positioning picture comprises a first color channel, and the first color channel is used to store the encoding of the visual semantic information.
  3. 根据权利要求2所述的方法，其特征在于，所述定位图片还包括第二色彩通道，所述第二色彩通道用于存储所述第一激光特征的编码。3. The method of claim 2, wherein the positioning picture further comprises a second color channel for storing the code of the first laser feature.
  4. 根据权利要求1-3任一项所述的方法，其特征在于，所述视觉语义信息的编码包括标志位、类型编码和亮度编码中的至少一种；所述标志位用于表示路面标志的类型，所述类型编码用于表示路面标志的内容，所述亮度信息编码用于表示图片的亮度信息。The method according to any one of claims 1-3, wherein the coding of the visual semantic information comprises at least one of a flag bit, type coding and luminance coding; the flag bit is used to represent the Type, the type code is used to represent the content of the road marking, and the brightness information code is used to represent the brightness information of the picture.
  5. 根据权利要求1-4任一项所述的方法，其特征在于，所述至少一个视觉语义信息通过以下步骤提取：The method according to any one of claims 1-4, wherein the at least one visual semantic information is extracted by the following steps:
    根据所述当前预测位姿从所述定位地图中提取局部定位地图，所述局部定位地图包含M张定位图片，所述M张定位图片包含所述当前预测位姿所在的第一图片，以及所述第一图片附近的M-1张第二图片，M为大于1的正整数；A local positioning map is extracted from the positioning map according to the current predicted pose, the local positioning map includes M positioning pictures, the M positioning pictures include the first picture where the current predicted pose is located, and the M-1 second pictures near the first picture, where M is a positive integer greater than 1;
    从所述局部定位地图中提取所述至少一个视觉语义信息。The at least one visual semantic information is extracted from the local localization map.
  6. 根据权利要求5所述的方法，其特征在于，所述对于任意采样点C n，根据所述当前估计位姿P n，将所述至少一个视觉语义信息与至少一个路面语义信息进行匹配，以确定所述当前估计位姿P n的第二权值，包括： The method according to claim 5, wherein, for any sampling point C n , according to the current estimated pose P n , the at least one visual semantic information and at least one road semantic information are matched to obtain Determining the second weight of the current estimated pose P n includes:
    从所述至少一个路面语义信息中确定至少一个有效路面语义信息，每一个所述有效路面语义信息的像素数量在预设范围内；determining at least one valid pavement semantic information from the at least one pavement semantic information, and the number of pixels of each valid pavement semantic information is within a preset range;
    根据所述当前估计位姿P n将所述至少一个有效路面语义信息投影到所述局部定位地图的坐标系中； Projecting the at least one valid road semantic information into the coordinate system of the local positioning map according to the current estimated pose P n ;
    确定所述至少一个有效路面语义信息和所述至少一个视觉语义信息的语义关联关系；determining the semantic association relationship between the at least one valid pavement semantic information and the at least one visual semantic information;
    对每一对语义关联的所述有效路面语义信息和所述视觉语义信息进行语义匹配，根据语义匹配结果确定所述第二权值。Semantic matching is performed on each pair of semantically associated valid pavement semantic information and visual semantic information, and the second weight is determined according to the semantic matching result.
  7. 根据权利要求6所述的方法，其特征在于，所述确定所述至少一个有效路面语义信息和所述至少一个视觉语义信息的语义关联关系，包括：The method according to claim 6, wherein the determining the semantic association relationship between the at least one valid pavement semantic information and the at least one visual semantic information comprises:
    计算任意有效路面语义信息a i的语义权值和任意视觉语义信息b j的语义权值； Calculate the semantic weight of any valid pavement semantic information a i and the semantic weight of any visual semantic information b j ;
    根据所述有效路面语义信息a i的语义权值和所述视觉语义信息b j的语义权值的差值，确定所述有效路面语义信息a i和所述视觉语义信息b j的语义关联度； According to the difference between the semantic weight of the effective pavement semantic information a i and the semantic weight of the visual semantic information b j , determine the semantic relevance of the effective pavement semantic information a i and the visual semantic information b j ;
    当所述语义关联度小于预设第一阈值时，确定所述有效路面语义信息a i和所述视觉语义信息b j具有语义关联。 When the semantic correlation degree is less than a preset first threshold, it is determined that the effective road semantic information a i and the visual semantic information b j have semantic correlation.
  8. 根据权利要求6或7所述的方法，其特征在于，所述对每一对语义关联的所述有效路面语义信息和所述视觉语义信息进行语义匹配，根据语义匹配结果确定所述第二权值，包括：The method according to claim 6 or 7, wherein the semantic matching is performed on each pair of semantically associated valid pavement semantic information and the visual semantic information, and the second weight is determined according to a semantic matching result. values, including:
    分别计算每一对语义关联的所述有效路面语义信息和所述视觉语义信息的匹配距离；respectively calculating the matching distance between the effective pavement semantic information and the visual semantic information for each pair of semantically associated semantic information;
    将计算得到的各个所述匹配距离的加权求和得到总匹配距离；A total matching distance is obtained by the weighted summation of each of the calculated matching distances;
    根据所述总匹配距离确定所述第二权值。The second weight is determined according to the total matching distance.
  9. 根据权利要求1-8任一项所述的方法，其特征在于，所述根据所述N个采样点C 1～C N的当前估计位姿P 1-P N及其所述第一权值和所述第二权值，计算所述当前估计位姿P 1-P N的加权平均值，以所述加权求平均值更新所述当前预测位姿，包括： The method according to any one of claims 1-8, wherein the currently estimated poses P 1 -PN and the first weights thereof according to the N sampling points C 1 -CN and the second weight, calculating the weighted average of the current estimated pose P 1 - P N , and updating the current predicted pose with the weighted average value, including:
    使用所述当前估计位姿P 1-P N的所述第一权值对所述当前估计位姿P 1-P N加权求平均，得到第一加权平均值； Using the first weight of the currently estimated pose P 1 -PN to weight the current estimated pose P 1 -PN to obtain a first weighted average;
    使用所述当前估计位姿P 1-P N的所述第二权值对所述当前估计位姿P 1-P N加权求平均，得到第二加权平均值； Using the second weight of the current estimated pose P 1 -PN to weight the current estimated pose P 1 -PN to obtain a second weighted average;
    对所述第一加权平均值和所述第二加权平均值加权求平均，得到所述加权平均值，以所述加权求平均值作为所述车辆的当前位姿。The first weighted average value and the second weighted average value are weighted and averaged to obtain the weighted average value, and the weighted average value is used as the current pose of the vehicle.
  10. 根据权利要求1-9任一项所述的方法，其特征在于，所述路面语义信息包括：包含有至少一个路面标志的像素块、所述像素块的像素数量、每个像素所属的路面标志的类型。The method according to any one of claims 1-9, wherein the pavement semantic information comprises: a pixel block containing at least one pavement mark, the number of pixels in the pixel block, and the pavement mark to which each pixel belongs. type.
  11. 根据权利要求1-10任一项所述的方法，其特征在于，所述当前预测位姿通过以下步骤确定：The method according to any one of claims 1-10, wherein the current predicted pose is determined by the following steps:
    根据里程计数据确定车辆在当前时刻t和第一历史时刻t-1之间产生的相对位姿；Determine the relative pose of the vehicle between the current time t and the first historical time t-1 according to the odometer data;
    将所述采样点C n在所述第一历史时刻t-1对应的预测位姿与所述相对位姿相加得到所述当前预测位姿。 The current predicted pose is obtained by adding the predicted pose corresponding to the sampling point C n at the first historical time t-1 and the relative pose.
  12. 根据权利要求1-11任一项所述的方法，其特征在于，还包括：The method according to any one of claims 1-11, further comprising:
    当有预设比例的所述第一权值或者所述第二权值低于第二阈值时，重新生成所述N个采样点C 1～C N。 When there is a preset proportion of the first weight or the second weight is lower than the second threshold, the N sampling points C 1 -C N are regenerated.
  13. 一种定位***，其特征在于，包括：A positioning system, comprising:
    安装于车辆的GNSS/INS组合模块、控制单元、存储器、激光雷达和摄像头；Vehicle-mounted GNSS/INS combination module, control unit, memory, lidar and camera;
    所述GNSS/INS组合模块，用于确定车辆的初始位姿；The GNSS/INS combination module is used to determine the initial pose of the vehicle;
    所述存储器，用于存储定位地图，所述定位地图包括相互拼接的多张定位图片，所述定位图片包括色彩通道，所述色彩通道存储有第一激光特征的编码和视觉语义信息的编码；The memory is used to store a positioning map, the positioning map includes a plurality of positioning pictures spliced with each other, and the positioning picture includes a color channel, and the color channel stores the encoding of the first laser feature and the encoding of the visual semantic information;
    所述激光雷达，用于采集点云数据，所述点云数据包含第二激光特征；The lidar is used to collect point cloud data, and the point cloud data includes second laser features;
    所述摄像头，用于采集图像数据，所述图像数据包含至少一个路面语义信息；the camera is used to collect image data, the image data includes at least one pavement semantic information;
    所述控制单元，用于在所述初始位姿周围生成N个采样点C 1～C N，N为正整数； the control unit, configured to generate N sampling points C 1 to C N around the initial pose, where N is a positive integer;
    所述控制单元，还用于根据所述车辆的当前预测位姿从定位地图中提取第一激光特征和至少一个视觉语义信息；The control unit is further configured to extract the first laser feature and at least one visual semantic information from the positioning map according to the current predicted pose of the vehicle;
    所述控制单元，还用于对于任意采样点C n，n为正整数，n≤N，根据其对应的当前估计位姿P n，将提取自定位地图的第一激光特征与第二激光特征进行匹配，以确定所述当前估计位姿P n的第一权值，所述第二激光特征是从激光雷达采集的点云数据中提取的； The control unit is further configured to, for any sampling point C n , where n is a positive integer, n≤N, according to its corresponding current estimated pose P n , extract the first laser feature and the second laser feature of the self-positioning map performing matching to determine the first weight of the currently estimated pose P n , and the second laser feature is extracted from the point cloud data collected by the lidar;
    所述控制单元，还用于对于任意采样点C n，根据所述当前估计位姿P n，将所述至少一个视觉语义信息与至少一个路面语义信息进行匹配，以确定所述当前估计位姿P n的第二权值，所述至少一个路面语义信息是从摄像头采集的图像数据中提取的； The control unit is further configured to, for any sampling point C n , match the at least one visual semantic information with at least one road semantic information according to the current estimated pose P n to determine the current estimated pose The second weight of P n , the at least one pavement semantic information is extracted from the image data collected by the camera;
    所述控制单元，还用于根据所述N个采样点C 1～C N的当前估计位姿P 1-P N及其所述第一权值和所述第二权值，计算所述当前估计位姿P 1-P N的加权平均值，以所述加权求平均值作为所述车辆的当前位姿。 The control unit is further configured to calculate the current estimated pose P 1 -PN of the N sampling points C 1 -CN and the first weight and the second weight of the current A weighted average of the poses P 1 - P N is estimated, and the weighted average is used as the current pose of the vehicle.
  14. 根据权利要求13所述的定位***，其特征在于，所述定位图片包括第一色彩通道，所述第一色彩通道用于存储所述视觉语义信息的编码。The positioning system according to claim 13, wherein the positioning picture includes a first color channel, and the first color channel is used to store the encoding of the visual semantic information.
  15. 根据权利要求13所述的定位***，其特征在于，所述定位图片还包括第二色彩通道，所述第二色彩通道用于存储所述第一激光特征的编码。14. The positioning system of claim 13, wherein the positioning picture further comprises a second color channel, wherein the second color channel is used to store the code of the first laser characteristic.
  16. 根据权利要求13-15任一项所述的定位***，其特征在于，所述视觉语义信息的编码包括标志位、类型编码和亮度编码中的至少一种；所述标志位用于表示路面标志的类型，所述类型编码用于表示路面标志的内容，所述亮度信息编码用于表示图片的亮度信息。The positioning system according to any one of claims 13-15, wherein the coding of the visual semantic information includes at least one of a marker bit, a type encoding and a brightness encoding; the marker bit is used to represent road signs type, the type code is used to represent the content of the road marking, and the brightness information code is used to represent the brightness information of the picture.
  17. 根据权利要求13-16任一项所述的定位***，其特征在于，当所述控制单元用于根据所述车辆的当前预测位姿从定位地图中提取所述至少一个视觉语义信息时：The positioning system according to any one of claims 13-16, wherein when the control unit is used to extract the at least one visual semantic information from the positioning map according to the current predicted pose of the vehicle:
    所述控制单元，具体用于根据所述当前预测位姿从所述定位地图中提取局部定位地图，所述局部定位地图包含M张定位图片，所述M张定位图片包含所述当前预测位姿所在的第一图片，以及所述第一图片附近的M-1张第二图片，M为大于1的正整数；The control unit is specifically configured to extract a local positioning map from the positioning map according to the current predicted pose, where the local positioning map includes M positioning pictures, and the M positioning pictures include the current predicted pose The first picture where it is located, and M-1 second pictures near the first picture, where M is a positive integer greater than 1;
    所述控制单元，还用于从所述局部定位地图中提取所述至少一个视觉语义信息。The control unit is further configured to extract the at least one visual semantic information from the local positioning map.
  18. 根据权利要求13所述的定位***，其特征在于，当所述控制单元用于对于任意采样点C n，根据所述当前估计位姿P n，将所述至少一个视觉语义信息与至少一个路面语义信息进行匹配，以确定所述当前估计位姿P n的第二权值时： The positioning system according to claim 13, wherein, when the control unit is used for any sampling point C n , according to the current estimated pose P n , the at least one visual semantic information is associated with the at least one road surface When matching semantic information to determine the second weight of the current estimated pose P n :
    所述控制单元，具体用于从所述至少一个路面语义信息中确定至少一个有效路面语义信息，每一个所述有效路面语义信息的像素数量在预设范围内；The control unit is specifically configured to determine at least one valid road semantic information from the at least one road semantic information, and the number of pixels of each valid road semantic information is within a preset range;
    所述控制单元，还用于根据所述当前估计位姿P n将所述至少一个有效路面语义信息投影到所述局部定位地图的坐标系中； The control unit is further configured to project the at least one valid road semantic information into the coordinate system of the local positioning map according to the current estimated pose Pn ;
    所述控制单元，还用于确定所述至少一个有效路面语义信息和所述至少一个视觉语义信息的语义关联关系；The control unit is further configured to determine the semantic association relationship between the at least one valid road semantic information and the at least one visual semantic information;
    所述控制单元，还用于对每一对语义关联的所述有效路面语义信息和所述视觉语义信息进行语义匹配，根据语义匹配结果确定所述第二权值。The control unit is further configured to perform semantic matching on each pair of the semantically associated valid road semantic information and the visual semantic information, and determine the second weight according to the semantic matching result.
  19. 根据权利要求18所述的定位***，其特征在于，当所述控制单元用于确定所述至少一个有效路面语义信息和所述至少一个视觉语义信息的语义关联关系时：The positioning system according to claim 18, wherein, when the control unit is used to determine the semantic association relationship between the at least one valid road semantic information and the at least one visual semantic information:
    所述控制单元，具体用于计算任意有效路面语义信息a i的语义权值和任意视觉语义信息b j的语义权值； The control unit is specifically used to calculate the semantic weight of any valid pavement semantic information a i and the semantic weight of any visual semantic information b j ;
    所述控制单元，还用于根据所述有效路面语义信息a i的语义权值和所述视觉语义信息b j的语义权值的差值，确定所述有效路面语义信息a i和所述视觉语义信息b j的语义关联度； The control unit is further configured to determine the effective pavement semantic information a i and the visual semantic information according to the difference between the semantic weight of the effective pavement semantic information a i and the semantic weight of the visual semantic information b j Semantic relevance of semantic information b j ;
    所述控制单元，还用于当所述语义关联度小于预设第一阈值时，确定所述有效路面语义 信息a i和所述视觉语义信息b j具有语义关联。 The control unit is further configured to determine that the effective pavement semantic information a i and the visual semantic information b j have semantic correlation when the semantic correlation degree is less than a preset first threshold.
  20. 根据权利要求18或19所述的定位***，其特征在于，当所述控制单元用于对每一对语义关联的所述有效路面语义信息和所述视觉语义信息进行语义匹配，根据语义匹配结果确定所述第二权值时：The positioning system according to claim 18 or 19, wherein when the control unit is used to perform semantic matching on each pair of semantically associated valid road semantic information and the visual semantic information, according to the semantic matching result When determining the second weight:
    所述控制单元，具体用于分别计算每一对语义关联的所述有效路面语义信息和所述视觉语义信息的匹配距离；The control unit is specifically configured to separately calculate the matching distance between the effective pavement semantic information and the visual semantic information for each pair of semantically associated semantic information;
    所述控制单元，还用于将计算得到的各个所述匹配距离的加权求和得到总匹配距离；The control unit is also used to obtain the total matching distance by the weighted summation of each of the matching distances obtained by calculation;
    所述控制单元，还用于根据所述总匹配距离确定所述第二权值。The control unit is further configured to determine the second weight according to the total matching distance.
  21. 根据权利要求13-20任一项所述的定位***，其特征在于，当所述控制单元用于根据所述N个采样点C 1～C N的当前估计位姿P 1-P N及其所述第一权值和所述第二权值，计算所述当前估计位姿P 1-P N的加权平均值，以所述加权求平均值作为所述车辆的当前位姿时： The positioning system according to any one of claims 13-20, characterized in that, when the control unit is used to estimate the poses P 1 -PN and the same according to the current estimated poses of the N sampling points C 1 -CN For the first weight and the second weight, the weighted average of the current estimated pose P 1 - P N is calculated, and the weighted average is used as the current pose of the vehicle:
    所述控制单元，具体用于使用所述当前估计位姿P 1-P N的所述第一权值对所述当前估计位姿P 1-P N加权求平均，得到第一加权平均值； The control unit is specifically configured to use the first weight of the current estimated pose P 1 -PN to perform a weighted average of the current estimated pose P 1 -PN to obtain a first weighted average;
    所述控制单元，还用于使用所述当前估计位姿P 1-P N的所述第二权值对所述当前估计位姿P 1-P N加权求平均，得到第二加权平均值； The control unit is further configured to use the second weight of the current estimated pose P 1 -PN to perform a weighted average of the current estimated pose P 1 -PN to obtain a second weighted average;
    所述控制单元，还用于对所述第一加权平均值和所述第二加权平均值加权求平均，得到所述加权平均值，以所述加权求平均值作为车辆的当前位姿。The control unit is further configured to perform a weighted average of the first weighted average value and the second weighted average value to obtain the weighted average value, and use the weighted average value as the current pose of the vehicle.
  22. 根据权利要求13-21任一项所述的定位***，其特征在于，所述路面语义信息包括：包含有至少一个路面标志的像素块、所述像素块的像素数量、每个像素所属的路面标志的类型。The positioning system according to any one of claims 13-21, wherein the pavement semantic information includes: a pixel block containing at least one pavement mark, the number of pixels in the pixel block, and the pavement to which each pixel belongs. Type of flag.
  23. 根据权利要求13-21任一项所述的定位***，其特征在于，还包括：里程计；The positioning system according to any one of claims 13-21, characterized in that, further comprising: an odometer;
    所述控制单元，还用于根据里程计数据确定车辆在当前时刻t和第一历史时刻t-1之间产生的相对位姿；The control unit is further configured to determine the relative pose of the vehicle between the current time t and the first historical time t-1 according to the odometer data;
    所述控制单元，还用于将所述采样点C n在所述第一历史时刻t-1对应的预测位姿与所述相对位姿相加得到所述当前预测位姿。 The control unit is further configured to add the predicted pose corresponding to the sampling point C n at the first historical time t-1 and the relative pose to obtain the current predicted pose.
  24. 根据权利要求13-23任一项所述的定位***，其特征在于，The positioning system according to any one of claims 13-23, wherein,
    所述控制单元，还用于当有预设比例的所述第一权值或者所述第二权值低于第二阈值时，重新生成所述N个采样点C 1～C N。 The control unit is further configured to regenerate the N sampling points C 1 -C N when there is a preset ratio of the first weight or the second weight is lower than a second threshold.
  25. 一种车辆，其特征在于，包括如权利要求13-24任一项所述的定位***。A vehicle, characterized by comprising the positioning system according to any one of claims 13-24.
  26. 一种计算机可读存储介质，其特征在于，所述计算机可读存储介质中存储有指令，当所述指令在计算机上运行时，使得所述计算机执行如权利要求1-12任一项所述的方法。A computer-readable storage medium, wherein instructions are stored in the computer-readable storage medium, and when the instructions are executed on a computer, the computer is made to perform the execution of any one of claims 1-12. Methods.
  27. 一种计算机程序产品，其特征在于，当所述计算机程序产品在计算机上运行时，使得所述计算机执行如权利要求1-12任一项所述的方法。A computer program product, characterized in that, when the computer program product runs on a computer, the computer is caused to execute the method according to any one of claims 1-12.
PCT/CN2021/082388 2020-08-28 2021-03-23 Positioning method, positioning system, and vehicle WO2022041706A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202010884916.3A CN114200481A (en) 2020-08-28 2020-08-28 Positioning method, positioning system and vehicle
CN202010884916.3 2020-08-28

Publications (1)

Publication Number Publication Date
WO2022041706A1 true WO2022041706A1 (en) 2022-03-03

Family

ID=80354480

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/082388 WO2022041706A1 (en) 2020-08-28 2021-03-23 Positioning method, positioning system, and vehicle

Country Status (2)

Country Link
CN (1) CN114200481A (en)
WO (1) WO2022041706A1 (en)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114972501A (en) * 2022-04-21 2022-08-30 东北大学 Visual positioning algorithm based on prior semantic map structure information and semantic information
CN115435795A (en) * 2022-09-21 2022-12-06 苏州轻棹科技有限公司 Vehicle positioning method based on all-around image and semantic map
CN115965682A (en) * 2022-12-16 2023-04-14 镁佳(北京)科技有限公司 Method and device for determining passable area of vehicle and computer equipment

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115294527B (en) * 2022-08-09 2023-05-16 中铁隧道局集团有限公司 Subway tunnel damage detection method based on computer vision
CN117804423A (en) * 2022-09-26 2024-04-02 华为云计算技术有限公司 Repositioning method and device
CN116559928B (en) * 2023-07-11 2023-09-22 新石器慧通（北京）科技有限公司 Pose information determining method, device and equipment of laser radar and storage medium

Citations (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160209846A1 (en) * 2015-01-19 2016-07-21 The Regents Of The University Of Michigan Visual Localization Within LIDAR Maps
CN105865449A (en) * 2016-04-01 2016-08-17 深圳杉川科技有限公司 Laser and vision-based hybrid location method for mobile robot
US20170307746A1 (en) * 2016-04-22 2017-10-26 Mohsen Rohani Systems and methods for radar-based localization
CN108319976A (en) * 2018-01-25 2018-07-24 北京三快在线科技有限公司 Build drawing method and device
US20180299273A1 (en) * 2017-04-17 2018-10-18 Baidu Online Network Technology (Beijing) Co., Ltd. Method and apparatus for positioning vehicle
WO2019018315A1 (en) * 2017-07-17 2019-01-24 Kaarta, Inc. Aligning measured signal data with slam localization data and uses thereof
CN109752724A (en) * 2018-12-26 2019-05-14 珠海市众创芯慧科技有限公司 A kind of image laser integral type navigation positioning system
US20190146062A1 (en) * 2017-11-15 2019-05-16 Baidu Online Network Technology (Beijing) Co., Ltd Laser point cloud positioning method and system
CN109887003A (en) * 2019-01-23 2019-06-14 亮风台（上海）信息科技有限公司 A kind of method and apparatus initialized for carrying out three-dimensional tracking
CN110389348A (en) * 2019-07-30 2019-10-29 四川大学 Positioning and air navigation aid and device based on laser radar and binocular camera
CN110986966A (en) * 2019-12-17 2020-04-10 苏州智加科技有限公司 Automatic driving positioning method and system for long-distance tunnel
CN111323004A (en) * 2018-12-16 2020-06-23 北京初速度科技有限公司 Initial position determining method and vehicle-mounted terminal
CN111429574A (en) * 2020-03-06 2020-07-17 上海交通大学 Mobile robot positioning method and system based on three-dimensional point cloud and vision fusion

Patent Citations (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160209846A1 (en) * 2015-01-19 2016-07-21 The Regents Of The University Of Michigan Visual Localization Within LIDAR Maps
CN105865449A (en) * 2016-04-01 2016-08-17 深圳杉川科技有限公司 Laser and vision-based hybrid location method for mobile robot
US20170307746A1 (en) * 2016-04-22 2017-10-26 Mohsen Rohani Systems and methods for radar-based localization
US20180299273A1 (en) * 2017-04-17 2018-10-18 Baidu Online Network Technology (Beijing) Co., Ltd. Method and apparatus for positioning vehicle
WO2019018315A1 (en) * 2017-07-17 2019-01-24 Kaarta, Inc. Aligning measured signal data with slam localization data and uses thereof
US20190146062A1 (en) * 2017-11-15 2019-05-16 Baidu Online Network Technology (Beijing) Co., Ltd Laser point cloud positioning method and system
CN108319976A (en) * 2018-01-25 2018-07-24 北京三快在线科技有限公司 Build drawing method and device
CN111323004A (en) * 2018-12-16 2020-06-23 北京初速度科技有限公司 Initial position determining method and vehicle-mounted terminal
CN109752724A (en) * 2018-12-26 2019-05-14 珠海市众创芯慧科技有限公司 A kind of image laser integral type navigation positioning system
CN109887003A (en) * 2019-01-23 2019-06-14 亮风台（上海）信息科技有限公司 A kind of method and apparatus initialized for carrying out three-dimensional tracking
CN110389348A (en) * 2019-07-30 2019-10-29 四川大学 Positioning and air navigation aid and device based on laser radar and binocular camera
CN110986966A (en) * 2019-12-17 2020-04-10 苏州智加科技有限公司 Automatic driving positioning method and system for long-distance tunnel
CN111429574A (en) * 2020-03-06 2020-07-17 上海交通大学 Mobile robot positioning method and system based on three-dimensional point cloud and vision fusion

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
LI RUI: "Research and System Design of SLAM Algorithm for Power Inspection Robot", MASTER THESIS, TIANJIN POLYTECHNIC UNIVERSITY, CN, no. 1, 15 January 2018 (2018-01-15), CN , XP055903995, ISSN: 1674-0246 *
SONG HARYONG; CHOI WONSUB; KIM HAEDONG: "Robust Vision-Based Relative-Localization Approach Using an RGB-Depth Camera and LiDAR Sensor Fusion", IEEE TRANSACTIONS ON INDUSTRIAL ELECTRONICS, IEEE SERVICE CENTER, PISCATAWAY, NJ., USA, vol. 63, no. 6, 1 June 2016 (2016-06-01), USA , pages 3725 - 3736, XP011609602, ISSN: 0278-0046, DOI: 10.1109/TIE.2016.2521346 *

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114972501A (en) * 2022-04-21 2022-08-30 东北大学 Visual positioning algorithm based on prior semantic map structure information and semantic information
CN115435795A (en) * 2022-09-21 2022-12-06 苏州轻棹科技有限公司 Vehicle positioning method based on all-around image and semantic map
CN115435795B (en) * 2022-09-21 2024-05-03 苏州轻棹科技有限公司 Vehicle positioning method based on looking-around image and semantic map
CN115965682A (en) * 2022-12-16 2023-04-14 镁佳(北京)科技有限公司 Method and device for determining passable area of vehicle and computer equipment
CN115965682B (en) * 2022-12-16 2023-09-01 镁佳(北京)科技有限公司 Vehicle passable area determining method and device and computer equipment

Also Published As

Publication number Publication date
CN114200481A (en) 2022-03-18

Similar Documents

Publication Publication Date Title
WO2022041706A1 (en) Positioning method, positioning system, and vehicle
JP7349792B2 (en) How to provide information for vehicle operation
CN107246868B (en) Collaborative navigation positioning system and navigation positioning method
CN108802785B (en) Vehicle self-positioning method based on high-precision vector map and monocular vision sensor
EP3497405B1 (en) System and method for precision localization and mapping
US20190051056A1 (en) Augmenting reality using semantic segmentation
JP2022106924A (en) Device and method for autonomous self-position estimation
CN113916242B (en) Lane positioning method and device, storage medium and electronic equipment
WO2021053393A1 (en) Systems and methods for monitoring traffic lane congestion
CN111238494A (en) Carrier, carrier positioning system and carrier positioning method
US11231283B2 (en) Localization with neural network based image registration of sensor data and map data
CN111856491A (en) Method and apparatus for determining the geographic position and orientation of a vehicle
Senlet et al. Satellite image based precise robot localization on sidewalks
CN112414417B (en) Automatic driving map generation method and device, electronic equipment and readable storage medium
Deng et al. Semantic segmentation-based lane-level localization using around view monitoring system
WO2023065342A1 (en) Vehicle, vehicle positioning method and apparatus, device, and computer-readable storage medium
CN114323050A (en) Vehicle positioning method and device and electronic equipment
Vaquero et al. Improving map re-localization with deep ‘movable’objects segmentation on 3D LiDAR point clouds
KR20220094416A (en) System for predicting position of objection near future
TW202020734A (en) Vehicle, vehicle positioning system, and vehicle positioning method
US11579622B2 (en) Systems and methods for utilizing images to determine the position and orientation of a vehicle
Tian et al. Vision-based mapping of lane semantics and topology for intelligent vehicles
Bao et al. Vehicle positioning with the integration of scene understanding and 3D map in urban environment
Lee et al. Map Matching-Based Driving Lane Recognition for Low-Cost Precise Vehicle Positioning on Highways
Velat et al. Vision based vehicle localization for autonomous navigation

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21859555

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21859555

Country of ref document: EP

Kind code of ref document: A1