CN112446234A - Position determination method and device based on data association - Google Patents
Position determination method and device based on data association Download PDFInfo
- Publication number
- CN112446234A CN112446234A CN201910800415.XA CN201910800415A CN112446234A CN 112446234 A CN112446234 A CN 112446234A CN 201910800415 A CN201910800415 A CN 201910800415A CN 112446234 A CN112446234 A CN 112446234A
- Authority
- CN
- China
- Prior art keywords
- road
- semantic features
- semantic
- group
- images
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 46
- 238000001514 detection method Methods 0.000 claims abstract description 26
- 238000006243 chemical reaction Methods 0.000 claims abstract description 17
- 230000004927 fusion Effects 0.000 claims description 8
- 238000012545 processing Methods 0.000 claims description 4
- 238000010586 diagram Methods 0.000 description 7
- 239000011159 matrix material Substances 0.000 description 4
- 239000003550 marker Substances 0.000 description 3
- 238000005096 rolling process Methods 0.000 description 2
- 238000012935 Averaging Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 230000004807 localization Effects 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
- G06V20/588—Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
- G01C21/00—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
- G01C21/26—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 specially adapted for navigation in a road network
- G01C21/28—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 specially adapted for navigation in a road network with correlation of data from several navigational instruments
- G01C21/30—Map- or contour-matching
- G01C21/32—Structuring or formatting of map data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/22—Matching criteria, e.g. proximity measures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
- G06F18/253—Fusion techniques of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
- G06V20/58—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- G06V20/582—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads of traffic signs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
- G06V20/58—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- G06V20/584—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads of vehicle lights or traffic lights
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Remote Sensing (AREA)
- Radar, Positioning & Navigation (AREA)
- Multimedia (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Evolutionary Computation (AREA)
- General Engineering & Computer Science (AREA)
- Evolutionary Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Computational Biology (AREA)
- Artificial Intelligence (AREA)
- Life Sciences & Earth Sciences (AREA)
- Automation & Control Theory (AREA)
- Image Analysis (AREA)
- Traffic Control Systems (AREA)
Abstract
The embodiment of the invention discloses a position determining method and device based on data association. The method comprises the following steps: acquiring a plurality of groups of road images acquired by camera equipment when a vehicle runs for a plurality of times in the same position area; carrying out feature detection on the road signs in each road image to obtain semantic features in each road image; matching semantic features among the road images in each group of road images to obtain matched semantic features belonging to the same road sign in the group of road images; performing three-dimensional reconstruction and coordinate system conversion on the matched semantic features to obtain a first position of the matched semantic features in a map; determining the associated semantic features belonging to the same road sign among the road images according to the matched semantic features in each group of road images; and fusing the first positions of the associated semantic features in each group of road images to obtain the second positions of the associated semantic features in the map. By applying the scheme provided by the embodiment of the invention, the accuracy of the determined semantic feature position in the road image can be improved.
Description
Technical Field
The invention relates to the technical field of intelligent driving, in particular to a position determining method and device based on data association.
Background
In the technical field of intelligent driving, positioning of vehicles is an important link in intelligent driving. Generally, the vehicle pose can be determined from a satellite positioning system while the vehicle is traveling. However, when the vehicle travels into a scene where the satellite signal is weak or no signal, in order to accurately determine the positioning pose of the vehicle, positioning may be performed based on visual positioning.
The vision-based localization is based on a pre-constructed high-precision map. In the map construction scheme of the high-precision map, detection can be usually performed according to a road image acquired by a vehicle in the driving process to obtain semantic features in the road image, and the high-precision map is constructed based on the positions of the semantic features in the map. But instead. Because the data of a vehicle during a trip is too simple, the semantic feature locations determined based on such data may also be less accurate.
Disclosure of Invention
The invention provides a position determining method and device based on data association, which are used for improving the accuracy of the determined semantic feature position in a road image. The specific technical scheme is as follows.
In a first aspect, an embodiment of the present invention discloses a method for determining a location based on data association, including:
acquiring a plurality of groups of road images acquired by camera equipment when a vehicle runs for a plurality of times in the same position area;
carrying out feature detection on the road signs in each road image to obtain semantic features in each road image;
aiming at each group of road images, matching semantic features among the road images in the group of road images to obtain matched semantic features belonging to the same road sign in the group of road images;
aiming at each matched semantic feature in each group of road images, performing three-dimensional reconstruction and coordinate system conversion on the matched semantic feature to obtain a first position of the matched semantic feature in a map;
performing data association on the matching semantic features among the road images according to the matching semantic features in each group of road images to obtain associated semantic features belonging to the same road sign among the road images;
and fusing the first positions of the associated semantic features in the road images to obtain a second position of the associated semantic features in the map.
Optionally, the step of performing data association on the matching semantic features between the road images according to the matching semantic features in each group of road images to obtain associated semantic features belonging to the same road sign between the road images includes:
when the proximity degree between the first positions of the matched semantic features in each group of road images meets a preset distance condition, determining the matched semantic features in each group of road images as associated semantic features belonging to the same road sign; alternatively, the first and second electrodes may be,
acquiring first attribute information of the matched semantic features in each group of road images, and determining the matched semantic features in each group of road images as associated semantic features belonging to the same road sign when the matching degree among the first attribute information meets a preset similar condition.
Optionally, the step of performing feature detection on the road sign in each road image to obtain the semantic features in each road image includes:
carrying out feature detection on the road signs in each road image to obtain each semantic area;
determining semantic models corresponding to the semantic regions from all pre-established semantic models;
representing the semantic region in each road image by adopting a corresponding semantic model to obtain each semantic feature in each road image;
wherein each semantic model comprises: a straight line model, a corner model and a spline curve model.
Optionally, the step of performing three-dimensional reconstruction and coordinate system conversion on each matching semantic feature in each group of road images to obtain a first position of the matching semantic feature in the map includes:
performing three-dimensional reconstruction on each matched semantic feature in each group of road images to obtain the position of the matched semantic feature in a camera coordinate system; converting the position of the matched semantic features in the camera coordinate system into a world coordinate system according to the conversion relation between the camera coordinate system and the world coordinate system to obtain a first position; and the world coordinate system is a coordinate system where the map is located.
Optionally, after obtaining the second position of the associated semantic feature in the map, the method further includes:
determining observation distribution data of the associated semantic features in each group of road images; wherein the observed distribution data comprises: the occurrence frequency of the associated semantic features in each group of road images and/or the distribution uniformity degree of the associated semantic features in different groups of road images;
and determining the confidence coefficient of the second position according to the observation distribution data and the preset corresponding relation between the observation distribution data and the confidence coefficient.
Optionally, the step of fusing the first positions of the associated semantic features in the road images to obtain the second position of the associated semantic features in the map includes:
and determining the mean value of each first position of the associated semantic features in each group of road images, and determining the second position of the associated semantic features in the map according to the mean value.
In a second aspect, an embodiment of the present invention provides a position determining apparatus based on data association, including:
the system comprises an image acquisition module, a processing module and a display module, wherein the image acquisition module is configured to acquire a plurality of groups of road images acquired by a camera device when a vehicle runs for a plurality of times in the same position area;
the feature detection module is configured to perform feature detection on the road signs in each road image to obtain semantic features in each road image;
the intra-group matching module is configured to match semantic features among the road images in each group of road images to obtain matched semantic features belonging to the same road sign in each group of road images;
the position determining module is configured to perform three-dimensional reconstruction and coordinate system conversion on each matched semantic feature in each group of road images to obtain a first position of the matched semantic feature in a map;
the data association module is configured to perform data association on the matching semantic features among the road images according to the matching semantic features in each group of road images to obtain associated semantic features belonging to the same road sign among the road images;
and the position fusion module is configured to fuse the first positions of the associated semantic features in the road images to obtain a second position of the associated semantic features in the map.
Optionally, the data association module is specifically configured to:
when the proximity degree between the first positions of the matched semantic features in each group of road images meets a preset distance condition, determining the matched semantic features in each group of road images as associated semantic features belonging to the same road sign; alternatively, the first and second electrodes may be,
acquiring first attribute information of the matched semantic features in each group of road images, and determining the matched semantic features in each group of road images as associated semantic features belonging to the same road sign when the matching degree among the first attribute information meets a preset similar condition.
Optionally, the feature detection module is specifically configured to:
carrying out feature detection on the road signs in each road image to obtain each semantic area;
determining semantic models corresponding to the semantic regions from all pre-established semantic models;
representing the semantic region in each road image by adopting a corresponding semantic model to obtain each semantic feature in each road image;
wherein each semantic model comprises: a straight line model, a corner model and a spline curve model.
Optionally, the position determining module is specifically configured to:
performing three-dimensional reconstruction on each matched semantic feature in each group of road images to obtain the position of the matched semantic feature in a camera coordinate system; converting the position of the matched semantic features in the camera coordinate system into a world coordinate system according to the conversion relation between the camera coordinate system and the world coordinate system to obtain a first position; and the world coordinate system is a coordinate system where the map is located.
Optionally, the apparatus further comprises a confidence determination module configured to:
after a second position of the associated semantic features in the map is obtained, determining observation distribution data of the associated semantic features in each group of road images; wherein the observed distribution data comprises: the occurrence frequency of the associated semantic features in each group of road images and/or the distribution uniformity degree of the associated semantic features in different groups of road images;
and determining the confidence coefficient of the second position according to the observation distribution data and the preset corresponding relation between the observation distribution data and the confidence coefficient.
Optionally, the location fusion module is specifically configured to:
and determining the mean value of each first position of the associated semantic features in each group of road images, and determining the second position of the associated semantic features in the map according to the mean value.
As can be seen from the above, the method and the device for determining a location based on data association according to the embodiments of the present invention may match semantic features of each internal road image group for a plurality of road images when a vehicle travels in the same location area for a plurality of times, determine locations of the matched semantic features in a map, perform semantic feature association for each group, and obtain a second location of the associated semantic features by fusing locations of the associated semantic features belonging to the same road sign among the groups. According to the embodiment of the invention, based on a plurality of groups of road images collected aiming at the same position area, the fusion position of the semantic features in the map is determined through data correlation among the semantic features, so that compared with the semantic feature position determined only by depending on the road image collected in the one-time driving process, the accuracy of the semantic feature position in the determined road image can be improved. Of course, not all of the advantages described above need to be achieved at the same time in the practice of any one product or method of the invention.
The innovation points of the embodiment of the invention comprise:
1. by means of multiple times of driving of the vehicle, multiple groups of road images are collected aiming at the same position area, the fusion position of the semantic features in the map is determined through data association among the groups, and data association and position determination can be carried out on the basis of a large amount of data through off-line data processing, so that the accuracy of the determined semantic feature position in the road image can be improved.
2. When data association is performed among the groups, the data association can be performed according to the proximity degree between the first positions of the matched semantic features in the road images of the groups or according to the attribute information of the matched semantic features, so that the accuracy of the data association is improved.
3. The semantic features are represented by a more simplified semantic model which is established in advance, so that the data volume of the semantic features can be reduced, the efficiency of a data association process is improved, and the method is suitable for large-scale application.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below. It is to be understood that the drawings in the following description are merely exemplary of some embodiments of the invention. For a person skilled in the art, without inventive effort, further figures can be obtained from these figures.
Fig. 1 is a schematic flowchart of a location determining method based on data association according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of the road images according to the embodiment of the present invention;
FIG. 3 is a schematic diagram of a principle of measuring depth information of image points based on a triangulation method;
FIG. 4 is a schematic view of an interrelationship between the vehicle body, camera and ground;
fig. 5 is a schematic structural diagram of a position determining apparatus based on data association according to an embodiment of the present invention.
Detailed Description
The technical solution in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. It is to be understood that the described embodiments are merely a few embodiments of the invention, and not all embodiments. All other embodiments, which can be obtained by a person skilled in the art without inventive effort based on the embodiments of the present invention, are within the scope of the present invention.
It is to be noted that the terms "comprises" and "comprising" and any variations thereof in the embodiments and drawings of the present invention are intended to cover non-exclusive inclusions. For example, a process, method, system, article, or apparatus that comprises a list of steps or elements is not limited to only those steps or elements listed, but may alternatively include other steps or elements not listed, or inherent to such process, method, article, or apparatus.
The embodiment of the invention discloses a position determining method and device based on data association, which can improve the accuracy of the determined semantic feature position in a road image. The following provides a detailed description of embodiments of the invention.
Fig. 1 is a schematic flowchart of a location determining method based on data association according to an embodiment of the present invention. The method is applied to the electronic equipment. The electronic device may be a general Computer, a server, an intelligent terminal device, or the like, or may be a vehicle-mounted Computer or a vehicle-mounted terminal such as an Industrial Personal Computer (IPC). In this embodiment, the vehicle-mounted terminal may be installed in a vehicle, and the vehicle refers to an intelligent vehicle. The method specifically comprises the following steps.
S110: and acquiring a plurality of groups of road images acquired by the camera equipment when the vehicle runs for a plurality of times in the same position area.
The location area may be understood as a section of a road area, or an area composed of several roads, etc. When a vehicle provided with a camera device and sensors such as an Inertial Measurement Unit (IMU), a wheel speed meter, or a Global Positioning System (GPS) travels in the same position area for a plurality of times, the camera device can acquire a road image. The plurality of road images captured by the camera device constitute a set of road images each time the vehicle travels from one end of the location area to the other. Each set of road images includes a plurality of road images, and the plurality of road images in each set of road images may be consecutive image frames.
Referring to fig. 2, when the vehicle travels the same road N times, a first group road image, a second group road image … …, an nth group road image, each group including a plurality of road images, may be obtained.
When the vehicle runs for many times in the same position area, the lanes where the vehicle runs for every time can be different, so that the shooting angles among each group of road images are different, and the diversity of data is improved.
The camera device may capture road images at a preset frequency, which may include image data of road signs or any other objects within the image capture range of the camera device. In this embodiment, the location of the road image may be outdoors or may be a parking lot.
When the camera device collects each road image, positioning can be performed according to data collected by a GPS and/or an IMU arranged in the vehicle, and a positioning pose when the road image is collected is determined. The positioning pose can be a positioning pose of a GPS or an IMU, and can also be a positioning pose of a vehicle.
According to the corresponding positioning pose of each road image, the road images belonging to the same position area can be determined from a large number of road images.
S120: and carrying out feature detection on the road signs in each road image to obtain semantic features in each road image.
The road signs may include lane lines, light poles, traffic signs, road edge lines, stop lines, ground marks, traffic lights, and the like.
The feature detection is performed on the road signs in each road image, which can be understood as performing the feature detection on the road signs in each road image in each group of road images, that is, performing the feature detection on all the road images in all the groups.
The semantic features in each road image may be one or more. For example, a road image may include semantic features of a traffic sign and semantic features of a lane line.
The road image may include road signs on the ground and road signs above the ground. When the camera device collects the road image, the image collection range includes a partial space area around the vehicle.
S130: and aiming at each group of road images, matching semantic features among the road images in the group of road images to obtain matched semantic features belonging to the same road sign in the group of road images.
In this step, inter-frame matching is performed on semantic features in each group of road images, so that matching semantic features belonging to the same road sign in the group of road images can be determined. Wherein each road image in each set of road images may be an acquired continuous image frame, and the matching may be performed between adjacent frames. All the road image groups perform the operation of this step. The determined matching semantic features in the set of road images may be one or more.
For example, after matching of semantic features, it is determined that the traffic sign 1 and the lane line 1 exist in the frames 1 to 30 in the road image group 1, and the image positions of the traffic sign 1 in the frames 1 to 30 may be different, and the image positions of the lane line 1 in the frames 1 to 30 may be different. Both the traffic sign 1 and the lane line 1 may be determined as matching semantic features.
Aiming at each group of road images, matching can be carried out according to the image positions of the semantic features among the road images in the group, and the semantic features with the difference between the image positions smaller than a threshold value are determined as matching semantic features belonging to the same road sign; or matching according to attribute information of semantic features among the road images in the group, and determining the semantic features with the similarity of the attribute information larger than a threshold value as matching semantic features. The attribute information may be a feature determined from the image pixels.
S140: and aiming at each matched semantic feature in each group of road images, performing three-dimensional reconstruction and coordinate system conversion on the matched semantic feature to obtain a first position of the matched semantic feature in the map.
The step may specifically include: performing three-dimensional reconstruction on each matched semantic feature in each group of road images to obtain the position of the matched semantic feature in a camera coordinate system; and converting the position of the matched semantic features in the camera coordinate system into the world coordinate system according to the conversion relation between the camera coordinate system and the world coordinate system to obtain a first position. All the road image groups perform the operation of this step.
The world coordinate system is a coordinate system where the map is located, and the world coordinate system is a three-dimensional coordinate system. The first location is a location expressed in a world coordinate system.
When the matching semantic features are semantic features of road signs above the ground, such as traffic signs, traffic lights, light poles and the like, and when the matching semantic features are subjected to three-dimensional reconstruction, road images corresponding to the matching semantic features can be obtained specifically, and the positions of the matching semantic features in a camera coordinate system are determined based on a triangulation method according to continuous road image frames. For example, if the matching semantic features exist in the frames 1-30 in the road image group 1, at least two frames of images can be obtained from the frames 1-30, and depth information of points in the matching semantic features is determined based on a triangulation method.
Fig. 3 is a schematic diagram illustrating a principle of measuring depth information of an image point based on a triangulation method. Viewing the same three-dimensional point P from different angles, the point P being in two road images I1And I2Respectively is p1And p2。O1And O2Respectively the positions of the origin of the camera coordinate system in different positioning poses if the position of the origin is known from O2To O1In the process, the pose change parameters of the camera coordinate system comprise a rotation matrix R and a translation matrix t, and a setting point P is arranged at O1And O2Depth information in the coordinate system is s1And s2I.e. point O1And O2The distances to the point P are respectively s1And s2The above parameters satisfy the following relationship:
S1*p1＝S2*R*p2+t
wherein, is the multiplication number,
is p1Is highly symmetrical. From the above formula, s can be obtained1And s2From the depth information, the three-dimensional coordinates of the point P in the camera coordinate system can be obtained.
In this embodiment, when the position of the matching semantic feature in the camera coordinate system is converted into the world coordinate system, the method may further include: and acquiring a positioning pose corresponding to the road image, determining a conversion relation between a camera coordinate system and a world coordinate system according to the positioning pose, and converting the position of the matched semantic features in the camera coordinate system into the world coordinate system according to the conversion relation to obtain a first position. The camera coordinate system is the coordinate system in which the camera device is located.
When the matching semantic features are semantic features of road signs on the ground, for example, for road signs such as a lane line and a road edge line, three-dimensional reconstruction is performed on the matching semantic features, depth information of the matching semantic features in a camera coordinate system can be determined according to the following projection principle, and then a third position of the matching semantic features in a world coordinate system is determined according to the depth information.
Referring to fig. 4, a schematic diagram of the relationship between the vehicle body, the camera and the ground is shown. Assuming that the vehicle body and the ground are rigid bodies and the ground near the vehicle body is a plane, three-dimensional information of ground points can be determined by calibrating a rotation matrix Rcv between a camera coordinate system and a vehicle body coordinate system and the height H from a camera to the ground. Suppose there is no rotation between the camera coordinate system and the vehicle body coordinate system, i.e., Rcv is an identity matrix, and then the X-axis of the camera coordinate system is out of the plane of the paper, the Y-axis is facing down vertically, and the Z-axis is facing forward parallel to the ground. The point on the camera imaging plane of the three-dimensional point P on the ground is P. From the above information, known quantities include: y (P coordinates of point Y), f (camera focal length), H (height of camera to ground), then the depth d of point P in the camera coordinate system can be calculated by the following formula:
s150: and performing data association on the matching semantic features among the road images according to the matching semantic features in each group of road images to obtain associated semantic features belonging to the same road sign among the road images.
The category of the associated semantic features belonging to the same road sign between the sets of road images may be one or more. For example, semantic features of the traffic sign 1, the traffic sign 2, the lane line 1, the lane line 2, and the lane line 3 exist in all of 3 sets of road images for the same road segment, and then the traffic sign 1, the traffic sign 2, the lane line 1, the lane line 2, and the lane line 3 may all be determined as associated semantic features, that is, the associated semantic features in this example include 5 and are represented by different id (identification) numbers.
S160: and fusing the first positions of the associated semantic features in the road images to obtain a second position of the associated semantic features in the map.
Aiming at each associated semantic feature, a first position exists in each road image group, and the more accurate position of the associated semantic feature can be obtained by fusing each first position.
In this step, when fusing each first position, the method may specifically include: and determining the mean value of each first position of the associated semantic features in each group of road images, and determining the second position of the associated semantic features in the map according to the mean value. When the second position of the associated semantic feature in the map is determined according to the average value, the average value may be directly determined as the second position of the associated semantic feature in the map, or a result of performing preset processing on the average value may be determined as the second position of the associated semantic feature in the map.
When determining the average value of each first position of the associated semantic features in each group of road images, the averaging operation may be directly performed, or the weighted average may be performed on each first position according to the reliability of the first position of each group of road images. The credibility of the first position of each road image group can be determined according to parameters such as semantic feature attributes, quantity and the like in the road image group.
As can be seen from the above, in this embodiment, for a plurality of groups of road images when a vehicle travels in the same position area for a plurality of times, semantic features of each group of internal road images are matched, positions of the matched semantic features in a map are determined, semantic feature association is performed for each group, and a second position of an associated semantic feature is obtained by fusing positions of the associated semantic features belonging to the same road sign among the groups. That is, in the present embodiment, based on a plurality of sets of road images acquired for the same location area, the fusion location of the semantic features in the map is determined by data association between the sets of semantic features, so that the accuracy of the semantic feature location in the determined road image can be improved compared to the semantic feature location determined by relying only on the road image acquired during one-time driving.
In this embodiment, the camera device in the vehicle may be a monocular camera, which may be a global shutter (global shutter) type camera or a cheaper rolling shutter (rolling shutter) type camera. For the positioning sensor, different levels of positioning data can be applied to the embodiment, for example, the positioning data can be high-precision positioning data (for example, positioning according to Real-time kinematic (RTK) Real-time dynamic carrier phase difference technology) or low-precision data (for example, positioning according to single-point GPS data).
In another embodiment of the present invention, based on the embodiment shown in fig. 1, in step S150, the step of performing data association on the matching semantic features between the road images according to the matching semantic features in each road image group to obtain associated semantic features belonging to the same road sign between the road images in each road image group may specifically include the following implementation manners.
In the first embodiment, when the proximity between the first positions of the matching semantic features in each group of road images meets the preset distance condition, the matching semantic features in each group of road images are determined as the associated semantic features belonging to the same road sign. The embodiment can be adopted in a high-precision track, namely when the positioning pose corresponding to each road image is determined according to GPS data.
And in the second implementation mode, first attribute information of the matched semantic features in each group of road images is obtained, and when the matching degree of the first attribute information meets the preset similar condition, the matched semantic features in each group of road images are determined as the associated semantic features belonging to the same road sign.
For example, when the matching semantic feature is a semantic feature of a traffic sign, the first attribute information thereof may include text information of the traffic sign and the like. When the matching semantic feature is a semantic feature of a lane line, the first attribute information thereof may include an imaginary attribute of the lane line, a distance between the lane line and a road edge, and the like.
In step S120, the text information of the traffic sign, the virtual and real attributes of the lane line, the distance between the lane line and the road edge, and the like may be detected when the feature of the road image is detected.
In summary, in the embodiment, when performing data association between groups, data association may be performed according to the proximity between the first positions of the matched semantic features in the road images of each group, or according to the attribute information of the matched semantic features, so as to improve the accuracy of data association.
In another embodiment of the present invention, based on the embodiment shown in fig. 1, in step S120, the step of performing feature detection on the road sign in each road image to obtain the semantic feature in each road image may include:
performing feature detection on the road signs in each road image to obtain each semantic area, and determining semantic models corresponding to each semantic area from each pre-established semantic model; and representing the semantic region in each road image by adopting a corresponding semantic model to obtain each semantic feature in each road image.
Wherein each semantic model comprises: a straight line model, a corner model and a spline curve model. The straight line model may comprise a model in two point representation, for example a light pole may be in two point representation. The corner models may include models that employ planar representations, for example, a rectangular traffic sign may employ four corner representations. The spline curve may include a model represented by a curve equation, for example, a lane line may be represented by a curve equation.
When the semantic model corresponding to each semantic region is determined from the pre-established semantic models, the semantic model corresponding to each semantic region can be determined according to the type of the marker corresponding to the semantic region and the corresponding relationship between the type of the marker and each semantic model. The marker types can comprise a traffic sign, lane lines, a light pole and the like, the traffic sign corresponds to the corner point model, the lane lines correspond to the spline curve model, and the light pole corresponds to the straight line model.
In summary, in the embodiment, the pre-established simplified semantic model is used to represent the semantic features, so that the data volume of the semantic features can be reduced, the efficiency of the data association process is improved, and the method is suitable for large-scale application.
In another embodiment of the present invention, based on the embodiment shown in fig. 1, after step S160, that is, after obtaining the second position of the associated semantic feature in the map, the method may further include:
and determining observation distribution data of the associated semantic features in each group of road images, and determining the confidence of the second position according to the observation distribution data and the preset corresponding relation between the observation distribution data and the confidence.
Wherein observing the distribution data comprises: the occurrence frequency of the associated semantic features in each group of road images and/or the distribution uniformity degree of the associated semantic features in different groups of road images. Specifically, the observation of the distribution data includes: the occurrence times and values of the associated semantic features in each group of road images, and/or the distribution uniformity of the associated semantic features in different groups of road images. When the occurrence frequency and value of the associated semantic features in each group of road images are larger, the distribution uniformity degree is higher, and the observation distribution data is larger.
The degree of uniformity of distribution of the associated semantic features in different groups of road images can be determined by adopting the following method: and determining the occurrence times of the associated semantic features in each group of road image frames aiming at each group of road image frames acquired when the vehicle runs in different lanes, and determining the distribution uniformity degree according to each occurrence time.
When the distribution uniformity degree is determined according to each occurrence frequency, when the difference value between the occurrence frequencies is smaller than a threshold value, namely the occurrence frequency distribution is balanced, determining the high distribution uniformity degree; and when the difference value between the occurrence times is not less than the threshold value, namely the occurrence times are not distributed uniformly, determining the low distribution uniformity degree.
For example, when the same road sign can be observed when the vehicles run on different lanes, the larger the observation distribution data of the road sign is; when the vehicle travels on a different lane, the road sign may be observed, and if the vehicle cannot observe the road sign, the observation distribution data of the road sign is small, and the information of the road sign can be deleted from the map.
When the observation distribution data is the occurrence frequency of the associated semantic features in each group of road images, the preset corresponding relationship can be set according to the following rule: more occurrences correspond to high confidence and less occurrences correspond to low confidence.
When the observation distribution data is the distribution uniformity degree of the associated semantic features in different groups of road images, the preset corresponding relation can be set according to the following rule: the distribution uniformity degree is high and corresponds to high confidence, and the distribution uniformity degree is low and corresponds to low confidence.
The preset correspondence may also be set in a combination of the above two ways.
In summary, in the embodiment, when the observation distribution data of the associated semantic features is insufficient, the validity of the related road image is insufficient, and finally the position reliability of the determined world coordinate system of the associated semantic features is not high enough. In this embodiment, the confidence of the second position can be effectively determined by associating the observation distribution data with the confidence of the world coordinate system position.
Fig. 5 is a schematic structural diagram of a location device based on data association according to an embodiment of the present invention. The embodiment of the device is applied to electronic equipment. This embodiment of the apparatus corresponds to the embodiment of the method shown in fig. 1. The device includes:
an image acquisition module 510 configured to acquire a plurality of sets of road images acquired by a camera device while a vehicle travels a plurality of times in the same location area;
a feature detection module 520 configured to perform feature detection on the road signs in each road image to obtain semantic features in each road image;
an intra-group matching module 530 configured to match semantic features between road images in each group of road images for each group of road images to obtain matching semantic features belonging to the same road sign in the group of road images;
the position determining module 540 is configured to perform three-dimensional reconstruction and coordinate system conversion on each matching semantic feature in each group of road images to obtain a first position of the matching semantic feature in the map;
the data association module 550 is configured to perform data association on the matching semantic features between the road images according to the matching semantic features in each group of road images to obtain associated semantic features belonging to the same road sign between the road images;
and a position fusion module 560 configured to fuse the first positions of the associated semantic features in the road images to obtain a second position of the associated semantic features in the map.
In another embodiment of the present invention, based on the embodiment shown in fig. 5, the data association module 550 is specifically configured to:
when the proximity degree between the first positions of the matched semantic features in each group of road images meets a preset distance condition, determining the matched semantic features in each group of road images as associated semantic features belonging to the same road sign; alternatively, the first and second electrodes may be,
acquiring first attribute information of the matched semantic features in each group of road images, and determining the matched semantic features in each group of road images as associated semantic features belonging to the same road sign when the matching degree among the first attribute information meets a preset similar condition.
In another embodiment of the present invention, based on the embodiment shown in fig. 5, the feature detection module 520 is specifically configured to:
carrying out feature detection on the road signs in each road image to obtain each semantic area;
determining semantic models corresponding to the semantic regions from all pre-established semantic models;
representing the semantic region in each road image by adopting a corresponding semantic model to obtain each semantic feature in each road image;
wherein each semantic model comprises: a straight line model, a corner model and a spline curve model.
In another embodiment of the present invention, based on the embodiment shown in fig. 5, the position determining module 540 is specifically configured to:
performing three-dimensional reconstruction on each matched semantic feature in each group of road images to obtain the position of the matched semantic feature in a camera coordinate system; converting the position of the matched semantic features in the camera coordinate system into a world coordinate system according to the conversion relation between the camera coordinate system and the world coordinate system to obtain a first position; wherein, the world coordinate system is the coordinate system of the map.
In another embodiment of the present invention, based on the embodiment shown in fig. 5, the apparatus further includes a confidence determination module (not shown in the figure) configured to:
after a second position of the associated semantic features in the map is obtained, observation distribution data of the associated semantic features in each group of road images are determined; wherein observing the distribution data comprises: the occurrence frequency of the associated semantic features in each group of road images and/or the distribution uniformity degree of the associated semantic features in different groups of road images;
and determining the confidence of the second position according to the observation distribution data and the preset corresponding relation between the observation distribution data and the confidence.
In another embodiment of the present invention, based on the embodiment shown in fig. 5, the location fusion module 560 is specifically configured to:
and determining the mean value of each first position of the associated semantic features in each group of road images, and determining the second position of the associated semantic features in the map according to the mean value.
The above device embodiment corresponds to the method embodiment, and has the same technical effect as the method embodiment, and for the specific description, refer to the method embodiment. The device embodiment is obtained based on the method embodiment, and for specific description, reference may be made to the method embodiment section, which is not described herein again.
Those of ordinary skill in the art will understand that: the figures are merely schematic representations of one embodiment, and the blocks or flow diagrams in the figures are not necessarily required to practice the present invention.
Those of ordinary skill in the art will understand that: modules in the devices in the embodiments may be distributed in the devices in the embodiments according to the description of the embodiments, or may be located in one or more devices different from the embodiments with corresponding changes. The modules of the above embodiments may be combined into one module, or further split into multiple sub-modules.
Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (10)
1. A method for determining a location based on data association, comprising:
acquiring a plurality of groups of road images acquired by camera equipment when a vehicle runs for a plurality of times in the same position area;
carrying out feature detection on the road signs in each road image to obtain semantic features in each road image;
aiming at each group of road images, matching semantic features among the road images in the group of road images to obtain matched semantic features belonging to the same road sign in the group of road images;
aiming at each matched semantic feature in each group of road images, performing three-dimensional reconstruction and coordinate system conversion on the matched semantic feature to obtain a first position of the matched semantic feature in a map;
performing data association on the matching semantic features among the road images according to the matching semantic features in each group of road images to obtain associated semantic features belonging to the same road sign among the road images;
and fusing the first positions of the associated semantic features in the road images to obtain a second position of the associated semantic features in the map.
2. The method according to claim 1, wherein the step of performing data association on the matching semantic features between the road images according to the matching semantic features in each road image group to obtain associated semantic features belonging to the same road sign between the road images of each road group comprises:
when the proximity degree between the first positions of the matched semantic features in each group of road images meets a preset distance condition, determining the matched semantic features in each group of road images as associated semantic features belonging to the same road sign;
alternatively, the first and second electrodes may be,
acquiring first attribute information of the matched semantic features in each group of road images, and determining the matched semantic features in each group of road images as associated semantic features belonging to the same road sign when the matching degree among the first attribute information meets a preset similar condition.
3. The method of claim 1, wherein the step of performing feature detection on the road signs in each road image to obtain semantic features in each road image comprises:
carrying out feature detection on the road signs in each road image to obtain each semantic area;
determining semantic models corresponding to the semantic regions from all pre-established semantic models;
representing the semantic region in each road image by adopting a corresponding semantic model to obtain each semantic feature in each road image;
wherein each semantic model comprises: a straight line model, a corner model and a spline curve model.
4. The method of claim 1, wherein the step of performing three-dimensional reconstruction and coordinate system conversion on each matching semantic feature in each set of road images to obtain a first position of the matching semantic feature in the map comprises:
performing three-dimensional reconstruction on each matched semantic feature in each group of road images to obtain the position of the matched semantic feature in a camera coordinate system; converting the position of the matched semantic features in the camera coordinate system into a world coordinate system according to the conversion relation between the camera coordinate system and the world coordinate system to obtain a first position; and the world coordinate system is a coordinate system where the map is located.
5. The method of any of claims 1-4, after obtaining the second location of the associated semantic feature in the map, further comprising:
determining observation distribution data of the associated semantic features in each group of road images; wherein the observed distribution data comprises: the occurrence frequency of the associated semantic features in each group of road images and/or the distribution uniformity degree of the associated semantic features in different groups of road images;
and determining the confidence coefficient of the second position according to the observation distribution data and the preset corresponding relation between the observation distribution data and the confidence coefficient.
6. The method according to any one of claims 1 to 5, wherein the step of fusing the first positions of the associated semantic features in the road images to obtain the second positions of the associated semantic features in the map comprises:
and determining the mean value of each first position of the associated semantic features in each group of road images, and determining the second position of the associated semantic features in the map according to the mean value.
7. A position determining apparatus based on data association, comprising:
the system comprises an image acquisition module, a processing module and a display module, wherein the image acquisition module is configured to acquire a plurality of groups of road images acquired by a camera device when a vehicle runs for a plurality of times in the same position area;
the feature detection module is configured to perform feature detection on the road signs in each road image to obtain semantic features in each road image;
the intra-group matching module is configured to match semantic features among the road images in each group of road images to obtain matched semantic features belonging to the same road sign in each group of road images;
the position determining module is configured to perform three-dimensional reconstruction and coordinate system conversion on each matched semantic feature in each group of road images to obtain a first position of the matched semantic feature in a map;
the data association module is configured to perform data association on the matching semantic features among the road images according to the matching semantic features in each group of road images to obtain associated semantic features belonging to the same road sign among the road images;
and the position fusion module is configured to fuse the first positions of the associated semantic features in the road images to obtain a second position of the associated semantic features in the map.
8. The apparatus of claim 7, wherein the data association module is specifically configured to:
when the proximity degree between the first positions of the matched semantic features in each group of road images meets a preset distance condition, determining the matched semantic features in each group of road images as associated semantic features belonging to the same road sign;
alternatively, the first and second electrodes may be,
acquiring first attribute information of the matched semantic features in each group of road images, and determining the matched semantic features in each group of road images as associated semantic features belonging to the same road sign when the matching degree among the first attribute information meets a preset similar condition.
9. The apparatus of claim 7, wherein the feature detection module is specifically configured to:
carrying out feature detection on the road signs in each road image to obtain each semantic area;
determining semantic models corresponding to the semantic regions from all pre-established semantic models;
representing the semantic region in each road image by adopting a corresponding semantic model to obtain each semantic feature in each road image;
wherein each semantic model comprises: a straight line model, a corner model and a spline curve model.
10. The apparatus of claim 7, wherein the position determination module is specifically configured to:
performing three-dimensional reconstruction on each matched semantic feature in each group of road images to obtain the position of the matched semantic feature in a camera coordinate system; converting the position of the matched semantic features in the camera coordinate system into a world coordinate system according to the conversion relation between the camera coordinate system and the world coordinate system to obtain a first position; and the world coordinate system is a coordinate system where the map is located.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910800415.XA CN112446234B (en) | 2019-08-28 | Position determining method and device based on data association |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910800415.XA CN112446234B (en) | 2019-08-28 | Position determining method and device based on data association |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112446234A true CN112446234A (en) | 2021-03-05 |
| CN112446234B CN112446234B (en) | 2024-07-19 |
Family
ID=
Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106919915A (en) * | 2017-02-22 | 2017-07-04 | 武汉极目智能技术有限公司 | Map road mark and road quality harvester and method based on ADAS systems |
| US20180045519A1 (en) * | 2016-08-09 | 2018-02-15 | Nauto, Inc. | System and method for precision localization and mapping |
| CN107742311A (en) * | 2017-09-29 | 2018-02-27 | 北京易达图灵科技有限公司 | A kind of method and device of vision positioning |
| CN107833250A (en) * | 2017-10-24 | 2018-03-23 | 北京易达图灵科技有限公司 | Semantic space map constructing method and device |
| US20180192035A1 (en) * | 2017-01-04 | 2018-07-05 | Qualcomm Incorporated | Systems and methods for object location |
| CN108833784A (en) * | 2018-06-26 | 2018-11-16 | Oppo（重庆）智能科技有限公司 | A kind of adaptive patterning process, mobile terminal and computer readable storage medium |
| WO2018227980A1 (en) * | 2017-06-13 | 2018-12-20 | 蔚来汽车有限公司 | Camera sensor based lane line map construction method and construction system |
| US10203210B1 (en) * | 2017-11-03 | 2019-02-12 | Toyota Research Institute, Inc. | Systems and methods for road scene change detection using semantic segmentation |
| CN109376631A (en) * | 2018-10-12 | 2019-02-22 | 中国人民公安大学 | A kind of winding detection method and device neural network based |
| CN110136058A (en) * | 2018-10-25 | 2019-08-16 | 北京初速度科技有限公司 | It is a kind of that drawing method and car-mounted terminal are built based on vertical view spliced map |
| CN110136199A (en) * | 2018-11-13 | 2019-08-16 | 北京初速度科技有限公司 | A kind of vehicle location based on camera, the method and apparatus for building figure |
| CN110146096A (en) * | 2018-10-24 | 2019-08-20 | 北京初速度科技有限公司 | A kind of vehicle positioning method and device based on image perception |
Patent Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20180045519A1 (en) * | 2016-08-09 | 2018-02-15 | Nauto, Inc. | System and method for precision localization and mapping |
| US20180192035A1 (en) * | 2017-01-04 | 2018-07-05 | Qualcomm Incorporated | Systems and methods for object location |
| CN106919915A (en) * | 2017-02-22 | 2017-07-04 | 武汉极目智能技术有限公司 | Map road mark and road quality harvester and method based on ADAS systems |
| WO2018227980A1 (en) * | 2017-06-13 | 2018-12-20 | 蔚来汽车有限公司 | Camera sensor based lane line map construction method and construction system |
| CN107742311A (en) * | 2017-09-29 | 2018-02-27 | 北京易达图灵科技有限公司 | A kind of method and device of vision positioning |
| CN107833250A (en) * | 2017-10-24 | 2018-03-23 | 北京易达图灵科技有限公司 | Semantic space map constructing method and device |
| US10203210B1 (en) * | 2017-11-03 | 2019-02-12 | Toyota Research Institute, Inc. | Systems and methods for road scene change detection using semantic segmentation |
| CN108833784A (en) * | 2018-06-26 | 2018-11-16 | Oppo（重庆）智能科技有限公司 | A kind of adaptive patterning process, mobile terminal and computer readable storage medium |
| CN109376631A (en) * | 2018-10-12 | 2019-02-22 | 中国人民公安大学 | A kind of winding detection method and device neural network based |
| CN110146096A (en) * | 2018-10-24 | 2019-08-20 | 北京初速度科技有限公司 | A kind of vehicle positioning method and device based on image perception |
| CN110136058A (en) * | 2018-10-25 | 2019-08-16 | 北京初速度科技有限公司 | It is a kind of that drawing method and car-mounted terminal are built based on vertical view spliced map |
| CN110136199A (en) * | 2018-11-13 | 2019-08-16 | 北京初速度科技有限公司 | A kind of vehicle location based on camera, the method and apparatus for building figure |
Non-Patent Citations (1)
| Title |
|---|
| 吕艳玲: "基于光电图像目标空间位置及轨迹确定方法的研究", 优秀硕士论文全文库信息科技, 15 February 2007 (2007-02-15), pages 1 - 61 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2021004548A1 (en) | Vehicle speed intelligent measurement method based on binocular stereo vision system | |
| US20200041284A1 (en) | Map road marking and road quality collecting apparatus and method based on adas system | |
| CN106407315B (en) | A kind of vehicle autonomic positioning method based on street view image database | |
| CN108694882B (en) | Method, device and equipment for labeling map | |
| JP6821712B2 (en) | Calibration of integrated sensor in natural scene | |
| JP4232167B1 (en) | Object identification device, object identification method, and object identification program | |
| JP2020525809A (en) | System and method for updating high resolution maps based on binocular images | |
| US10909395B2 (en) | Object detection apparatus | |
| JP4978615B2 (en) | Target identification device | |
| JP5404861B2 (en) | Stationary object map generator | |
| CN111275960A (en) | Traffic road condition analysis method, system and camera | |
| JP6950832B2 (en) | Position coordinate estimation device, position coordinate estimation method and program | |
| CN104280036A (en) | Traffic information detection and positioning method, device and electronic equipment | |
| JP2008065087A (en) | Apparatus for creating stationary object map | |
| CN112308913B (en) | Vehicle positioning method and device based on vision and vehicle-mounted terminal | |
| CN112446915B (en) | Picture construction method and device based on image group | |
| CN112749584B (en) | Vehicle positioning method based on image detection and vehicle-mounted terminal | |
| CN103453882B (en) | A kind of ceiling of clouds measuring system based on aircraft and ceiling of clouds measuring method | |
| CN113137973A (en) | Image semantic feature point truth value determining method and device | |
| WO2020113425A1 (en) | Systems and methods for constructing high-definition map | |
| CN112530270B (en) | Mapping method and device based on region allocation | |
| CN112446234B (en) | Position determining method and device based on data association | |
| Jiang et al. | Precise vehicle ego-localization using feature matching of pavement images | |
| CN112837365B (en) | Image-based vehicle positioning method and device | |
| CN112446234A (en) | Position determination method and device based on data association |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant |