CN112381876B - Traffic sign marking method and device and computer equipment - Google Patents
Traffic sign marking method and device and computer equipment Download PDFInfo
- Publication number
- CN112381876B CN112381876B CN202110060876.5A CN202110060876A CN112381876B CN 112381876 B CN112381876 B CN 112381876B CN 202110060876 A CN202110060876 A CN 202110060876A CN 112381876 B CN112381876 B CN 112381876B
- Authority
- CN
- China
- Prior art keywords
- traffic sign
- matrix
- position information
- pixel
- camera
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 45
- 239000011159 matrix material Substances 0.000 claims description 191
- 238000002372 labelling Methods 0.000 claims description 57
- 238000000354 decomposition reaction Methods 0.000 claims description 31
- 238000001514 detection method Methods 0.000 claims description 12
- 238000012549 training Methods 0.000 claims description 9
- 238000004590 computer program Methods 0.000 claims description 7
- 238000004364 calculation method Methods 0.000 claims description 6
- 238000010586 diagram Methods 0.000 description 9
- 238000004891 communication Methods 0.000 description 5
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 238000012545 processing Methods 0.000 description 3
- 238000013519 translation Methods 0.000 description 3
- 239000003086 colorant Substances 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 241000287196 Asthenes Species 0.000 description 1
- 102000002274 Matrix Metalloproteinases Human genes 0.000 description 1
- 108010000684 Matrix Metalloproteinases Proteins 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000013213 extrapolation Methods 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
- G06V20/58—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- G06V20/582—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads of traffic signs
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Multimedia (AREA)
- Traffic Control Systems (AREA)
- Image Analysis (AREA)
Abstract
The embodiment of the application relates to the technical field of unmanned driving, and provides a traffic sign marking method, a device and computer equipment, wherein the method comprises the following steps: acquiring camera pose information corresponding to each traffic sign image of a traffic sign, wherein each traffic sign image is an image of the traffic sign under different camera poses; acquiring position information of each pixel of the central point of the traffic sign in each traffic sign image; calculating the position information of the traffic sign in the three-dimensional space according to the position information of each camera and the position information of each pixel; and marking the traffic sign in the three-dimensional map according to the position information. The embodiment of the application can effectively reduce the complexity of acquiring the position information of the traffic sign and save time.
Description
Technical Field
The embodiment of the application relates to the technical field of unmanned driving, in particular to a traffic sign marking method, a traffic sign marking device and computer equipment.
Background
At present, unmanned driving is the latest development direction of the automobile industry, in an unmanned driving system, a high-precision map is one of indispensable elements, and in the process of marking a traffic sign on the high-precision map, position information of the traffic sign in a three-dimensional space needs to be acquired as one input data, so that the position information of the traffic sign in the three-dimensional space needs to be accurately determined.
Currently, a device with a depth measurement capability, such as a laser radar, is generally used to measure position information of a traffic sign in a three-dimensional space to obtain the position information.
However, the above-mentioned method for acquiring the position information has the problems of high complexity and much time consumption.
Disclosure of Invention
The embodiment of the application provides a traffic sign marking method, a traffic sign marking device and computer equipment, which can effectively reduce the complexity of obtaining the position information of a traffic sign and save time.
In a first aspect, an embodiment of the present application provides a traffic sign labeling method, including: acquiring camera pose information corresponding to each traffic sign image of a traffic sign, wherein each traffic sign image is an image of the traffic sign under different camera poses;
acquiring position information of each pixel of the central point of the traffic sign in each traffic sign image;
calculating the position information of the traffic sign in the three-dimensional space according to the position information of each camera and the position information of each pixel;
and marking the traffic sign in the three-dimensional map according to the position information.
Optionally, in some possible implementation manners of the present application, the step of calculating, according to the pose information of each camera and the position information of each pixel, to obtain position information of the traffic sign in a three-dimensional space includes:
generating a pixel matrix according to the position information of each pixel;
generating a pose matrix associated with the pixel matrix according to the pose information of each camera;
obtaining a projection matrix according to the pose matrix and the acquired camera internal parameters;
generating a target matrix according to the pixel matrix and the projection matrix;
performing singular value decomposition on the target matrix to obtain a decomposition result;
and calculating the position information of the traffic sign in the three-dimensional space according to the decomposition result.
Optionally, in some possible implementations of the present application, after the step of obtaining the position information of the traffic sign in the three-dimensional space by calculating according to the decomposition result, the method further includes:
projecting the coordinate points corresponding to the position information through the projection matrix to obtain projection pixels of the coordinate points;
constructing an adjustment equation according to the projection pixels and the pixel matrix;
and adjusting the position information according to the adjustment equation.
Optionally, in some possible implementations of the present application, the step of generating a pose matrix associated with the pixel matrix according to the pose information of each camera includes:
acquiring the row number of the pixel matrix;
generating a pose matrix associated with the pixel matrix according to the line number of the pixel matrix, and the camera position and the camera orientation respectively corresponding to the pose information of each camera;
and the number of rows of the pose matrix is the same as that of the pixel matrix.
Optionally, in some possible implementation manners of the present application, the step of obtaining a projection matrix according to the pose matrix and the acquired camera internal parameters includes:
acquiring an inverse matrix of the pose matrix;
and multiplying the inverse matrix by the acquired camera internal parameters to obtain a projection matrix.
Optionally, in some possible implementations of the present application, the target matrix is an m × n matrix, and the step of calculating the position information of the traffic sign in the three-dimensional space according to the decomposition result includes:
determining an m-order matrix, an m x n matrix and an n-order matrix in the decomposition result;
dividing each element of the last column front n-1 of the n-order matrix by the last element of the last column of the n-order matrix to obtain the position information of the traffic sign in the three-dimensional space;
wherein m and n are both positive integers not less than 2.
Optionally, in some possible implementations of the present application, the step of obtaining position information of each pixel of a central point of the traffic sign in each traffic sign image includes:
labeling the traffic signs in the traffic sign images to obtain labeling information;
and obtaining the position information of each pixel of the central point of the traffic sign in each traffic sign image according to the labeling information.
Optionally, in some possible implementations of the present application, the method further includes:
acquiring color information and circumscribed rectangle information in the labeling information;
and inputting the color information and the circumscribed rectangle information serving as a training data set to a traffic sign detection model to obtain the trained traffic sign detection model.
In a second aspect, an embodiment of the present application provides a traffic sign labeling apparatus, including:
the first acquisition unit is used for acquiring the position and orientation information of each camera corresponding to each traffic sign image of a traffic sign, wherein each traffic sign image is an image of the traffic sign under different camera positions and orientations;
a second acquisition unit configured to acquire position information of each pixel of a center point of the traffic sign in each traffic sign image;
the calculation unit is used for calculating position information of the traffic sign in a three-dimensional space according to the pose information of each camera and the position information of each pixel;
and the marking unit is used for marking the traffic sign in the three-dimensional map according to the position information.
Yet another aspect of the embodiments of the present application provides a computer apparatus, which includes a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor is configured to call the computer program in the memory to execute the method according to the first aspect.
In yet another aspect, embodiments of the present application provide a storage medium including instructions that, when executed on a computer, cause the computer to perform the method of the first aspect.
According to a first aspect of the present application, there is provided a computer program product or computer program comprising computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method provided in the first aspect and the various implementation manners of the first aspect.
Compared with the prior art, in the scheme provided by the embodiment of the application, the camera pose information corresponding to each traffic sign image of the traffic sign is firstly acquired, the pixel position information of the central point of the traffic sign in each traffic sign image is acquired, then the position information of the traffic sign in the three-dimensional space is calculated according to the camera pose information and the pixel position information, and finally the traffic sign is marked in the three-dimensional map according to the position information. The method and the device have the advantages that the position information of the traffic sign in the three-dimensional space is obtained in a mode of automatically processing the traffic sign image without an external measuring device, and obviously, the complexity of obtaining the position information of the traffic sign can be effectively reduced and the time can be saved compared with the mode that the position information of the traffic sign in the three-dimensional space is measured by a device with depth measuring capability such as a laser radar in the prior art so as to obtain the position information.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without inventive labor.
FIG. 1 is a schematic diagram illustrating a traffic sign tagging method according to an embodiment of the present disclosure;
fig. 2 is a flowchart of a traffic sign labeling method according to an embodiment of the present application;
fig. 3 is a flowchart for calculating position information of a traffic sign in a three-dimensional space according to an embodiment of the present application;
fig. 4 is a schematic structural diagram of a traffic sign labeling apparatus according to an embodiment of the present application;
fig. 5 is a schematic physical structure diagram of a computer device according to an embodiment of the present application.
Detailed Description
The terms "first," "second," and the like in the description and in the claims of the embodiments of the application and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It will be appreciated that the data so used may be interchanged under appropriate circumstances such that the embodiments described herein may be practiced otherwise than as specifically illustrated or described herein. Furthermore, the terms "comprise" and "have," and any variations thereof, are intended to cover non-exclusive inclusions, such that a process, method, system, article, or apparatus that comprises a list of steps or modules is not necessarily limited to those steps or modules expressly listed, but may include other steps or modules not expressly listed or inherent to such process, method, article, or apparatus, such that the division of modules presented in the present application is merely a logical division and may be implemented in a practical application in a different manner, such that multiple modules may be combined or integrated into another system or some features may be omitted or not implemented, and such that couplings or direct couplings or communicative connections shown or discussed may be through interfaces, indirect couplings or communicative connections between modules may be electrical or the like, the embodiments of the present application are not limited. Moreover, the modules or sub-modules described as separate components may or may not be physically separated, may or may not be physical modules, or may be distributed in a plurality of circuit modules, and some or all of the modules may be selected according to actual needs to achieve the purpose of the embodiments of the present application.
The embodiment of the application provides a traffic sign marking method which is mainly applied to scenes such as unmanned driving and high-precision map drawing and is executed by a traffic sign marking device. The traffic sign may be a traffic light, a fixed traffic sign, or the like. Referring to fig. 1, fig. 1 is a schematic diagram of a traffic sign labeling method according to an embodiment of the present disclosure, which includes acquiring, by using an acquisition device such as a camera, images including a traffic sign at different camera poses to obtain a plurality of traffic sign images, and simultaneously recording pose information of the camera corresponding to each traffic sign image in a world coordinate System during the acquisition process to obtain pose information of the plurality of cameras, where the pose information of the cameras can be obtained by using a device such as a Global Positioning System (GPS), an Inertial Navigation System (INS) (Inertial Navigation for short), and the like. Then, performing two-dimensional (2D) labeling on the traffic sign in each traffic sign image, where the labeled information includes a color of the traffic sign and a Bounding rectangle (Bounding Box), and based on the labeled information, pixel position information of a center point of the traffic sign in the traffic sign image can be obtained, for example, a pixel coordinate of the pixel position information can be obtained directly by labeling, or obtained by calculating a center of an angular point of the Bounding rectangle, so as to obtain a plurality of pixel position information. And calculating the position information of the traffic sign in a three-dimensional (3D) space based on the position information of the cameras and the position information of the pixels. And finally, taking the position information of the traffic sign in the three-dimensional space as input, and labeling the traffic sign in a three-dimensional map (high-precision map). It should be noted that the labeled information can also be used as a data set for training a traffic sign detection model, and is used for training the traffic sign detection model.
It should be noted that the schematic diagram of the traffic sign labeling method shown in fig. 1 is only an example, and the schematic diagram described in this application is for more clearly illustrating the technical solutions of the embodiments of the application, and does not constitute a limitation on the technical solutions provided by the embodiments of the application.
With reference to the foregoing schematic diagram, the following describes a traffic sign labeling method in the present application, please refer to fig. 2, where fig. 2 is a flowchart of a traffic sign labeling method according to an embodiment of the present application, and the embodiment of the present application at least includes the following steps:
201. acquiring camera pose information corresponding to each traffic sign image of a traffic sign, wherein each traffic sign image is an image of the traffic sign under different camera poses;
in this embodiment, the traffic sign labeling apparatus may acquire an image (a traffic sign image) including a traffic sign by using a camera, and may acquire an image of the traffic sign by using different cameras at different poses in order to obtain traffic sign images at different camera poses, for example, the camera a acquires an image of the traffic sign a at pose 1, the camera B acquires an image of the traffic sign a at pose 2, and the camera C acquires an image of the traffic sign a at pose 3; the images of the traffic sign can also be acquired by the same camera in different poses, for example, the camera a acquires the images of the traffic sign a in pose 1, pose 2 and pose 3, which is not limited herein.
The traffic sign includes but is not limited to a traffic signal lamp, a fixed traffic sign, etc., the traffic signal lamp is a signal lamp for directing traffic operation, the type of the traffic signal lamp is not limited, optionally, the type of the traffic signal lamp is divided according to the role of the traffic signal lamp, and the type of the traffic signal lamp may include: motor vehicle signal lamp, non-motor vehicle signal lamp, pedestrian crossing signal lamp, direction indicating lamp (arrow signal lamp), lane signal lamp, flash warning signal lamp, road and railway crossing signal lamp. Optionally, the type of the traffic signal lamp is divided according to the expression form of the traffic signal lamp, and the type of the traffic signal lamp may include: a color light and a countdown light. The color lamps can be understood as lamps which mainly represent different traffic signals through different colors, the colors of the color lamps can include red, green, yellow and the like, and it can be understood that the shapes of the color lamps are not limited, and can be square, circular or similar to human shapes.
The camera pose information can be acquired through equipment such as a GPS or inertial navigation system.
The camera may be a monocular camera, a binocular camera, or a multi-view camera, or a camera, and the present application does not particularly limit the manner in which images are acquired, as long as clear images can be obtained.
202. Acquiring position information of each pixel of the central point of the traffic sign in each traffic sign image;
in this embodiment, for each traffic sign image, the traffic sign labeling device obtains the pixel position information of the central point of the traffic sign in the traffic sign image, so as to obtain a plurality of pixel position information. It should be noted that, the traffic sign image may include the traffic sign and the surrounding environment of the traffic sign, so before the traffic sign labeling device acquires the position information of each pixel of the center point of the traffic sign in each traffic sign image, the traffic sign labeling device needs to extract the region (region of interest) including only the traffic sign from each traffic sign image, and then determine the center point of the traffic sign from the region of the traffic sign.
In some possible embodiments, the step 202 of obtaining the position information of each pixel of the central point of the traffic sign in each traffic sign image includes:
labeling the traffic signs in the traffic sign images to obtain labeling information;
and obtaining the position information of each pixel of the central point of the traffic sign in each traffic sign image according to the labeling information.
Specifically, the traffic sign labeling device performs 2D labeling on the traffic sign in the traffic sign image to obtain labeling information, and then obtains pixel position information of a center point of the traffic sign in the traffic sign image according to the labeling information. It should be noted that the pixel position of the central point of the traffic sign in the traffic sign image refers to the pixel position where the central point of the traffic sign in the three-dimensional space is projected onto the traffic sign image.
The pixel position of the center point of the traffic sign in the traffic sign image can be obtained by direct labeling, or by calculating the center of the corner point of the circumscribed rectangle, which is not limited herein. It can be understood that if the traffic sign in the traffic sign image is in a regular shape and the center point can be directly determined, the traffic sign can be obtained by adopting a direct sign mode, otherwise, the traffic sign can be obtained by calculating the center of the corner point of the circumscribed rectangle.
Further, in some possible embodiments, the method further includes:
acquiring color information and circumscribed rectangle information in the labeling information;
and inputting the color information and the circumscribed rectangle information serving as a training data set to a traffic sign detection model to obtain the trained traffic sign detection model.
Specifically, after the labeling information is obtained, the traffic sign labeling device may obtain color information (such as color of a traffic light) and circumscribed rectangle information in the labeling information, and then use the color information and the circumscribed rectangle information as a data set for training a traffic sign detection model, which may be a model based on a deep neural network. Therefore, when the labeling information is obtained, the traffic sign labeling device can also make full use of the labeling information to perform model training, so that the accuracy of a traffic sign detection model is improved, and the utilization range of the labeling information is expanded.
203. Calculating the position information of the traffic sign in the three-dimensional space according to the position information of each camera and the position information of each pixel;
in this embodiment, after obtaining the pose information of each camera and the position information of each pixel, the traffic sign labeling device may calculate the position information of the traffic sign in the three-dimensional space according to the pose information of each camera and the position information of each pixel.
In some possible embodiments, step 203 may refer to fig. 3, where a specific scheme of calculating the position information of the traffic sign in the three-dimensional space according to the pose information of each camera and the position information of each pixel is provided, and fig. 3 is a flowchart of calculating the position information of the traffic sign in the three-dimensional space according to the embodiment of the present application, and as shown in fig. 3, the method includes the following steps:
301. generating a pixel matrix according to the position information of each pixel;
specifically, after obtaining the position information of each pixel at the center point of the traffic sign in each traffic sign image, the traffic sign labeling device may generate a pixel matrix according to the position information of each pixel.
For example, each pixelThe position information is expressed as
Then the generated pixel matrix
Comprises the following steps:
it will be appreciated that each of the traffic sign images includes a center point of the traffic sign, and each of the center points of the traffic sign includes pixel position information, which can be considered as coordinate information of the center point, for example, as described above
Is shown as the abscissa of the graph,
is the ordinate. The pixel matrix
In the expression of
Refers to the number of pixel position information, pixel matrix
Is composed of
A matrix of order x 2 is formed,
is the number of rows of the pixel matrix and 2 is the number of columns of the pixel matrix.
302. Generating a pose matrix associated with the pixel matrix according to the pose information of each camera;
in particular, for each camera pose information, the traffic sign tagging apparatus may generate a pose matrix associated with the pixel matrix.
For example, in some possible embodiments, the step 302 of generating a pose matrix associated with the pixel matrix from the camera pose information comprises:
acquiring the row number of the pixel matrix;
generating a pose matrix associated with the pixel matrix according to the line number of the pixel matrix, and the camera position and the camera orientation respectively corresponding to the pose information of each camera;
and the number of rows of the pose matrix is the same as that of the pixel matrix.
Specifically, the traffic sign labeling device obtains the row number of the pixel matrix
Then, according to the number of rows of the pixel matrix
And the camera position and the camera orientation included in each camera pose information generate a pose matrix associated with the pixel matrix. For example, the pose matrix may be a matrix
：
Wherein,
can constitute three-dimensional coordinates (
) For indicating one of the camera positions,
can form four-dimensional coordinates (
) For indicating one of the camera orientations (corresponding three-dimensional coordinates: (
) Represented camera position), pose matrix
The number of lines of the position matrix is the same as that of the pixel matrix
Each row of (a) corresponds to each row of the pixel matrix.
303. Obtaining a projection matrix according to the pose matrix and the acquired camera internal parameters;
specifically, the traffic sign labeling device can obtain a projection matrix according to the pose matrix and the acquired camera internal parameters.
For example, in some possible embodiments, the step 303 of obtaining a projection matrix according to the pose matrix and the acquired camera internal parameters includes:
acquiring an inverse matrix of the pose matrix;
and multiplying the inverse matrix by the acquired camera internal parameters to obtain a projection matrix.
In particular, due to camera reference
Is known for a matrix of poses
Obtaining a projection matrix of the camera under each camera pose
Wherein
representing a pose matrix
To (1) a
The rows of the image data are, in turn,
representing the inverse of the pose matrix.
304. Generating a target matrix according to the pixel matrix and the projection matrix;
specifically, the traffic sign labeling device may generate the target matrix according to the pixel matrix and the projection matrix.
Specifically, the traffic sign labeling device generates the target matrix according to a Singular Value Decomposition (SVD) principle, and in the SVD principle, a rotation matrix and a translation matrix can be obtained by performing least square on a certain matrix. The traffic sign marking device in the scheme is a known rotation matrix and a translation matrix, and the matrix is reversely deduced. Specifically, a pixel matrix is characterized as a translation matrix, and a projection matrix is characterized as a rotation matrix, so that a target matrix is obtained based on the pixel matrix and the projection matrix through reverse extrapolation.
For example, an object matrix
Wherein
305. Performing singular value decomposition on the target matrix to obtain a decomposition result;
in particular, the traffic sign labeling means may be directed to a target matrix
Performing singular value decomposition, i.e. object matrix
Decomposing to obtain
、
And
is expressed by formula as
Wherein
Is a matrix of m x m orders (matrix of m orders),
is an m x n matrix and is,
is an n × n matrix (n matrix). It should be understood that the object matrix
Obtained by singular value decomposition
、
And
is uniquely determined.
306. And calculating the position information of the traffic sign in the three-dimensional space according to the decomposition result.
Specifically, in some possible embodiments, assuming that the target matrix is an m × n matrix, the step 306 of calculating the position information of the traffic sign in the three-dimensional space according to the decomposition result includes:
determining an m-order matrix, an m x n matrix and an n-order matrix in the decomposition result;
dividing each element of the last column front n-1 of the n-order matrix by the last element of the last column of the n-order matrix to obtain the position information of the traffic sign in the three-dimensional space;
wherein m and n are both positive integers not less than 2.
For example, the traffic sign labeling means obtains the decomposition result
The position information of the traffic sign in the three-dimensional space is calculated by the following formula
：
The
To represent
Each element of the last column of (1) front
Divided by the last element
。
Further, in some possible embodiments, after the step of calculating the position information of the traffic sign in the three-dimensional space according to the decomposition result in step 306, the method further includes:
projecting the coordinate points corresponding to the position information through the projection matrix to obtain projection pixels of the coordinate points;
constructing an adjustment equation according to the projection pixels and the pixel matrix;
and adjusting the position information according to the adjustment equation.
Specifically, after obtaining the position information of the traffic sign in the three-dimensional space, the traffic sign labeling device projects the initial value of the 3D coordinate of the position information back to each camera pose through the projection matrix to obtain a 2D projection pixel of the 3D coordinate, that is, the 2D projection pixel is expressed as:
constructing an adjustment equation:
the position information may be adjusted, typically using a bundle adjustment method, to obtain the final 3D position information of the traffic sign. The adjustment equation can be understood as an optimization equation.
204. And marking the traffic sign in the three-dimensional map according to the position information.
In this embodiment, after obtaining the position information of the traffic sign in the three-dimensional space, the traffic sign labeling device may label the traffic sign in a three-dimensional map (high-precision map) based on the position information, so that the subsequent unmanned vehicle can recognize the traffic sign based on the high-precision map during the driving process, thereby ensuring the driving safety.
To sum up, in the scheme provided in the embodiment of the present application, first, camera pose information corresponding to each traffic sign image of a traffic sign is obtained, and pixel position information of a central point of the traffic sign in each traffic sign image is obtained, then, position information of the traffic sign in a three-dimensional space is obtained by calculation according to the camera pose information and the pixel position information, and finally, the traffic sign is marked in the three-dimensional map according to the position information. The method and the device have the advantages that the position information of the traffic sign in the three-dimensional space is obtained in a mode of automatically processing the traffic sign image without an external measuring device, and obviously, the complexity of obtaining the position information of the traffic sign can be effectively reduced and the time can be saved compared with the mode that the position information of the traffic sign in the three-dimensional space is measured by a device with depth measuring capability such as a laser radar in the prior art so as to obtain the position information.
In order to better implement the above solution of the embodiment of the present application, a related device for implementing the above solution is further provided below, please refer to fig. 4, where fig. 4 is a schematic structural diagram of a traffic sign marking device provided in the embodiment of the present application, and the traffic sign marking device includes:
a first obtaining unit 401, configured to obtain position information of each camera corresponding to each traffic sign image of a traffic sign, where each traffic sign image is an image of the traffic sign in a different camera position;
a second obtaining unit 402, configured to obtain position information of each pixel of a central point of the traffic sign in each traffic sign image;
a calculating unit 403, configured to calculate, according to the pose information of each camera and the position information of each pixel, position information of the traffic sign in a three-dimensional space;
and a labeling unit 404, configured to label the traffic sign in the three-dimensional map according to the position information.
In this embodiment, first, camera pose information corresponding to each traffic sign image of a traffic sign is acquired, and position information of each pixel of a central point of the traffic sign in each traffic sign image is acquired, then, position information of the traffic sign in a three-dimensional space is obtained through calculation according to the camera pose information and the position information of each pixel, and finally, the traffic sign is marked in a three-dimensional map according to the position information. The method and the device have the advantages that the position information of the traffic sign in the three-dimensional space is obtained in a mode of automatically processing the traffic sign image without an external measuring device, and obviously, the complexity of obtaining the position information of the traffic sign can be effectively reduced and the time can be saved compared with the mode that the position information of the traffic sign in the three-dimensional space is measured by a device with depth measuring capability such as a laser radar in the prior art so as to obtain the position information.
Optionally, in some possible embodiments of the present application, the calculating unit 403 is specifically configured to generate a pixel matrix according to the position information of each pixel;
generating a pose matrix associated with the pixel matrix according to the pose information of each camera;
obtaining a projection matrix according to the pose matrix and the acquired camera internal parameters;
generating a target matrix according to the pixel matrix and the projection matrix;
performing singular value decomposition on the target matrix to obtain a decomposition result;
and calculating the position information of the traffic sign in the three-dimensional space according to the decomposition result.
Optionally, in some possible embodiments of the present application, the method further includes:
the projection unit is used for projecting the coordinate points corresponding to the position information through the projection matrix to obtain projection pixels of the coordinate points;
a construction unit for constructing an adjustment equation from the projected pixels and the pixel matrix;
and the adjusting unit is used for adjusting the position information according to the adjusting equation.
Optionally, in some possible embodiments of the present application, the method further includes:
a third obtaining unit configured to obtain a number of rows of the pixel matrix;
the generating unit is used for generating a pose matrix associated with the pixel matrix according to the line number of the pixel matrix, and the camera position and the camera orientation which respectively correspond to the pose information of each camera;
and the number of rows of the pose matrix is the same as that of the pixel matrix.
Optionally, in some possible embodiments of the present application, the method further includes:
a fourth acquisition unit configured to acquire an inverse matrix of the pose matrix;
and the fifth acquisition unit is used for multiplying the inverse matrix and the acquired camera internal parameters to obtain a projection matrix.
Optionally, in some possible embodiments of the present application, the target matrix is an m × n matrix, and further includes:
a determining unit, configured to determine an m-order matrix, an m × n matrix, and an n-order matrix in the decomposition result;
a sixth obtaining unit, configured to divide each element n-1 before a last column of the n-order matrix by a last element of the last column of the n-order matrix to obtain position information of the traffic sign in a three-dimensional space;
wherein m and n are both positive integers not less than 2.
Optionally, in some possible embodiments of the present application, the second obtaining unit 402 is specifically configured to label the traffic sign in each traffic sign image to obtain label information;
and obtaining the position information of each pixel of the central point of the traffic sign in each traffic sign image according to the labeling information.
Optionally, in some possible embodiments of the present application, the method further includes:
a seventh obtaining unit, configured to obtain color information and circumscribed rectangle information in the labeling information;
and the training unit is used for inputting the color information and the circumscribed rectangle information serving as a training data set into the traffic sign detection model to obtain the trained traffic sign detection model.
Fig. 5 illustrates a physical structure diagram of a computer device, and as shown in fig. 5, the computer device may include: a processor (processor)501, a communication Interface (Communications Interface)502, a memory (memory)503, and a communication bus 504, wherein the processor 501, the communication Interface 502, and the memory 503 are configured to communicate with each other via the communication bus 504. The processor 501 may call logic instructions in the memory 503 to perform the following method: acquiring camera pose information corresponding to each traffic sign image of a traffic sign, wherein each traffic sign image is an image of the traffic sign under different camera poses; acquiring position information of each pixel of the central point of the traffic sign in each traffic sign image; calculating the position information of the traffic sign in the three-dimensional space according to the position information of each camera and the position information of each pixel; and marking the traffic sign in the three-dimensional map according to the position information.
In addition, the logic instructions in the memory 503 may be implemented in the form of software functional units and stored in a computer readable storage medium when the logic instructions are sold or used as independent products. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
On the other hand, the embodiments of the present application also provide a storage medium, on which a computer program is stored, where the computer program is implemented to perform the method provided by the foregoing embodiments when executed by a processor, for example, the method includes: acquiring camera pose information corresponding to each traffic sign image of a traffic sign, wherein each traffic sign image is an image of the traffic sign under different camera poses; acquiring position information of each pixel of the central point of the traffic sign in each traffic sign image; calculating the position information of the traffic sign in the three-dimensional space according to the position information of each camera and the position information of each pixel; and marking the traffic sign in the three-dimensional map according to the position information.
The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.
Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware. With this understanding in mind, the above-described technical solutions may be embodied in the form of a software product, which can be stored in a computer-readable storage medium such as ROM/RAM, magnetic disk, optical disk, etc., and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in the embodiments or some parts of the embodiments.
Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (8)
1. A method of traffic sign labeling, the method comprising:
acquiring camera pose information corresponding to each traffic sign image of a traffic sign, wherein each traffic sign image is an image of the traffic sign under different camera poses;
acquiring position information of each pixel of the central point of the traffic sign in each traffic sign image;
calculating the position information of the traffic sign in the three-dimensional space according to the position information of each camera and the position information of each pixel;
marking the traffic sign in the three-dimensional map according to the position information;
the step of calculating the position information of the traffic sign in the three-dimensional space according to the pose information of each camera and the position information of each pixel comprises the following steps:
generating a pixel matrix according to the position information of each pixel;
generating a pose matrix associated with the pixel matrix according to the pose information of each camera;
obtaining a projection matrix according to the pose matrix and the acquired camera internal parameters;
generating a target matrix according to the pixel matrix and the projection matrix, wherein the target matrix is an m multiplied by n matrix;
performing singular value decomposition on the target matrix to obtain a decomposition result;
calculating the position information of the traffic sign in the three-dimensional space according to the decomposition result;
the step of calculating the position information of the traffic sign in the three-dimensional space according to the decomposition result comprises the following steps:
determining an m-order matrix, an m x n matrix and an n-order matrix in the decomposition result;
dividing each element of the last column front n-1 of the n-order matrix by the last element of the last column of the n-order matrix to obtain the position information of the traffic sign in the three-dimensional space;
wherein m and n are both positive integers not less than 2.
2. The method for labeling a traffic sign according to claim 1, wherein after the step of calculating the position information of the traffic sign in the three-dimensional space according to the decomposition result, the method further comprises:
projecting the coordinate points corresponding to the position information through the projection matrix to obtain projection pixels of the coordinate points;
constructing an adjustment equation according to the projection pixels and the pixel matrix;
and adjusting the position information according to the adjustment equation.
3. The method according to claim 1, wherein the step of generating a pose matrix associated with the pixel matrix according to the pose information of each camera comprises:
acquiring the row number of the pixel matrix;
generating a pose matrix associated with the pixel matrix according to the line number of the pixel matrix, and the camera position and the camera orientation respectively corresponding to the pose information of each camera;
and the number of rows of the pose matrix is the same as that of the pixel matrix.
4. The method for labeling a traffic sign according to claim 1, wherein the step of obtaining a projection matrix according to the pose matrix and the acquired camera parameters comprises:
acquiring an inverse matrix of the pose matrix;
and multiplying the inverse matrix by the acquired camera internal parameters to obtain a projection matrix.
5. The method for labeling a traffic sign according to any one of claims 1 to 4, wherein the step of obtaining the position information of each pixel of the center point of the traffic sign in each image of the traffic sign comprises:
labeling the traffic signs in the traffic sign images to obtain labeling information;
and obtaining the position information of each pixel of the central point of the traffic sign in each traffic sign image according to the labeling information.
6. The method of claim 5, further comprising:
acquiring color information and circumscribed rectangle information in the labeling information;
and inputting the color information and the circumscribed rectangle information serving as a training data set to a traffic sign detection model to obtain the trained traffic sign detection model.
7. A traffic sign marking apparatus, comprising:
the first acquisition unit is used for acquiring the position and orientation information of each camera corresponding to each traffic sign image of a traffic sign, wherein each traffic sign image is an image of the traffic sign under different camera positions and orientations;
a second acquisition unit configured to acquire position information of each pixel of a center point of the traffic sign in each traffic sign image;
the calculation unit is used for calculating position information of the traffic sign in a three-dimensional space according to the pose information of each camera and the position information of each pixel;
the marking unit is used for marking the traffic sign in the three-dimensional map according to the position information;
the computing unit is specifically configured to generate a pixel matrix according to the position information of each pixel; generating a pose matrix associated with the pixel matrix according to the pose information of each camera; obtaining a projection matrix according to the pose matrix and the acquired camera internal parameters; generating a target matrix according to the pixel matrix and the projection matrix; performing singular value decomposition on the target matrix to obtain a decomposition result; calculating the position information of the traffic sign in a three-dimensional space according to the decomposition result, wherein the target matrix is an m multiplied by n matrix;
the calculation unit is specifically configured to determine an m-order matrix, an m × n matrix, and an n-order matrix in the decomposition result when the calculation according to the decomposition result is performed to obtain the position information of the traffic sign in the three-dimensional space; dividing each element of the last column front n-1 of the n-order matrix by the last element of the last column of the n-order matrix to obtain the position information of the traffic sign in the three-dimensional space; wherein m and n are both positive integers not less than 2.
8. A computer arrangement comprising a memory, a processor and a computer program stored on the memory and running on the processor, the processor when executing the program implementing the steps of the traffic sign labeling method according to any of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110060876.5A CN112381876B (en) | 2021-01-18 | 2021-01-18 | Traffic sign marking method and device and computer equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110060876.5A CN112381876B (en) | 2021-01-18 | 2021-01-18 | Traffic sign marking method and device and computer equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112381876A CN112381876A (en) | 2021-02-19 |
| CN112381876B true CN112381876B (en) | 2021-04-16 |
Family
ID=74581990
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110060876.5A Active CN112381876B (en) | 2021-01-18 | 2021-01-18 | Traffic sign marking method and device and computer equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112381876B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113763572B (en) * | 2021-09-17 | 2023-06-27 | 北京京航计算通讯研究所 | 3D entity labeling method based on AI intelligent recognition and storage medium |
| CN113763573B (en) * | 2021-09-17 | 2023-07-11 | 北京京航计算通讯研究所 | Digital labeling method and device for three-dimensional object |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110163064A (en) * | 2018-11-30 | 2019-08-23 | 腾讯科技（深圳）有限公司 | A kind of recognition methods of Sign for road, device and storage medium |
| CN111664845A (en) * | 2019-03-05 | 2020-09-15 | 千寻位置网络有限公司 | Traffic sign positioning method, visual map making method, device and positioning system |
| CN111783820A (en) * | 2020-05-08 | 2020-10-16 | 北京沃东天骏信息技术有限公司 | Image annotation method and device |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111247460A (en) * | 2017-08-29 | 2020-06-05 | 艾利丹尼森公司 | Retroreflective sheeting for projector-based display systems |
| CN110766760B (en) * | 2019-10-21 | 2022-08-02 | 北京百度网讯科技有限公司 | Method, device, equipment and storage medium for camera calibration |
-
2021
- 2021-01-18 CN CN202110060876.5A patent/CN112381876B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110163064A (en) * | 2018-11-30 | 2019-08-23 | 腾讯科技（深圳）有限公司 | A kind of recognition methods of Sign for road, device and storage medium |
| CN111664845A (en) * | 2019-03-05 | 2020-09-15 | 千寻位置网络有限公司 | Traffic sign positioning method, visual map making method, device and positioning system |
| CN111783820A (en) * | 2020-05-08 | 2020-10-16 | 北京沃东天骏信息技术有限公司 | Image annotation method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112381876A (en) | 2021-02-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11967109B2 (en) | Vehicle localization using cameras | |
| EP3407294B1 (en) | Information processing method, device, and terminal | |
| CN106980813B (en) | Gaze generation for machine learning | |
| CN110869974B (en) | Point cloud processing method, equipment and storage medium | |
| CN108694882B (en) | Method, device and equipment for labeling map | |
| CN109858374B (en) | Automatic extraction method and device for arrow mark lines in high-precision map making | |
| EP4016457A1 (en) | Positioning method and apparatus | |
| CN111539484B (en) | Method and device for training neural network | |
| CN112381876B (en) | Traffic sign marking method and device and computer equipment | |
| CN112802092B (en) | Obstacle sensing method and device and electronic equipment | |
| CN109840463B (en) | Lane line identification method and device | |
| EP4198901A1 (en) | Camera extrinsic parameter calibration method and apparatus | |
| WO2020199057A1 (en) | Self-piloting simulation system, method and device, and storage medium | |
| CN114295139A (en) | Cooperative sensing positioning method and system | |
| CN111695497B (en) | Pedestrian recognition method, medium, terminal and device based on motion information | |
| CN112115820A (en) | Vehicle-mounted driving assisting method and device, computer device and readable storage medium | |
| CN114919584A (en) | Motor vehicle fixed point target distance measuring method and device and computer readable storage medium | |
| CN115457354A (en) | Fusion method, 3D target detection method, vehicle-mounted device and storage medium | |
| CN112767412B (en) | Vehicle part classification method and device and electronic equipment | |
| Petrovai et al. | Obstacle detection using stereovision for Android-based mobile devices | |
| CN112580489A (en) | Traffic light detection method and device, electronic equipment and storage medium | |
| CN117079238A (en) | Road edge detection method, device, equipment and storage medium | |
| CN112639822A (en) | Data processing method and device | |
| CN116681965A (en) | Training method of target detection model and target detection method | |
| CN112837365B (en) | Image-based vehicle positioning method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20211209 Address after: 215000 room 808, 8 / F, building 9a, launch area of Yangtze River Delta International R & D community, No. 286, qinglonggang Road, high speed rail new town, Xiangcheng District, Suzhou City, Jiangsu Province Patentee after: Tianyi Transportation Technology Co.,Ltd. Address before: 2nd floor, building A3, Hongfeng science and Technology Park, Nanjing Economic and Technological Development Zone, Nanjing, Jiangsu Province 210033 Patentee before: CIIC Technology Co.,Ltd. |
|
| TR01 | Transfer of patent right | ||
| EE01 | Entry into force of recordation of patent licensing contract |
Application publication date: 20210219 Assignee: Zhongzhixing (Shanghai) Transportation Technology Co.,Ltd. Assignor: Tianyi Transportation Technology Co.,Ltd. Contract record no.: X2022980005387 Denomination of invention: A traffic sign marking method, device and computer equipment Granted publication date: 20210416 License type: Common License Record date: 20220518 |
|
| EE01 | Entry into force of recordation of patent licensing contract | ||
| EE01 | Entry into force of recordation of patent licensing contract |
Application publication date: 20210219 Assignee: CIIC Technology Co.,Ltd. Assignor: Tianyi Transportation Technology Co.,Ltd. Contract record no.: X2022980005922 Denomination of invention: A traffic sign marking method, device and computer equipment Granted publication date: 20210416 License type: Common License Record date: 20220524 |
|
| EE01 | Entry into force of recordation of patent licensing contract |