CN111915672B - Target labeling method and device based on 3D virtual driving scene - Google Patents
Target labeling method and device based on 3D virtual driving scene Download PDFInfo
- Publication number
- CN111915672B CN111915672B CN202010695891.2A CN202010695891A CN111915672B CN 111915672 B CN111915672 B CN 111915672B CN 202010695891 A CN202010695891 A CN 202010695891A CN 111915672 B CN111915672 B CN 111915672B
- Authority
- CN
- China
- Prior art keywords
- virtual
- scene
- virtual driving
- driving
- background picture
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000002372 labelling Methods 0.000 title claims abstract description 51
- 230000033001 locomotion Effects 0.000 claims description 20
- 238000000034 method Methods 0.000 claims description 20
- 238000012937 correction Methods 0.000 claims description 12
- 230000007175 bidirectional communication Effects 0.000 claims description 3
- 230000009466 transformation Effects 0.000 claims description 3
- 238000011161 development Methods 0.000 description 4
- 230000000694 effects Effects 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 2
- 238000002474 experimental method Methods 0.000 description 2
- 238000004088 simulation Methods 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000009795 derivation Methods 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 230000001788 irregular Effects 0.000 description 1
- 230000001795 light effect Effects 0.000 description 1
- 239000002245 particle Substances 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 238000013139 quantization Methods 0.000 description 1
- 230000008707 rearrangement Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/60—Rotation of whole images or parts thereof
- G06T3/604—Rotation of whole images or parts thereof using coordinate rotation digital computer [CORDIC] devices
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/235—Processing of additional data, e.g. scrambling of additional data or processing content descriptors
- H04N21/2353—Processing of additional data, e.g. scrambling of additional data or processing content descriptors specifically adapted to content descriptors, e.g. coding, compressing or processing of metadata
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10004—Still image; Photographic image
- G06T2207/10012—Stereo images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30204—Marker
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Geometry (AREA)
- Computer Graphics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Software Systems (AREA)
- Library & Information Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Traffic Control Systems (AREA)
- Processing Or Creating Images (AREA)
Abstract
The invention provides a target labeling method and equipment based on a 3D virtual driving scene, wherein the target labeling method comprises the steps of constructing the 3D virtual driving scene, setting virtual scene elements to be labeled at preset positions of the virtual driving scene, obtaining spatial three-dimensional coordinates of four vertexes of a minimum external rectangle tangent to the front surface of the virtual scene elements, carrying out virtual driving shooting according to a virtual driving route through a virtual vehicle with a virtual camera, defining pictures containing the virtual scene elements to be labeled in pictures as key background pictures, and carrying out batch labeling on the virtual scene elements in the minimum external rectangle in the key background pictures. The invention realizes batch marking and batch exporting of the targets by depending on the 3D virtual scene, and improves marking efficiency and precision.
Description
Technical Field
The invention relates to the field of intelligent driving of automobiles, in particular to a target labeling method and device based on a 3D virtual driving scene.
Background
Currently, most data annotation is for video images. In the traditional labeling method, each frame of image is generally extracted first, and coordinate position mark information of key concern factors in the image is given, so that the efficiency is low and the resource consumption of a system CPU is high.
In addition, patent CN106791937A proposes a method and system for labeling a video image, in which a canvas is created, pixels of the canvas correspond to pixels of a video image to be labeled, labeling content is drawn at a designated position of the canvas according to a received labeling command and a corresponding relationship between the pixels, format conversion is performed on the canvas to obtain a first labeled image, a pixel value of the labeling content on the first labeled image is assigned to a corresponding position on the video image to be labeled according to the corresponding relationship between the pixels to obtain a labeled video image, and thus, labeling of the video image is completed. The canvas is used for drawing the annotation content, so that the resource consumption of a system CPU (central processing unit) during video image annotation can be reduced, but the annotation efficiency is relatively low.
Moreover, intelligent driving is taken as the development direction of the future automobile industry, relevant policy and regulation are gradually formed, the technology in the industry is continuously improved, and the development prospect is promising. No matter the intelligent driving real vehicle experiment or the simulation experiment, the vehicle-mounted camera can acquire road condition images in the driving process to assist driving decision. Therefore, labeling the data of the driving image becomes a starting point of the machine-perceived world. At present, data marking of driving images is to manually mark a large number of shot road condition images one by one, and marked data is used for learning of a vehicle-mounted camera.
The current data labeling is to process two-dimensional images, the result of labeling specific areas in a 3D virtual scene for reconstructing an objective world is less, and the quantization labeling precision and the labeling efficiency in the existing labeling method are few.
In view of the above, the present invention is particularly proposed.
Disclosure of Invention
The invention aims to provide a target labeling method based on a 3D virtual driving scene, which realizes batch labeling and batch export of targets by means of the 3D virtual scene and improves the efficiency and the precision of labeling.
In order to achieve the purpose, the invention provides the following technical scheme:
in a first aspect, the present invention provides a target labeling method based on a 3D virtual driving scene, the method including the steps of:
step 1: building a 3D virtual driving scene, wherein the 3D virtual driving scene comprises a virtual road surface and first virtual scene elements arranged at a plurality of preset positions along a virtual path formed by the virtual road surface;
step 2: creating a minimum bounding rectangle tangent to the front face of the first virtual scene element model, and combining the spatial three-dimensional coordinates P of four vertexes of the minimum bounding rectangle in (X in ,Y in ,Z in Where i is an integer, and n ═ 1,2,3,4) are defined as the four bin points of the first virtual scene element;
and step 3: in the 3D virtual driving scene, virtual driving shooting is carried out through a virtual vehicle provided with a virtual camera according to a virtual driving route, and a first position L of each first virtual scene element entering a shooting picture of the virtual camera is determined i1 And a second position L away from the virtual camera to shoot pictures i2 ；
And 4, step 4: setting a preset movement rate of the virtual vehicle on the virtual driving route, and calculating the first position L of the virtual vehicle passing through based on the preset movement rate i1 First time node T of i1 And passing through said second position L i2 Second time node T i2 So as to obtain a label configuration table [ T ] of each first virtual driving scene i1 ,T i2 ,P i1 ,P i2 ,P i3 ,P i4 ]；
And 5: in the 3D virtual driving scene, the virtual vehicle-mounted camera performs virtual driving shooting at the preset speed according to a virtual driving route, and the shooting is performed at the first time node T i1 And a second time node T i2 The background picture shot in the time interval is defined as a key background pictureSlice and to the spatial three-dimensional coordinate P in the key background picture in Marking the first virtual scene element in a rectangle formed by the corresponding four slot points to obtain marking information, and storing the key background picture and the marking information;
preferably, the spatial three-dimensional coordinates P in the key background picture in step 5 are determined in Marking the first virtual scene element in the rectangle formed by the corresponding four slot points to obtain marking information, and specifically comprising the following steps of:
step 5-1: setting the spatial three-dimensional coordinate P of the four slot points in the 3D virtual driving scene in (X in ,Y in ,Z in ) Converting into view coordinates S in the key background picture through coordinate transformation in (x in ,y in )；
Step 5-2: setting the mark information format of the key background picture as [ x ] i1 ,y i1 ,x i2 ,y i2 ,x i3 ,y i3 ,x i4 ,y i4 Type text description of the first virtual scene element]；
Preferably, view coordinates S of four bin points of said first virtual scene element are also taken after step 5-2 in The correction is carried out, and the method specifically comprises the following steps:
step 5-1-1: comparing the view coordinates S of four slot points in (x in ,y in ) Separately obtaining Min (x) in )，Min(y in ),Max(x in ),Max(y in ) And then the view correction coordinates of the four slot points are S' i1 (x imin ,y imin ),S′ i2 (x imax ,y imin ),S′ i3 (x imax ,y imax ),S′ i4 (x imin ,y imax )；
Preferably, the method also comprises correcting the coordinate S 'if the views of the four slot points are corrected after the step 5-1-1' i1 (x imin ,y imin ),S′ i2 (x imax ,y imin ),S′ i3 (x imax ,y imax ),S′ i4 (x imin ,y imax ) If the boundary of the key background picture is exceeded, correcting the coordinates of the slot points exceeding the boundary into the coordinate values of the intersection points of the minimum external rectangle and the boundary of the key background picture;
preferably, the first virtual scene element comprises a plurality of different category models, and the arrangement position of each category model in the space is set according to the requirement of diversification of the 3D virtual driving scene;
preferably, the 3D virtual driving scene further includes a plurality of other virtual scene elements, and the configuration relationship of the other virtual scene elements with respect to the first virtual scene element is set by the program.
On the other hand, the invention provides a target labeling device based on a 3D virtual driving scene, which realizes labeling of virtual scene elements of the 3D virtual driving scene by adopting the target labeling method, and the target labeling device comprises,
the system comprises a scene building module, a scene building module and a driving control module, wherein the scene building module is used for building a 3D virtual driving scene, and the 3D virtual driving scene comprises a virtual road surface and first virtual scene elements arranged at a plurality of preset positions along a virtual path formed by the virtual road surface;
a model editing module for creating a minimum bounding rectangle tangent to the front surface of the first virtual scene element and defining the spatial three-dimensional coordinates P of four vertexes of the minimum bounding rectangle in Four slot points defined as first virtual scene elements, said spatial three-dimensional coordinate P in Is X in ,Y in ,Z in Wherein i is an integer, n is 1,2,3, 4;
a position determining module, configured to perform virtual driving shooting according to a virtual driving route by a virtual vehicle equipped with a virtual camera in the 3D virtual driving scene, and determine that each first virtual scene element enters a first position L of a shooting picture of the virtual camera i1 And a second position L away from the virtual camera to shoot pictures i2 ；
A mark configuration module for setting a preset movement rate of the virtual vehicle on the virtual driving route and calculating the virtual vehicle based on the preset movement rateThe vehicle passing said first position L i1 First time node T of i1 And passing through said second position L i2 Second time node T i2 So as to obtain a label configuration table [ T ] of each first virtual driving scene i1 ,T i2 ,P i1 ,P i2 ,P i3 ,P i4 ]；
A picture marking module, configured to, in the 3D virtual driving scene, perform virtual driving shooting by the virtual vehicle-mounted camera at the preset rate according to a virtual driving route, to be at the first time node T i1 And a second time node T i2 Defining the background picture shot in the time interval as a key background picture, and calculating the space three-dimensional coordinate P in the key background picture in Marking the first virtual scene element in a rectangle formed by the corresponding four slot points to obtain marking information, and storing the key background picture and the marking information;
the scene building module, the model editing module, the position determining module, the labeling configuration module and the picture labeling module are sequentially in bidirectional communication connection.
Preferably, the target annotation device further comprises a coordinate correction module, configured to correct view coordinates S of four slot points of the first virtual scene element in And (6) performing correction.
Compared with the prior art, the invention has the following beneficial technical effects:
(1) compared with the traditional method that a large number of road condition images are collected by a real vehicle and manually marked one by one, the method has the advantages that the 3D virtual driving scene is built, the virtual vehicle-mounted camera shooting is carried out in the 3D virtual driving scene, the batch marking and batch exporting of the images are further realized, the efficiency is high, and the exporting speed can reach 100 pieces/minute;
(2) the virtual scene element configuration in the 3D virtual driving scene can be conveniently set through a program, so that the marked image set is richer and closer to reality;
(3) the data labeling precision is high, and the coordinate error is within 1 pixel value through the special traffic element target labeled by the minimum circumscribed rectangle;
(4) the resolution of the derived pictures is high, and each picture is 3200 x 1800 pixels and has the size of about 10 MB.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts.
FIG. 1 is a key background picture taken by a virtual vehicle-mounted camera in a 3D virtual driving scene according to the present invention;
fig. 2 is a traffic sign board photographed by a virtual vehicle-mounted camera in a 3D virtual driving scene in a non-frontal manner according to the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the technical solutions of the present invention will be clearly and completely described below. It is to be understood that the described embodiments are merely exemplary of the invention, and not restrictive of the full scope of the invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
According to one aspect of the invention, a target labeling method based on a 3D virtual driving scene is provided, and the method comprises the following steps:
step 1: the method comprises the steps of building a 3D virtual driving scene, wherein the 3D virtual driving scene comprises a virtual road surface and first virtual scene elements arranged at a plurality of preset positions along a virtual path formed by the virtual road surface.
Specifically, intelligent driving is taken as the development direction of the future automobile industry, relevant policy and regulations are gradually formed, the technology in the industry is continuously improved, and the development prospect is promising. No matter the intelligent driving real vehicle experiment or the simulation experiment, the vehicle-mounted camera can acquire road condition images in the driving process to assist driving decision. Therefore, the data annotation on the driving image becomes the starting point of the machine-perceived world. In this embodiment, a 3D virtual driving scene is constructed, which includes a virtual road surface and first virtual scene elements arranged at a plurality of preset positions along a virtual path formed by the virtual road surface, where the first virtual scene elements may be any traffic participation elements that may appear in an actual traffic scene, including but not limited to a traffic sign, a vehicle, a pedestrian, a building, and the like, and in this embodiment, the first virtual scene elements are traffic signs. The 3D virtual driving scene comprises a plurality of virtual road surfaces, and traffic signboards are arranged at preset positions of the virtual road surfaces according to the requirements of subsequent picture marking, namely, the spatial positions of the traffic signboards in the picture frames where the traffic signboards are located are set artificially and are known.
Step 2: creating a minimum bounding rectangle tangent to the front face of the first virtual scene element model, and combining the spatial three-dimensional coordinates P of four vertexes of the minimum bounding rectangle in (X in ,Y in ,Z in Where i is an integer and n is 1,2,3,4) are defined as the four bin points of the first virtual scene element.
Specifically, after a first virtual scene element is arranged at a preset position of a 3D virtual driving scene, a minimum bounding rectangle tangent to the front surface of a first virtual scene element model is manually created artificially, and the spatial three-dimensional coordinates P of four vertexes of the minimum bounding rectangle are used for representing the three-dimensional coordinates P of the three-dimensional space in (X in ,Y in ,Z in Where i is an integer and n is 1,2,3,4) are defined as the four bin points of the first virtual scene element. And the subsequent picture content positioned in the minimum circumscribed rectangle is the content needing to be learned by the machine.
And step 3: in the 3D virtual driving scene, virtual driving shooting is carried out through a virtual vehicle provided with a virtual camera according to a virtual driving route, and a first position L of each first virtual scene element entering a shooting picture of the virtual camera is determined i1 And a second position L away from the virtual camera to shoot pictures i2 。
In particular, according to a 3D virtual driving scenarioSetting a virtual driving route, wherein the virtual driving route can be selected according to the specific situation of the 3D virtual driving scene, the selection principle includes but is not limited to that the virtual driving route includes as many first virtual scene elements as possible, in this embodiment, when the virtual driving route is selected, a route including as many traffic signs as possible is preferably used as the virtual driving route; after a virtual driving route is selected, virtual driving shooting is carried out through a virtual vehicle with a virtual camera according to the virtual driving route, in the invention, pictures with traffic signs appearing in pictures shot by the virtual camera are pictures which need to be marked subsequently, the pictures are defined as key background pictures, and each traffic sign arranged along the way is determined to enter a first position L of a picture shot by the virtual camera i1 And a second position L away from the virtual camera to shoot pictures i2 Then in the first position L i1 And a second position L i2 The pictures shot by the virtual camera in the position interval are all key background pictures, and the subsequent pictures need to be marked.
And 4, step 4: setting a preset movement rate of the virtual vehicle on the virtual driving route, and calculating the virtual vehicle passing through the first position L based on the preset movement rate i1 First time node T of i1 And passing through said second position L i2 Second time node T i2 So as to obtain a label configuration table [ T ] of each first virtual driving scene i1 ,T i2 ,P i1 ,P i2 ,P i3 ,P i4 ]；
Specifically, the first position L corresponding to each traffic sign is determined in step 3 i1 And a second position L i2 Then, setting a preset movement rate of the virtual vehicle on the virtual driving route, and calculating the virtual vehicle to pass through the first position L based on the preset movement rate i1 First time node T of i1 And passing through said second position L i2 Second time node T i2 So as to obtain a label configuration table [ T ] of each first virtual driving scene i1 ,T i2 ,P i1 ,P i2 ,P i3 ,P i4 ]. Based on the information of the labeling configuration table, the virtual vehicle is located at (T) when the virtual vehicle runs on the virtual driving route according to the preset movement speed i1 ,T i2 ) All background pictures shot in a time interval are key background pictures, wherein a space three-dimensional coordinate P i1 ,P i2 ,P i3 ,P i4 The picture content in the rectangular frame formed by the corresponding four slot points is the content to be marked in the key background picture.
And 5: in the 3D virtual driving scene, the virtual vehicle-mounted camera performs virtual driving shooting at the preset speed according to a virtual driving route, and the shooting is performed at the first time node T i1 And a second time node T i2 Defining the background picture shot in the time interval as a key background picture, and calculating a spatial three-dimensional coordinate P in the key background picture in Marking the first virtual scene element in a rectangle formed by the corresponding four slot points to obtain marking information, and storing the key background picture and the marking information;
specifically, in the 3D virtual driving scene, virtual driving shooting is performed according to a virtual driving route through a virtual vehicle-mounted camera, as shown in fig. 1, at the first time node T i1 And a second time node T i2 In the time interval, the background picture shot by the virtual vehicle-mounted camera comprises a traffic sign board, the picture frame comprising the traffic sign board is defined as a key background picture, and the space three-dimensional coordinate P in the key background picture shot by the virtual vehicle-mounted camera i1 ,P i2 ,P i3 ,P i4 And marking the traffic sign board in the rectangular frame formed by the corresponding four groove points to obtain marking information, and storing the key background picture and the marking information.
Preferably, the spatial three-dimensional coordinates P in the key background picture in step 5 are determined in Marking the first virtual scene element in the rectangle formed by the corresponding four slot points to obtain marking information, and specifically comprising the following steps of:
step 5-1: empty the four slot points in the 3D virtual driving sceneThree-dimensional coordinate P in (X in ,Y in ,Z in ) Converting into view coordinates S in the key background picture through coordinate transformation in (x in ,y in )；
Step 5-2: setting the mark information format of the key background picture as (x) i1 ,y i1 ,x i2 ,y i2 ,x i3 ,y i3 ,x i4 ,y i4 Type caption of the first virtual scene element);
specifically, in this embodiment, the first virtual scene element is a traffic sign, the traffic sign has multiple categories, such as a highest running speed sign, a lowest running speed sign, a left turn prohibition sign, a warning sign, and the like, and for a key background picture including the traffic sign, the labeling information of the key background picture is view coordinates of four slot points of the traffic sign and the category of the traffic sign, for example, if the traffic sign is that the vehicle speed is not more than 50 km/h, the labeling information format of the key background picture is [ x [, [ x ] m i1 ,y i1 ,x i2 ,y i2 ,x i3 ,y i3 ,x i4 ,y i4 The speed of the vehicle must not exceed 50 km/h]。
Preferably, the view coordinates S of the four bin points of the first virtual scene element are also applied after step 5-2 in The correction is carried out, and the method specifically comprises the following steps:
step 5-1-1: comparing the view coordinates S of four slot points in (x in ,y in ) Separately obtaining Min (x) in )，Min(y in ),Max(x in ),Max(y in ) And then the view correction coordinates of the four slot points are S' i1 (x imin ,y imin ),S′ i2 (x imax ,y imin ),S′ i3 (x imax ,y imax ),S′ i4 (x imin ,y imax )；
Preferably, the method also comprises the step of correcting the coordinate S 'if the view of the four slot points is corrected after the step 5-1-1' i1 (x imin ,y imin ),S′ i2 (x imax ,y imin ),S′ i3 (x imax ,y imax ),S′ i4 (x imin ,y imax ) If the boundary of the key background picture is exceeded, correcting the coordinates of the slot points exceeding the boundary into the coordinate values of the intersection points of the minimum external rectangle and the boundary of the key background picture;
specifically, as shown in fig. 2, the view coordinates S of the four slot points obtained in step 5-2 are due to the problem of the shooting angle of the virtual in-vehicle camera in the 3D virtual driving environment in (x in ,y in ) The quadrilateral formed may be an irregular quadrilateral. In order to ensure that the shape of the marking frame is always the minimum circumscribed rectangle, view coordinates S of four groove points are compared in (x in ,y in ) Separately obtaining Min (x) in )，Min(y in ),Max(x in ),Max(y in ) And then the view correction coordinates of the four slot points are S' i1 (x imin ,y imin ),S′ i2 (x imax ,y imin ),S′ i3 (x imax ,y imax ),S′ i4 (x imin ,y imax )；
Preferably, the method also comprises the step of correcting the coordinate S 'if the view of the four slot points is corrected after the step 5-1-1' i1 (x imin ,y imin ),S′ i2 (x imax ,y imin ),S′ i3 (x imax ,y imax ),S′ i4 (x imin ,y imax ) And if the boundary exceeds the boundary of the key background picture, correcting the coordinates of the slot points exceeding the boundary into the coordinate values of the intersection points of the minimum external rectangle and the boundary of the key background picture.
In the invention, the motion trail and the motion rate of the virtual vehicle-mounted camera in the 3D virtual driving scene are planned according to the virtual driving route. And setting the extraction quantity and time interval of the key background pictures of the camera based on the motion rate, and finally, simultaneously and massively deriving the labeled key background pictures and labeled information in the csv format at the efficiency of 100 pieces/minute, wherein the error of the labeled coordinate of the traffic signboard is within one pixel.
Preferably, the first virtual scene element comprises a plurality of different category models, and the arrangement position of each category model on the space is reasonably set according to the diversified requirements of the 3D virtual driving scene;
specifically, in this embodiment, the first virtual scene element is a traffic sign, the traffic sign includes multiple traffic sign types, such as a highest running speed sign, a lowest running speed sign, a left-turn prohibition sign, a warning sign, and the like, multiple types of traffic signs are reasonably configured on the virtual road according to the requirement of diversification of the 3D virtual driving scene, and the traffic signs of the same type are arranged at different azimuth angles;
preferably, the 3D virtual driving scene further includes a plurality of other virtual scene elements, and the configuration relationship of the other virtual scene elements with respect to the first virtual scene element is set by the program.
Specifically, in this embodiment, the 3D virtual driving scene may further include traffic participants such as trees, buildings, pedestrians, and weather, and the weather setting in the scene sets different light effects according to different weather, so that an environmental effect similar to the real world is created by matching the particle special effect. In order to solve the scene singleness of an image marked by a static scene and realize the diversification of the marked image, the method is used for the scene marked by the image, the diversification of the marked scene is ensured, and the diversification of the scene is realized by modifying the positions, directions, existence and nonexistence of participators such as trees, buildings, pedestrians and the like in the scene through program control on the basis of the static scene so as to realize the batch derivation of different pictures. Meanwhile, the weather setting in the scene can be modified, so that different weather effects are reflected during marking, and the marked image set is richer and closer to reality.
On the other hand, the invention also provides a target labeling device based on the 3D virtual driving scene, which adopts the target labeling method to label the virtual scene elements of the 3D virtual driving scene, the target labeling device comprises,
the system comprises a scene building module, a scene building module and a driving control module, wherein the scene building module is used for building a 3D virtual driving scene, and the 3D virtual driving scene comprises a virtual road surface and first virtual scene elements arranged at a plurality of preset positions along a virtual path formed by the virtual road surface;
a model editing module for creating a minimum bounding rectangle tangent to the front surface of the first virtual scene element and combining the spatial three-dimensional coordinates P of four vertexes of the minimum bounding rectangle in Four slot points defined as first virtual scene elements, said spatial three-dimensional coordinate P in Is X in ,Y in ,Z in Wherein i is an integer, n is 1,2,3, 4;
a position determining module, configured to perform virtual driving shooting according to a virtual driving route by a virtual vehicle equipped with a virtual camera in the 3D virtual driving scene, and determine that each first virtual scene element enters a first position L of a shooting picture of the virtual camera i1 And a second position L away from the virtual camera to shoot pictures i2 ；
A mark configuration module for setting a preset movement rate of the virtual vehicle on the virtual driving route, and calculating the virtual vehicle passing through the first position L based on the preset movement rate i1 First time node T of i1 And passing through said second position L i2 Second time node T i2 So as to obtain a label configuration table [ T ] of each first virtual driving scene i1 ,T i2 ,P i1 ,P i2 ,P i3 ,P i4 ]；
A picture marking module, configured to, in the 3D virtual driving scene, perform virtual driving shooting by the virtual vehicle-mounted camera at the preset rate according to a virtual driving route, to be at the first time node T i1 And a second time node T i2 Defining the background picture shot in the time interval as a key background picture, and calculating the space three-dimensional coordinate P in the key background picture in Marking the first virtual scene element in a rectangle formed by the corresponding four slot points to obtain marking information, and storing the key background picture and the marking information;
the scene building module, the model editing module, the position determining module, the labeling configuration module and the picture labeling module are sequentially in bidirectional communication connection.
Preferably, the target annotation device further comprises a coordinate correction module, configured to correct view coordinates S of four slot points of the first virtual scene element in And (6) performing correction.
From the above description, it can be seen that the above-described embodiments of the present invention achieve the following technical effects:
(1) compared with the traditional method that a large number of road condition images are collected by a real vehicle and manually marked one by one, the method has the advantages that the 3D virtual driving scene is built, the virtual vehicle-mounted camera shooting is carried out in the 3D virtual driving scene, the batch marking and batch exporting of the images are further realized, the efficiency is high, and the exporting speed can reach 100 pieces/minute;
(2) the virtual scene element configuration in the 3D virtual driving scene can be conveniently set through a program, so that the marked image set is richer and closer to reality;
(3) the data labeling precision is high, and the coordinate error is within 1 pixel value through the special traffic element target labeled by the minimum circumscribed rectangle;
(4) the resolution of the derived pictures is high, and each picture is 3200 x 1800 pixels and has the size of about 10 MB.
The foregoing is considered as illustrative of the preferred embodiments of the invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments illustrated herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims (8)
1. A target labeling method based on a 3D virtual driving scene is characterized by comprising the following steps,
step 1: building a 3D virtual driving scene, wherein the 3D virtual driving scene comprises a virtual road surface and first virtual scene elements arranged at a plurality of preset positions along a virtual path formed by the virtual road surface;
and 2, step: creating a minimum bounding rectangle tangent to the front of the first virtual scene element and relating the spatial three-dimensional coordinates P of the four vertices of the minimum bounding rectangle in Four slot points defined as first virtual scene elements, said spatial three-dimensional coordinate P in Is X in ,Y in ,Z in Wherein i is an integer, n is 1,2,3, 4;
and step 3: in the 3D virtual driving scene, virtual driving shooting is carried out through a virtual vehicle provided with a virtual camera according to a virtual driving route, and a first position L of each first virtual scene element entering a shooting picture of the virtual camera is determined i1 And a second position L away from the virtual camera to shoot pictures i2 ；
And 4, step 4: setting a preset movement rate of the virtual vehicle on the virtual driving route, and calculating the virtual vehicle passing through the first position L based on the preset movement rate i1 First time node T of i1 And passing through said second position L i2 Second time node T i2 So as to obtain a label configuration table [ T ] of each first virtual driving scene i1 ,T i2 ,P i1 ,P i2 ,P i3 ,P i4 ]；
And 5: in the 3D virtual driving scene, the virtual camera performs virtual driving shooting at the preset movement rate according to a virtual driving route, and the virtual camera is to be shot at the first time node T i1 And a second time node T i2 Defining the background picture shot in the time interval as a key background picture, and calculating a spatial three-dimensional coordinate P in the key background picture in Marking the first virtual scene element in the rectangle formed by the corresponding four slot points to obtain marking information, and storing the key background picture and the marking information.
2. Target labeling method based on 3D virtual driving scene as claimed in claim 1The method is characterized in that the spatial three-dimensional coordinate P in the key background picture is processed in step 5 in Marking the first virtual scene element in the rectangle formed by the corresponding four slot points to obtain marking information, specifically comprising the following steps,
step 5-1: setting the spatial three-dimensional coordinate P of the four slot points in the 3D virtual driving scene in Converting into view coordinates S in the key background picture through coordinate transformation in Said view coordinate S in Is x in ,y in ；
Step 5-2: setting the mark information format of the key background picture as [ x ] i1 ,y i1 ,x i2 ,y i2 ,x i3 ,y i3 ,x i4 ,y i4 Type text description of the first virtual scene element]。
3. The method for labeling targets based on 3D virtual driving scene as claimed in claim 2, characterized in that, after step 5-1, the view coordinates S of four slot points of the first virtual scene element are also measured in The correction is carried out, which specifically comprises the following steps,
step 5-1-1: comparing the view coordinates S of four slot points in Separately obtaining Min (x) in )，Min(y in ),Max(x in ),Max(y in ) And then the view correction coordinates of the four slot points are S' i1 (x imin ,y imin ),S’ i2 (x imax ,y imin ),S’ i3 (x imax ,y imax ),S’ i4 (x imin ,y imax )。
4. The target labeling method based on the 3D virtual driving scene as claimed in claim 3, further comprising, after the step 5-1-1, correcting the coordinates S 'if the views of the four slot points correct' i1 (x imin ,y imin ),S’ i2 (x imax ,y imin ),S’ i3 (x imax ,y imax ),S’ i4 (x imin ,y imax ) Beyond the key background pictureAnd (4) correcting the coordinates of the slot points beyond the boundary into the coordinate values of the intersection points of the minimum circumscribed rectangle and the boundary of the key background picture.
5. The method for labeling the target based on the 3D virtual driving scene as claimed in any one of claims 1 to 4, wherein the first virtual scene element comprises a plurality of different category models, and the arrangement orientation of each category model in the space is set according to the diversified requirements of the 3D virtual driving scene.
6. The target labeling method based on the 3D virtual driving scene as claimed in claim 5, wherein the 3D virtual driving scene further comprises a plurality of other virtual scene elements, and the configuration relationship of the other virtual scene elements with respect to the first virtual scene element is set through a program.
7. A target labeling device based on a 3D virtual driving scene is characterized in that batch labeling of virtual scene elements of the 3D virtual driving scene is realized by adopting the target labeling method of any one of claims 1 to 6, the target labeling device comprises,
the system comprises a scene building module, a scene building module and a driving control module, wherein the scene building module is used for building a 3D virtual driving scene, and the 3D virtual driving scene comprises a virtual road surface and first virtual scene elements arranged at a plurality of preset positions along a virtual path formed by the virtual road surface;
a model editing module for creating a minimum bounding rectangle tangent to the front surface of the first virtual scene element and combining the spatial three-dimensional coordinates P of four vertexes of the minimum bounding rectangle in Four slot points defined as first virtual scene elements, said spatial three-dimensional coordinate P in Is X in ,Y in ,Z in Wherein i is an integer, n is 1,2,3, 4;
a position determining module, configured to perform virtual driving shooting according to a virtual driving route by a virtual vehicle equipped with a virtual camera in the 3D virtual driving scene, and determine that each first virtual scene element enters the virtual camera shooting screenFirst position L of i1 And a second position L away from the virtual camera to shoot pictures i2 ；
A mark configuration module for setting a preset movement rate of the virtual vehicle on the virtual driving route, and calculating the virtual vehicle passing through the first position L based on the preset movement rate i1 First time node T of i1 And passing said second position L i2 Second time node T i2 So as to obtain a label configuration table [ T ] of each first virtual driving scene i1 ,T i2 ,P i1 ,P i2 ,P i3 ,P i4 ]；
A picture marking module, configured to perform virtual driving shooting at the preset motion rate according to a virtual driving route by the virtual camera in the 3D virtual driving scene, where the virtual camera is to be at the first time node T i1 And a second time node T i2 Defining the background picture shot in the time interval as a key background picture, and calculating the space three-dimensional coordinate P in the key background picture in Marking the first virtual scene element in a rectangle formed by the corresponding four slot points to obtain marking information, and storing the key background picture and the marking information;
the scene building module, the model editing module, the position determining module, the labeling configuration module and the picture labeling module are sequentially in bidirectional communication connection.
8. The 3D virtual driving scene-based target annotation device of claim 7, further comprising a coordinate rectification module for correcting the view coordinates S of the four bin points of the first virtual scene element in And (6) performing correction.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010695891.2A CN111915672B (en) | 2020-07-20 | 2020-07-20 | Target labeling method and device based on 3D virtual driving scene |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010695891.2A CN111915672B (en) | 2020-07-20 | 2020-07-20 | Target labeling method and device based on 3D virtual driving scene |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111915672A CN111915672A (en) | 2020-11-10 |
| CN111915672B true CN111915672B (en) | 2022-08-16 |
Family
ID=73281134
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010695891.2A Active CN111915672B (en) | 2020-07-20 | 2020-07-20 | Target labeling method and device based on 3D virtual driving scene |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111915672B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112819969A (en) * | 2021-02-08 | 2021-05-18 | 广东三维家信息科技有限公司 | Virtual scene path generation method and device, electronic equipment and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105739704A (en) * | 2016-02-02 | 2016-07-06 | 上海尚镜信息科技有限公司 | Remote guidance method and system based on augmented reality |
| CN108932749A (en) * | 2017-05-26 | 2018-12-04 | 上海为森车载传感技术有限公司 | The method for generating object sample based on CG |
| CN111160261A (en) * | 2019-12-30 | 2020-05-15 | 北京每日优鲜电子商务有限公司 | Sample image labeling method and device for automatic sales counter and storage medium |
| CN111274927A (en) * | 2020-01-17 | 2020-06-12 | 北京三快在线科技有限公司 | Training data generation method and device, electronic equipment and storage medium |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10268201B2 (en) * | 2017-02-28 | 2019-04-23 | Mitsubishi Electric Research Laboratories, Inc. | Vehicle automated parking system and method |
-
2020
- 2020-07-20 CN CN202010695891.2A patent/CN111915672B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105739704A (en) * | 2016-02-02 | 2016-07-06 | 上海尚镜信息科技有限公司 | Remote guidance method and system based on augmented reality |
| CN108932749A (en) * | 2017-05-26 | 2018-12-04 | 上海为森车载传感技术有限公司 | The method for generating object sample based on CG |
| CN111160261A (en) * | 2019-12-30 | 2020-05-15 | 北京每日优鲜电子商务有限公司 | Sample image labeling method and device for automatic sales counter and storage medium |
| CN111274927A (en) * | 2020-01-17 | 2020-06-12 | 北京三快在线科技有限公司 | Training data generation method and device, electronic equipment and storage medium |
Non-Patent Citations (3)
| Title |
|---|
| PedX: Benchmark Dataset for Metric 3D Pose Estimation of Pedestrians in Complex Urban Intersections;Wonhui Kim et.al;《arXiv:1809.03605v1 [cs.CV]》;20180910;第1-8页 * |
| Synthesizing Image and Automated Annotation Tool for CNN based Under Water Object Detection;MyungHwan Jeon et.al;《Journal of Korea Robotics Society》;20191231;第14卷(第2期);第139-149页 * |
| 基于道路消失点的远距离路面微小障碍物检测;俞骏威 等;《同济大学学报（自然科学版）》;20191231;第47卷;第213-216页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111915672A (en) | 2020-11-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Geyer et al. | A2d2: Audi autonomous driving dataset | |
| US20150138310A1 (en) | Automatic scene parsing | |
| CN114413881B (en) | Construction method, device and storage medium of high-precision vector map | |
| CN102222341A (en) | Method and device for detecting motion characteristic point and method and device for detecting motion target | |
| CN113128507B (en) | License plate recognition method and device, electronic equipment and storage medium | |
| CN109544635B (en) | Camera automatic calibration method based on enumeration heuristic | |
| Ma et al. | Crlf: Automatic calibration and refinement based on line feature for lidar and camera in road scenes | |
| CN114821507A (en) | Multi-sensor fusion vehicle-road cooperative sensing method for automatic driving | |
| CN110009675A (en) | Generate method, apparatus, medium and the equipment of disparity map | |
| CN113763569A (en) | Image annotation method and device used in three-dimensional simulation and electronic equipment | |
| CN116524364B (en) | Heterogeneous multi-source data processing system and method based on feature recognition | |
| CN112257668A (en) | Main and auxiliary road judging method and device, electronic equipment and storage medium | |
| CN111915672B (en) | Target labeling method and device based on 3D virtual driving scene | |
| CN114972767A (en) | Vehicle track and course angle extraction method based on high-altitude unmanned aerial vehicle video | |
| CN114898353A (en) | License plate identification method based on video sequence image characteristics and information | |
| CN110415299B (en) | Vehicle position estimation method based on set guideboard under motion constraint | |
| CN112115737B (en) | Vehicle orientation determining method and device and vehicle-mounted terminal | |
| CN114820931B (en) | Virtual reality-based CIM (common information model) visual real-time imaging method for smart city | |
| CN115565155A (en) | Training method of neural network model, generation method of vehicle view and vehicle | |
| CN112101268B (en) | Vehicle line pressing detection method based on geometric projection | |
| Babahajiani et al. | Semantic parsing of street scene images using 3d lidar point cloud | |
| CN114648549A (en) | Traffic scene target detection and positioning method fusing vision and laser radar | |
| Du et al. | Validation of vehicle detection and distance measurement method using virtual vehicle approach | |
| CN106856013A (en) | The method and system that a kind of augmented reality identification figure off card shows | |
| CN113269088A (en) | Scene description information determining method and device based on scene feature extraction |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |