CN112419385A - 3D depth information estimation method and device and computer equipment - Google Patents
3D depth information estimation method and device and computer equipment Download PDFInfo
- Publication number
- CN112419385A CN112419385A CN202110093028.4A CN202110093028A CN112419385A CN 112419385 A CN112419385 A CN 112419385A CN 202110093028 A CN202110093028 A CN 202110093028A CN 112419385 A CN112419385 A CN 112419385A
- Authority
- CN
- China
- Prior art keywords
- coordinate system
- target
- coordinate
- image
- direction angle
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 45
- 238000001514 detection method Methods 0.000 claims abstract description 47
- 230000015654 memory Effects 0.000 claims description 21
- 239000011159 matrix material Substances 0.000 claims description 18
- 238000004422 calculation algorithm Methods 0.000 claims description 14
- 238000013519 translation Methods 0.000 claims description 9
- 238000003384 imaging method Methods 0.000 claims description 4
- 238000004590 computer program Methods 0.000 claims description 3
- 238000004364 calculation method Methods 0.000 abstract description 6
- 238000012549 training Methods 0.000 description 9
- 238000010586 diagram Methods 0.000 description 5
- 230000003287 optical effect Effects 0.000 description 3
- 238000007781 pre-processing Methods 0.000 description 3
- 238000006243 chemical reaction Methods 0.000 description 2
- 238000002372 labelling Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 239000007787 solid Substances 0.000 description 2
- 241001465754 Metazoa Species 0.000 description 1
- 101100314144 Mus musculus Tnip1 gene Proteins 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000005484 gravity Effects 0.000 description 1
- 238000002156 mixing Methods 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 101150049349 setA gene Proteins 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/50—Depth or shape recovery
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
- G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- General Health & Medical Sciences (AREA)
- Computing Systems (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Biomedical Technology (AREA)
- Molecular Biology (AREA)
- Biophysics (AREA)
- General Engineering & Computer Science (AREA)
- Artificial Intelligence (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Life Sciences & Earth Sciences (AREA)
- Health & Medical Sciences (AREA)
- Image Analysis (AREA)
- Traffic Control Systems (AREA)
Abstract
The invention discloses a method, a device and computer equipment for estimating 3D depth information, wherein the method comprises the following steps: acquiring size information of a 2D detection frame of the target running object and a running direction angle of the target running object, wherein the size information is formed in an image coordinate system of an image where the target running object is located, the running direction angle is formed in a self-vehicle coordinate system of the target running object, and the image coordinate system and the self-vehicle coordinate system are calibrated through a camera coordinate system corresponding to a camera for shooting the target running object; and determining the 3D depth information of the target driving object according to the size information and the driving direction angle of the target driving object. By implementing the method, a 3D target detection depth learning network does not need to be trained, the calculation speed is improved, the image coordinate system and the self-vehicle coordinate system are calibrated through the camera coordinate system, the internal reference and the external reference of the camera are considered in each calibration process, and the problem that the predicted depth information is inaccurate due to the fact that the internal reference of the camera is changed when one camera is replaced is solved.
Description
Technical Field
The invention relates to the technical field of computer vision, in particular to a method and a device for estimating 3D depth information and computer equipment.
Background
As an important direction in computer vision research, the main objective of depth information estimation is to measure the depth of a target object in an image and obtain a depth map at a pixel level. In the related art, the depth information estimation is generally to locate the obtained thermodynamic diagram of the target object through a depth learning network, and obtain the coordinates of the center point of the 3D frame of the target and the length, width and height of the 3D frame. However, the trained depth learning network is usually trained by using fixed camera parameters, such as centeret 3D, the trained model can only be used in a fixed scene, and if a camera is changed to take a picture or record a video, the depth information predicted by using the model is inaccurate, and the applicability of the model is poor.
Disclosure of Invention
Therefore, the technical problem to be solved by the present invention is to overcome the defect in the prior art that a depth learning network model obtained by using fixed camera internal reference training has poor applicability, so as to provide a 3D depth information estimation method, device and computer equipment.
According to a first aspect, the invention discloses a method for estimating 3D depth information, comprising the steps of: acquiring size information of a 2D detection frame of a target running object and a running direction angle of the target running object, wherein the size information is formed in an image coordinate system of an image where the target running object is located, the running direction angle is formed in a self-vehicle coordinate system of the target running object, and the image coordinate system and the self-vehicle coordinate system are calibrated through a camera coordinate system corresponding to a camera for shooting the target running object; and determining the 3D depth information of the target driving object according to the size information and the driving direction angle of the target driving object.
Optionally, the acquiring the driving direction angle of the target driving object includes: acquiring a first speed component and a second speed component of the target running object along a target coordinate axis of the self-vehicle coordinate system, wherein the direction of the target coordinate axis is the same as that of the coordinate axis of the image coordinate system; and determining the driving direction angle of the target driving object according to the first speed component and the second speed component.
Optionally, the method further comprises: acquiring an image to be identified; and inputting the image to be recognized into a preset recognition model for recognition to obtain the size information of the 2D detection frame of the target driving object.
Optionally, the driving direction angle in the image coordinate system is determined by: acquiring a first coordinate of any ground point in the 2D detection frame; converting the first coordinate according to a preset algorithm to obtain a second coordinate corresponding to the coordinate of any ground point in the own vehicle coordinate system; selecting a third coordinate from the own vehicle coordinate system; selecting a fourth coordinate in the own vehicle coordinate system according to the second coordinate, the third coordinate and the driving direction angle of the target driving object; converting the third coordinate and the fourth coordinate according to the preset algorithm to obtain a fifth coordinate of the third coordinate in an image coordinate system and a sixth coordinate of the fourth coordinate in the image coordinate system; and determining the driving direction angle of the target driving object in the image coordinate system according to the first coordinate, the fifth coordinate and the sixth coordinate.
Optionally, the determining the 3D depth information of the target driving object according to the driving direction angle in the image coordinate system and the size information includes:
Wherein,
3D depth information representing a target traveling object;
length information in the size information of the 2D detection frame representing the target traveling object;
width information in the size information of the 2D detection frame representing the target traveling object;
representing the driving direction angle under the image coordinate system; n is a constant, and is determined according to length information in the size information of the 2D detection frame of the target traveling object and 3D length information of the target traveling object.
Optionally, the preset algorithm is:
wherein,
representing a camera coordinate system;
is a coordinate in the image coordinate system;
is a coordinate under a coordinate system of the self-vehicle;
are intrinsic parameters of the camera, in which,
，
indicating how many pixels are represented per length of F on an imaging plane having a focal length of F; the unit of Sx and Sy is pixel/mm; u. of0，v0Representing imagesThe number of pixels in the x direction and the number of pixels in the y direction of the phase difference between the central pixel coordinate and the image origin pixel coordinate;
respectively an external reference rotation matrix and a translation matrix of the camera; and the internal parameters, the external parameter rotation matrix and the translation matrix are obtained by calibration.
According to a second aspect, the present invention also discloses a 3D depth information estimation apparatus, comprising: the device comprises a first acquisition module, a second acquisition module and a third acquisition module, wherein the first acquisition module is used for acquiring the size information of a 2D detection frame of a target running object and the running direction angle of the target running object, the size information is formed in an image coordinate system of an image where the target running object is located, the running direction angle is formed in a self-vehicle coordinate system of the target running object, and the image coordinate system and the self-vehicle coordinate system are calibrated through a camera coordinate system corresponding to a camera for shooting the target running object; and the first determination module is used for determining the 3D depth information of the target running object according to the size information and the running direction angle of the target running object.
Optionally, the obtaining module includes: the first obtaining submodule is used for obtaining a first speed component and a second speed component of the target running object along a target coordinate axis of the self-vehicle coordinate system, and the direction of the target coordinate axis is the same as that of the coordinate axis of the image coordinate system; and the first determining submodule is used for determining the driving direction angle of the target driving object according to the first speed component and the second speed component.
According to a third aspect, the invention also discloses a computer device comprising: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to cause the at least one processor to perform the steps of the 3D depth information estimation method according to the first aspect or any one of the optional embodiments of the first aspect.
According to a fourth aspect, the present invention also discloses a computer readable storage medium having stored thereon a computer program which, when being executed by a processor, carries out the steps of the 3D depth information estimation method according to the first aspect or any one of the optional embodiments of the first aspect.
The technical scheme of the invention has the following advantages:
according to the 3D depth information estimation method and device, the size information of the 2D detection frame of the target running object and the running direction angle of the target running object are obtained, the size information is formed in the image coordinate system of the image where the target running object is located, the running direction angle is formed in the own vehicle coordinate system of the target running object, and the image coordinate system and the own vehicle coordinate system are calibrated through the camera coordinate system corresponding to the camera for shooting the target running object; and determining the 3D depth information of the target driving object according to the size information and the driving direction angle of the target driving object. According to the method, the 3D depth information is determined through the size information of the 2D detection frame of the target running object in the image coordinate system and the running direction angle of the target running object in the self-vehicle coordinate system, the 3D target detection depth learning network does not need to be trained, the calculation speed is improved, the image coordinate system and the self-vehicle coordinate system are calibrated through the camera coordinate system, the internal reference and the external reference of the camera are considered in each calibration process, and the problem that the predicted depth information is inaccurate due to the fact that the internal reference of the camera changes when one camera is changed is solved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts.
Fig. 1 is a flowchart of a specific example of a 3D depth information estimation method in an embodiment of the present invention;
FIG. 2(a) is a diagram illustrating an exemplary 3D detection box according to an embodiment of the present invention;
FIG. 2(b) is a diagram illustrating an exemplary embodiment of a 2D detection frame surrounding a 3D detection frame according to the present invention;
FIG. 3 is a view showing a specific example of a traveling direction angle of a target traveling object according to an embodiment of the present invention;
fig. 4 is a schematic block diagram of a specific example of a 3D depth information estimating apparatus in the embodiment of the present invention;
FIG. 5 is a diagram of an exemplary computer device according to an embodiment of the present invention.
Detailed Description
The technical solutions of the present invention will be described clearly and completely with reference to the accompanying drawings, and it should be understood that the described embodiments are some, but not all embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicate orientations or positional relationships based on the orientations or positional relationships shown in the drawings, and are only for convenience of description and simplicity of description, but do not indicate or imply that the device or element being referred to must have a particular orientation, be constructed and operated in a particular orientation, and thus, should not be construed as limiting the present invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance.
In the description of the present invention, it should be noted that, unless otherwise explicitly specified or limited, the terms "mounted," "connected," and "connected" are to be construed broadly, e.g., as meaning either a fixed connection, a removable connection, or an integral connection; can be mechanically or electrically connected; the two elements may be directly connected or indirectly connected through an intermediate medium, or may be communicated with each other inside the two elements, or may be wirelessly connected or wired connected. The specific meanings of the above terms in the present invention can be understood in specific cases to those skilled in the art.
In addition, the technical features involved in the different embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.
The embodiment of the invention discloses a 3D depth information estimation method, which comprises the following steps as shown in figure 1:
s11: the method comprises the steps of obtaining size information of a 2D detection frame of a target running object and a running direction angle of the target running object, wherein the size information is formed in an image coordinate system of an image where the target running object is located, the running direction angle is formed in a self-vehicle coordinate system of the target running object, and the image coordinate system and the self-vehicle coordinate system are calibrated through a camera coordinate system corresponding to a camera for shooting the target running object.
For example, the target traveling object may include: pedestrians, animals, motor vehicles, non-motor vehicles, and the like. The target traveling object is not particularly limited in the embodiment of the present invention, and may be determined by a person skilled in the art according to actual conditions. The 2D detection frame is a rectangular frame, and the size information of the 2D detection frame may include: length information and width information of the 2D detection frame. The image of the target running object is shot and acquired by the vehicle-mounted monocular camera, so that the cost is low.
The size information of the 2D detection frame and the traveling direction angle of the target traveling object may be acquired from the memory through a wired network or a wireless network. The method for acquiring the size information of the 2D detection frame and the driving direction angle of the target driving object in the embodiment of the present invention is not particularly limited, and may be determined by a person skilled in the art according to actual conditions. In the embodiment of the invention, the camera calibration refers to the association of the image coordinate system and the own vehicle coordinate system. The image coordinate system may be established by using any vertex of the image as an origin and 2 sides drawn from the vertex as X and Y axes, respectively. The camera coordinate system is established in a direction in which the optical center of the camera is taken as an origin point, the optical axis of the camera is taken as a Z axis, and the X axis and the Y axis are consistent with the directions of the X axis and the Y axis of the image coordinate system; the self-vehicle coordinate system is established in a direction with the gravity center of the vehicle as an origin and the X axis and the Y axis consistent with the directions of the X axis and the Y axis of the camera coordinate system.
During calibration, the coordinate system of the self-vehicle and the coordinate system of the camera can be calibrated according to external parameters (a rotation matrix and a translation matrix) of the camera, and the coordinate system of the camera and the coordinate system of an image can be calibrated according to internal parameters of the camera.
S12: and determining the 3D depth information of the target driving object according to the size information and the driving direction angle of the target driving object.
For example, as shown in fig. 2(a) and 2(b), in the triangle abc, determining the 3D depth information of the target traveling object from the traveling direction angle in the image coordinate system and the size information may be determined from the length information in the size information and the traveling direction angle in the image coordinate system, that is:
or according to the width information in the size information and the driving direction angle in the image coordinate system, namely:
wherein,
3D depth information representing a target traveling object;
length information in the size information of the 2D detection frame representing the target traveling object;
width information in the size information of the 2D detection frame representing the target traveling object;
representing the image in a coordinate systemThe driving direction angle of (1); n is a constant, and is determined according to length information in the size information of the 2D detection frame of the target traveling object and 3D length information of the target traveling object.
The 3D depth information estimation method provided by the invention comprises the steps of obtaining the size information of a 2D detection frame of a target running object and the running direction angle of the target running object, wherein the size information is formed in an image coordinate system of an image where the target running object is located, the running direction angle is formed in a vehicle coordinate system of the target running object, and the image coordinate system and the vehicle coordinate system are calibrated through a camera coordinate system corresponding to a camera for shooting the target running object; and determining the 3D depth information of the target driving object according to the size information and the driving direction angle of the target driving object. According to the method, the 3D depth information is determined through the size information of the 2D detection frame of the target running object in the image coordinate system and the running direction angle of the target running object in the self-vehicle coordinate system, the 3D target detection depth learning network does not need to be trained, the calculation speed is improved, the image coordinate system and the self-vehicle coordinate system are calibrated through the camera coordinate system, the internal reference and the external reference of the camera are considered in each calibration process, and the problem that the predicted depth information is inaccurate due to the fact that the internal reference of the camera changes when one camera is changed is solved.
As an optional implementation manner of the embodiment of the present invention, the step S11 includes:
first, a first speed component and a second speed component of a target running object along a target coordinate axis of a self-vehicle coordinate system are obtained, and the direction of the target coordinate axis is the same as that of the coordinate axis of an image coordinate system. The first velocity component and the second velocity component may be acquired by an in-vehicle millimeter wave radar. The cost is greatly reduced compared with the use of laser radar.
Specifically, the vehicle-mounted millimeter wave radar transmits a column of continuous frequency modulation millimeter waves outwards through an antenna and receives a reflected signal of a target, the frequency of the transmitted wave changes along with the time according to the rule of modulation voltage, and a general modulation signal is a triangular wave signal. The reflected wave and the transmitted wave have the same shape, but have a delay in time, the frequency difference between the transmitted signal and the reflected signal at a certain moment is the intermediate frequency signal frequency of the mixing output, and the target distance is in direct proportion to the intermediate frequency signal frequency. If the reflected signal is from a relatively moving object, the reflected signal includes a doppler shift caused by the relative motion of the object. The distance and relative motion velocity can be calculated according to the doppler principle.
Next, a travel direction angle of the target travel object is determined from the first velocity component and the second velocity component.
Illustratively, as shown in fig. 2, X1 and Y1 represent the horizontal axis and the vertical axis of the camera coordinate system, respectively, X2 and Y2 represent the horizontal axis and the vertical axis of the vehicle coordinate system, respectively, and the ray P forms an angle with the horizontal axis
Namely, the driving direction angle of the target driving object.
Determining the driving direction angle of the target driving object according to the first velocity component and the second velocity component may specifically be:
wherein,
is a first velocity component;
representing the second velocity component.
As an optional implementation manner of the embodiment of the present invention, the 3D depth information estimation method further includes:
first, an image to be recognized is acquired.
Illustratively, the image to be recognized is captured by a vehicle-mounted camera. The image to be recognized can be directly acquired from the vehicle-mounted camera through a protocol, and the image stored in advance can be called from a memory. The method for acquiring the image to be identified is not particularly limited in the embodiment of the present invention, and may be determined by a person skilled in the art according to actual conditions.
And secondly, inputting the image to be recognized into a preset recognition model for recognition to obtain the size information of the 2D detection frame of the target driving object.
Illustratively, the preset recognition model is trained in advance, and the specific training process is as follows:
the method comprises the steps of obtaining a plurality of training images, preprocessing and data labeling the training images, inputting the preprocessed training images into an initial recognition model for training, and taking a model meeting preset conditions as a preset recognition model.
The training image may be a real image captured by a camera (for example, a vehicle-mounted camera, a roadside camera, or the like) or an artificially synthesized image. The pre-processing may include: and (5) rotating, zooming and blurring the image. The data annotation can be manual annotation or automatic annotation. The preset condition may be that the depth information estimation accuracy is greater than 95%. The training image, the preprocessing content and the data labeling method are not particularly limited in the embodiment of the invention, and can be determined by a person skilled in the art according to actual conditions.
And after the training is finished, loading the weight of the trained preset recognition model, inputting the image to be recognized, extracting the features through the preset recognition model, classifying and positioning the extracted features, and finally obtaining the category of the target and the coordinate position in the image.
As an alternative implementation manner of the embodiment of the present invention, the driving direction angle in the image coordinate system is determined by the following steps:
first, first coordinates of any ground point in the 2D detection frame are acquired.
For example, a point on the side of the 2D detection frame of the image may not be a ground point due to the driving posture of the target driving object. Therefore, in the embodiment of the present invention, the image coordinates of the lower right vertex A of the image are set
As a first coordinate, this coordinate may be acquired from the 2D detection frame and the image coordinate system.
And secondly, converting the first coordinate according to a preset algorithm to obtain a second coordinate corresponding to the coordinate of any ground point in the own vehicle coordinate system.
Illustratively, the preset algorithm is:
wherein,
representing a camera coordinate system;
is a coordinate in the image coordinate system;
is a coordinate under a coordinate system of the self-vehicle;
are intrinsic parameters of the camera, in which,
，
indicating how many pixels are represented per length of F on an imaging plane having a focal length of F; the unit of Sx and Sy is pixel/mm; u. of0，v0An x-direction pixel number and a y-direction pixel number representing a phase difference between a central pixel coordinate of the image and an origin pixel coordinate of the image;
respectively an external reference rotation matrix and a translation matrix of the camera; and the internal parameters, the external parameter rotation matrix and the translation matrix are obtained by calibration.
Coordinate of the image according to the formula corresponding to the preset algorithm
Converting the coordinate into the coordinate of the self vehicle to obtain a second coordinate
。
And thirdly, selecting a third coordinate in the own vehicle coordinate system.
For example, the third coordinate may be arbitrarily selected, and in order to simplify the calculation process, the third coordinate may be selected as a point where the X-axis coordinate and the Z-axis coordinate are the same as the second coordinate and the Y-axis coordinate is different from the second coordinate, or may be selected as a point where the Y-axis coordinate and the Z-axis coordinate are the same as the second coordinate and the X-axis coordinate is different from the second coordinate. The selection of the third coordinate is not particularly limited in the embodiment of the present invention, and may be determined by a person skilled in the art according to actual situations. In the embodiment of the invention, the coordinates are selected as
Point B as a third coordinate.
And thirdly, selecting a fourth coordinate in the own vehicle coordinate system according to the second coordinate, the third coordinate and the driving direction angle of the target driving object.
Illustratively, the travel direction angle in the own vehicle coordinate system provided in conjunction with the millimeter wave radar
We can get a point B and make an angle with the straight line AB
In order to simplify the calculation, in the embodiment of the present invention, the X coordinate of the point C is the midpoint of the two points AB, according to the known driving direction angle
If the Y-axis coordinate of the point C can be calculated, the fourth coordinate is
And obtaining the coordinates of the ABC three points in the coordinate system of the bicycle.
And thirdly, converting the third coordinate and the fourth coordinate according to a preset algorithm to obtain a fifth coordinate of the third coordinate in the image coordinate system and a sixth coordinate of the fourth coordinate in the image coordinate system. Converting coordinates of the B, C two points in the vehicle coordinate system into an image coordinate system according to a formula corresponding to the preset algorithm to obtain a fifth coordinate B in the image coordinate system
And a sixth coordinate C
。
And thirdly, determining the driving direction angle of the target driving object in the image coordinate system according to the first coordinate, the fifth coordinate and the sixth coordinate. According to the image coordinates and the cosine theorem of the ABC three points, the pixel coordinate system can be calculated
And (4) an angle.
The embodiment of the present invention further discloses a device for estimating 3D depth information, as shown in fig. 4, including:
a first obtaining module 21, configured to obtain size information of a 2D detection frame of the target traveling object and a traveling direction angle of the target traveling object, where the size information is formed in an image coordinate system of an image in which the target traveling object is located, the traveling direction angle is formed in a vehicle coordinate system of the target traveling object, and the image coordinate system and the vehicle coordinate system are both calibrated by a camera coordinate system corresponding to a camera that captures the target traveling object; the specific implementation manner is described in the above embodiment in relation to step S11, and is not described herein again.
And the first determining module 22 is used for determining the 3D depth information of the target running object according to the size information and the running direction angle of the target running object. The specific implementation manner is described in the above embodiment in relation to step S12, and is not described herein again.
The 3D depth information estimation device provided by the invention acquires the size information of the 2D detection frame of the target running object and the running direction angle of the target running object, wherein the size information is formed in an image coordinate system of an image where the target running object is located, the running direction angle is formed in a vehicle coordinate system of the target running object, and the image coordinate system and the vehicle coordinate system are calibrated through a camera coordinate system corresponding to a camera for shooting the target running object; and determining the 3D depth information of the target driving object according to the size information and the driving direction angle of the target driving object. According to the method, the 3D depth information is determined through the size information of the 2D detection frame of the target running object in the image coordinate system and the running direction angle of the target running object in the self-vehicle coordinate system, the 3D target detection depth learning network does not need to be trained, the calculation speed is improved, the image coordinate system and the self-vehicle coordinate system are calibrated through the camera coordinate system, the internal reference and the external reference of the camera are considered in each calibration process, and the problem that the predicted depth information is inaccurate due to the fact that the internal reference of the camera changes when one camera is changed is solved.
As an optional implementation manner of the embodiment of the present invention, the obtaining module 21 includes:
the first obtaining submodule is used for obtaining a first speed component and a second speed component of the target running object along a target coordinate axis of the self-vehicle coordinate system, and the direction of the target coordinate axis is the same as that of the coordinate axis of the image coordinate system; the specific implementation manner is described in the relevant description of the corresponding steps in the above embodiments, and is not described herein again.
And the first determining submodule is used for determining the driving direction angle of the target driving object according to the first speed component and the second speed component. The specific implementation manner is described in the relevant description of the corresponding steps in the above embodiments, and is not described herein again.
As an optional implementation manner of the embodiment of the present invention, the 3D depth information estimation apparatus further includes:
the second acquisition module is used for acquiring an image to be identified; the specific implementation manner is described in the relevant description of the corresponding steps in the above embodiments, and is not described herein again.
And the recognition module is used for inputting the image to be recognized into a preset recognition model for recognition to obtain the size information of the 2D detection frame of the target driving object. The specific implementation manner is described in the relevant description of the corresponding steps in the above embodiments, and is not described herein again.
As an optional implementation manner of the embodiment of the present invention, the step determining module 22 includes:
the third acquisition module is used for acquiring a first coordinate of any ground point in the 2D detection frame; the specific implementation manner is described in the relevant description of the corresponding steps in the above embodiments, and is not described herein again.
The first conversion module is used for converting the first coordinate according to a preset algorithm to obtain a second coordinate corresponding to the coordinate of any ground point in the self-vehicle coordinate system; the specific implementation manner is described in the relevant description of the corresponding steps in the above embodiments, and is not described herein again.
The first selection module is used for selecting a third coordinate from the own vehicle coordinate system; the specific implementation manner is described in the relevant description of the corresponding steps in the above embodiments, and is not described herein again.
The second selection module is used for selecting a fourth coordinate from the own vehicle coordinate system according to the second coordinate, the third coordinate and the driving direction angle of the target driving object; the specific implementation manner is described in the relevant description of the corresponding steps in the above embodiments, and is not described herein again.
The second conversion module is used for converting the third coordinate and the fourth coordinate according to a preset algorithm to obtain a fifth coordinate of the third coordinate in the image coordinate system and a sixth coordinate of the fourth coordinate in the image coordinate system; the specific implementation manner is described in the relevant description of the corresponding steps in the above embodiments, and is not described herein again.
The second determining module is used for determining the driving direction angle of the target driving object in the image coordinate system according to the first coordinate, the fifth coordinate and the sixth coordinate; the specific implementation manner is described in the relevant description of the corresponding steps in the above embodiments, and is not described herein again.
As an optional implementation manner of the embodiment of the present invention, the first determining module 22 includes:
Wherein,
3D depth information representing a target traveling object;
length information in the size information of the 2D detection frame representing the target traveling object;
width information in the size information of the 2D detection frame representing the target traveling object;
representing the driving direction angle under the image coordinate system; n is a constant, and is determined according to length information in the size information of the 2D detection frame of the target traveling object and 3D length information of the target traveling object. The specific implementation manner is described in the relevant description of the corresponding steps in the above embodiments, and is not described herein again.
As an optional implementation manner of the embodiment of the present invention, the preset algorithm is:
wherein,
representing a camera coordinate system;
is sitting under the image coordinate systemMarking;
is a coordinate under a coordinate system of the self-vehicle;
are intrinsic parameters of the camera, in which,
，
indicating how many pixels are represented per length of F on an imaging plane having a focal length of F; the unit of Sx and Sy is pixel/mm; u. of0，v0An x-direction pixel number and a y-direction pixel number representing a phase difference between a central pixel coordinate of the image and an origin pixel coordinate of the image;
respectively an external reference rotation matrix and a translation matrix of the camera; and the internal parameters, the external parameter rotation matrix and the translation matrix are obtained by calibration. The specific implementation manner is described in the relevant description of the corresponding steps in the above embodiments, and is not described herein again.
An embodiment of the present invention further provides a computer device, as shown in fig. 5, the computer device may include a processor 31 and a memory 32, where the processor 31 and the memory 32 may be connected by a bus or in another manner, and fig. 5 takes the example of connection by a bus as an example.
The processor 31 may be a Central Processing Unit (CPU). The Processor 31 may also be other general purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components, or combinations thereof.
The memory 32, which is a non-transitory computer-readable storage medium, may be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as program instructions/modules corresponding to the 3D depth information estimation method in the embodiment of the present invention (for example, the first obtaining module 21 and the first determining module 22 shown in fig. 4). The processor 31 executes various functional applications and data processing of the processor by running non-transitory software programs, instructions and modules stored in the memory 32, that is, implements the 3D depth information estimation method in the above method embodiment.
The memory 32 may include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created by the processor 31, and the like. Further, the memory 32 may include high speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid state storage device. In some embodiments, the memory 32 may optionally include memory located remotely from the processor 31, and these remote memories may be connected to the processor 31 via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The one or more modules are stored in the memory 32 and, when executed by the processor 31, perform a 3D depth information estimation method as in the embodiment shown in fig. 1.
The details of the computer device can be understood with reference to the corresponding related descriptions and effects in the embodiment shown in fig. 1, and are not described herein again.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by a computer program, which can be stored in a computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. The storage medium may be a magnetic Disk, an optical Disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a Flash Memory (Flash Memory), a Hard Disk (Hard Disk Drive, abbreviated as HDD), a Solid State Drive (SSD), or the like; the storage medium may also comprise a combination of memories of the kind described above.
Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations fall within the scope defined by the appended claims.
Claims (10)
1. A method for estimating 3D depth information, comprising the steps of:
acquiring size information of a 2D detection frame of a target running object and a running direction angle of the target running object, wherein the size information is formed in an image coordinate system of an image where the target running object is located, the running direction angle is formed in a self-vehicle coordinate system of the target running object, and the image coordinate system and the self-vehicle coordinate system are calibrated through a camera coordinate system corresponding to a camera for shooting the target running object;
and determining the 3D depth information of the target driving object according to the size information and the driving direction angle of the target driving object.
2. The method according to claim 1, wherein the acquiring of the traveling direction angle of the target traveling object includes:
acquiring a first speed component and a second speed component of the target running object along a target coordinate axis of the self-vehicle coordinate system, wherein the direction of the target coordinate axis is the same as that of the coordinate axis of the image coordinate system;
and determining the driving direction angle of the target driving object according to the first speed component and the second speed component.
3. The method of claim 1, further comprising:
acquiring an image to be identified;
and inputting the image to be recognized into a preset recognition model for recognition to obtain the size information of the 2D detection frame of the target driving object.
4. The method according to claim 1, characterized in that the direction of travel angle in the image coordinate system is determined by:
acquiring a first coordinate of any ground point in the 2D detection frame;
converting the first coordinate according to a preset algorithm to obtain a second coordinate corresponding to the coordinate of any ground point in the own vehicle coordinate system;
selecting a third coordinate from the own vehicle coordinate system;
selecting a fourth coordinate in the own vehicle coordinate system according to the second coordinate, the third coordinate and the driving direction angle of the target driving object;
converting the third coordinate and the fourth coordinate according to the preset algorithm to obtain a fifth coordinate of the third coordinate in an image coordinate system and a sixth coordinate of the fourth coordinate in the image coordinate system;
and determining the driving direction angle of the target driving object in the image coordinate system according to the first coordinate, the fifth coordinate and the sixth coordinate.
5. The method according to claim 4, wherein the determining 3D depth information of the target traveling object from the traveling direction angle in the image coordinate system and the size information comprises:
Wherein,
3D depth information representing a target traveling object;
length information in the size information of the 2D detection frame representing the target traveling object;
width information in the size information of the 2D detection frame representing the target traveling object;
representing the driving direction angle under the image coordinate system; n is a constant, and is determined according to length information in the size information of the 2D detection frame of the target traveling object and 3D length information of the target traveling object.
6. The method of claim 4, wherein the predetermined algorithm is:
wherein,
representing a camera coordinate system;
is a coordinate in the image coordinate system;
is a coordinate under a coordinate system of the self-vehicle;
are intrinsic parameters of the camera, in which,
=F*Sx，
= F × Sy, representing how many pixels are represented per length of F on an imaging plane with focal length F; the unit of Sx and Sy is pixel/mm; u. of0，v0An x-direction pixel number and a y-direction pixel number representing a phase difference between a central pixel coordinate of the image and an origin pixel coordinate of the image;
respectively an external reference rotation matrix and a translation matrix of the camera; and the internal parameters, the external parameter rotation matrix and the translation matrix are obtained by calibration.
7. A 3D depth information estimation apparatus, characterized by comprising:
the device comprises a first acquisition module, a second acquisition module and a third acquisition module, wherein the first acquisition module is used for acquiring the size information of a 2D detection frame of a target running object and the running direction angle of the target running object, the size information is formed in an image coordinate system of an image where the target running object is located, the running direction angle is formed in a self-vehicle coordinate system of the target running object, and the image coordinate system and the self-vehicle coordinate system are calibrated through a camera coordinate system corresponding to a camera for shooting the target running object;
and the first determination module is used for determining the 3D depth information of the target running object according to the size information and the running direction angle of the target running object.
8. The apparatus of claim 7, wherein the obtaining module comprises:
the first obtaining submodule is used for obtaining a first speed component and a second speed component of the target running object along a target coordinate axis of the self-vehicle coordinate system, and the direction of the target coordinate axis is the same as that of the coordinate axis of the image coordinate system;
and the first determining submodule is used for determining the driving direction angle of the target driving object according to the first speed component and the second speed component.
9. A computer device, comprising: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to cause the at least one processor to perform the steps of the 3D depth information estimation method of any one of claims 1-6.
10. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the 3D depth information estimation method according to any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110093028.4A CN112419385B (en) | 2021-01-25 | 2021-01-25 | 3D depth information estimation method and device and computer equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110093028.4A CN112419385B (en) | 2021-01-25 | 2021-01-25 | 3D depth information estimation method and device and computer equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112419385A true CN112419385A (en) | 2021-02-26 |
| CN112419385B CN112419385B (en) | 2021-04-09 |
Family
ID=74782574
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110093028.4A Active CN112419385B (en) | 2021-01-25 | 2021-01-25 | 3D depth information estimation method and device and computer equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112419385B (en) |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113075716A (en) * | 2021-03-19 | 2021-07-06 | 地平线(上海)人工智能技术有限公司 | Image-based vehicle positioning method and device, storage medium and electronic equipment |
| CN113344998A (en) * | 2021-06-25 | 2021-09-03 | 北京市商汤科技开发有限公司 | Depth detection method and device, computer equipment and storage medium |
| CN114266859A (en) * | 2021-12-02 | 2022-04-01 | 国汽智控(北京)科技有限公司 | Data processing method, device, equipment and storage medium |
| CN114814758A (en) * | 2022-06-24 | 2022-07-29 | 国汽智控(北京)科技有限公司 | Camera-millimeter wave radar-laser radar combined calibration method and device |
| CN114882115A (en) * | 2022-06-10 | 2022-08-09 | 国汽智控(北京)科技有限公司 | Vehicle pose prediction method and device, electronic equipment and storage medium |
| WO2022213729A1 (en) * | 2021-04-07 | 2022-10-13 | 地平线征程(杭州)人工智能科技有限公司 | Method and apparatus for detecting motion information of target, and device and medium |
| CN115205365A (en) * | 2022-07-14 | 2022-10-18 | 小米汽车科技有限公司 | Vehicle distance detection method and device, vehicle, readable storage medium and chip |
| CN115965942A (en) * | 2023-03-03 | 2023-04-14 | 安徽蔚来智驾科技有限公司 | Position estimation method, vehicle control method, device, medium, and vehicle |
| CN115861741B (en) * | 2023-03-01 | 2023-10-17 | 小米汽车科技有限公司 | Target calibration method and device, electronic equipment, storage medium and vehicle |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101479765A (en) * | 2006-06-23 | 2009-07-08 | 图象公司 | Methods and systems for converting 2D motion pictures for stereoscopic 3D exhibition |
| CN102771110A (en) * | 2010-02-25 | 2012-11-07 | 想象技术有限公司 | Object tracking using graphics engine derived vectors in a motion estimation system |
| US9214025B2 (en) * | 2012-11-30 | 2015-12-15 | Adobe Systems Incorporated | Depth estimation using normalized displacement of image pairs |
| CN108597009A (en) * | 2018-04-10 | 2018-09-28 | 上海工程技术大学 | A method of objective detection is carried out based on direction angle information |
| CN112132908A (en) * | 2020-09-23 | 2020-12-25 | 西安邮电大学 | Camera external parameter calibration method and device based on intelligent detection technology |
-
2021
- 2021-01-25 CN CN202110093028.4A patent/CN112419385B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101479765A (en) * | 2006-06-23 | 2009-07-08 | 图象公司 | Methods and systems for converting 2D motion pictures for stereoscopic 3D exhibition |
| CN102771110A (en) * | 2010-02-25 | 2012-11-07 | 想象技术有限公司 | Object tracking using graphics engine derived vectors in a motion estimation system |
| US9214025B2 (en) * | 2012-11-30 | 2015-12-15 | Adobe Systems Incorporated | Depth estimation using normalized displacement of image pairs |
| CN108597009A (en) * | 2018-04-10 | 2018-09-28 | 上海工程技术大学 | A method of objective detection is carried out based on direction angle information |
| CN112132908A (en) * | 2020-09-23 | 2020-12-25 | 西安邮电大学 | Camera external parameter calibration method and device based on intelligent detection technology |
Cited By (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113075716A (en) * | 2021-03-19 | 2021-07-06 | 地平线(上海)人工智能技术有限公司 | Image-based vehicle positioning method and device, storage medium and electronic equipment |
| WO2022213729A1 (en) * | 2021-04-07 | 2022-10-13 | 地平线征程(杭州)人工智能科技有限公司 | Method and apparatus for detecting motion information of target, and device and medium |
| CN113344998A (en) * | 2021-06-25 | 2021-09-03 | 北京市商汤科技开发有限公司 | Depth detection method and device, computer equipment and storage medium |
| CN113344998B (en) * | 2021-06-25 | 2022-04-29 | 北京市商汤科技开发有限公司 | Depth detection method and device, computer equipment and storage medium |
| CN114266859A (en) * | 2021-12-02 | 2022-04-01 | 国汽智控(北京)科技有限公司 | Data processing method, device, equipment and storage medium |
| CN114882115A (en) * | 2022-06-10 | 2022-08-09 | 国汽智控(北京)科技有限公司 | Vehicle pose prediction method and device, electronic equipment and storage medium |
| CN114882115B (en) * | 2022-06-10 | 2023-08-25 | 国汽智控(北京)科技有限公司 | Vehicle pose prediction method and device, electronic equipment and storage medium |
| CN114814758A (en) * | 2022-06-24 | 2022-07-29 | 国汽智控(北京)科技有限公司 | Camera-millimeter wave radar-laser radar combined calibration method and device |
| CN114814758B (en) * | 2022-06-24 | 2022-09-06 | 国汽智控(北京)科技有限公司 | Camera-millimeter wave radar-laser radar combined calibration method and device |
| CN115205365A (en) * | 2022-07-14 | 2022-10-18 | 小米汽车科技有限公司 | Vehicle distance detection method and device, vehicle, readable storage medium and chip |
| CN115861741B (en) * | 2023-03-01 | 2023-10-17 | 小米汽车科技有限公司 | Target calibration method and device, electronic equipment, storage medium and vehicle |
| CN115965942A (en) * | 2023-03-03 | 2023-04-14 | 安徽蔚来智驾科技有限公司 | Position estimation method, vehicle control method, device, medium, and vehicle |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112419385B (en) | 2021-04-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112419385B (en) | 3D depth information estimation method and device and computer equipment | |
| CA3028653C (en) | Methods and systems for color point cloud generation | |
| US10937231B2 (en) | Systems and methods for updating a high-resolution map based on binocular images | |
| US10964054B2 (en) | Method and device for positioning | |
| US11205284B2 (en) | Vehicle-mounted camera pose estimation method, apparatus, and system, and electronic device | |
| US9197866B2 (en) | Method for monitoring a traffic stream and a traffic monitoring device | |
| US10996072B2 (en) | Systems and methods for updating a high-definition map | |
| JP6678605B2 (en) | Information processing apparatus, information processing method, and information processing program | |
| CN113029128B (en) | Visual navigation method and related device, mobile terminal and storage medium | |
| CN113240813B (en) | Three-dimensional point cloud information determining method and device | |
| CN110378836B (en) | Method, system and equipment for acquiring 3D information of object | |
| JP2011254128A (en) | Plane view generating device and plane view generating method | |
| CN112255604B (en) | Method and device for judging accuracy of radar data and computer equipment | |
| JP6715205B2 (en) | Work machine surrounding image display device | |
| CN114754779B (en) | Positioning and mapping method and device and electronic equipment | |
| CN111243021A (en) | Vehicle-mounted visual positioning method and system based on multiple combined cameras and storage medium | |
| CN116343165A (en) | 3D target detection system, method, terminal equipment and storage medium | |
| WO2022133986A1 (en) | Accuracy estimation method and system | |
| CN115100290A (en) | Monocular vision positioning method, monocular vision positioning device, monocular vision positioning equipment and monocular vision positioning storage medium in traffic scene | |
| AU2018102199A4 (en) | Methods and systems for color point cloud generation | |
| CN113874681B (en) | Evaluation method and system for point cloud map quality | |
| CN113066100A (en) | Target tracking method, device, equipment and storage medium | |
| WO2024139375A1 (en) | Data processing method and computer device | |
| US20240098231A1 (en) | Image processing device, image processing method, and computer-readable medium | |
| EP3608832A1 (en) | Image display apparatus, image display method, and carrier means |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |