CN112990050B - Monocular 3D target detection method based on lightweight characteristic pyramid structure - Google Patents
Monocular 3D target detection method based on lightweight characteristic pyramid structure Download PDFInfo
- Publication number
- CN112990050B CN112990050B CN202110326713.7A CN202110326713A CN112990050B CN 112990050 B CN112990050 B CN 112990050B CN 202110326713 A CN202110326713 A CN 202110326713A CN 112990050 B CN112990050 B CN 112990050B
- Authority
- CN
- China
- Prior art keywords
- sampling
- feature map
- candidate key
- outputting
- time down
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Engineering & Computer Science (AREA)
- Computing Systems (AREA)
- Software Systems (AREA)
- Molecular Biology (AREA)
- Computational Linguistics (AREA)
- Biophysics (AREA)
- Biomedical Technology (AREA)
- Mathematical Physics (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Multimedia (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Computational Biology (AREA)
- Evolutionary Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a monocular 3D target detection method based on a lightweight characteristic pyramid structure, which comprises the following steps: collecting RGB images of a vehicle-mounted camera; inputting the RGB image into a pre-established and trained monocular 3D target detection network, and outputting a target detection result; the monocular 3D object detection network comprises: the system comprises a feature extraction network, a detection head and a post-processing module; the feature extraction network is used for performing down-sampling on the RGB image to extract high-level semantic features, generating 4-time, 8-time and 16-time down-sampling feature maps and inputting the feature maps to the detection head; the detection head is used for generating candidate key point category vectors and candidate key point pixel position index vectors based on the 4-time down-sampling feature map, generating candidate key point 3D regression frame coding vectors based on the 4-time down-sampling feature map, the 8-time down-sampling feature map and the 16-time down-sampling feature map, and outputting the candidate key point category vectors and the 3D regression frame coding vectors to the post-processing module; and the post-processing module is used for decoding the 3D regression frame coding vector and outputting a target detection result by combining the candidate key point category vector.
Description
Technical Field
The invention relates to the technical field of automatic driving, in particular to a monocular 3D target detection method based on a lightweight characteristic pyramid structure.
Background
In an automatic driving system, 3D target detection is a very important task in a perception module, and rear-end prediction, planning, motion control and other modules all depend on reliable detection results of specific category targets around a main vehicle. By means of the advantage that high-line-beam laser radar can accurately model the surrounding environment in centimeter level, the 3D target detection algorithm based on the laser radar has advanced greatly in recent years, but due to the inherent defects that a laser radar sensor has high cost and poor adaptability to severe weather such as rain, snow, fog and the like, the large-scale landing of the laser radar and related algorithms in the field of automatic driving is severely limited. Compared with the laser radar, the vision sensor is low in cost, the capability of adapting to severe weather such as rain and snow is superior to the laser radar, and the requirements of marketization and large-scale batch production are met more easily, so the 3D target detection algorithm based on pure vision gradually receives attention from academic circles and industrial circles, on one hand, the 3D target detection technology based on pure vision can avoid expensive laser radar, the automatic driving solution with low cost is realized, on the other hand, the technology can also be matched with the 3D target detection module based on the laser radar to realize module redundancy, the serious consequences caused by the failure of the laser radar are avoided, and the automatic driving solution with higher safety and reliability is realized.
Due to the double consideration of cost and power consumption, the calculation power of a vehicle-end calculation platform carried by an automatic driving vehicle is relatively limited, and a large-scale complex model cannot be supported, so that the pure visual 3D target detection algorithm for automatic driving application must simultaneously take into account the double indexes of detection precision and efficiency: on the premise that the accuracy index meets the requirement of an actual scene, a faster model reasoning speed is pursued to ensure a more timely response of the sensing system, and an advanced early warning is provided for a rear-end prediction, planning and motion planning module, so that a safer and more reliable automatic driving system is realized.
The anchor-free frame idea is a latest research hotspot in the field of target detection, and the newly proposed pure visual 3D target detection algorithms based on key points (such as CenterNet, SMOKE and RTM3D) meet the requirements of real-time deployment and engineering landing of an automatic driving edge computing platform with higher algorithm efficiency (CenterNet:30ms, SMOKE:30ms and RTM3D:50ms), but the perception requirements under an automatic driving scene cannot be completely met due to low precision indexes.
The subsequent improvement method mainly comprises the following steps: the method has the advantages that a traditional characteristic pyramid structure, a series multistage cascade regression structure, a deep reinforcement learning introduction structure and the like are added to realize the optimization of the detection frame, the accuracy index of the algorithm is effectively improved through the improvement schemes, but due to the fact that structural branches are additionally introduced into the model, the time delay of model reasoning is greatly increased, and the real-time deployment of the algorithm on an automatic driving edge computing platform is influenced; therefore, on the premise of ensuring that the efficiency of the existing method is not reduced, the method for greatly improving the precision index of the method has huge actual engineering value.
Disclosure of Invention
The invention aims to overcome the technical defects, provides a lightweight characteristic pyramid structure and an attention loss function, can be applied to most of the existing monocular 3D target detection methods based on key points, and can simultaneously improve the two aspects of detection precision and efficiency of the existing methods.
In order to achieve the above object, embodiment 1 of the present invention provides a monocular 3D object detection method based on a lightweight feature pyramid structure, where the method includes:
collecting RGB images of a vehicle-mounted camera;
inputting the RGB image into a pre-established and trained monocular 3D target detection network, and outputting a target detection result; the monocular 3D object detection network comprises: the system comprises a feature extraction network, a detection head and a post-processing module;
the feature extraction network is used for performing down-sampling on the RGB image to extract high-level semantic features, generating a 4-time down-sampling feature map and inputting the feature map to the detection head;
the detection head is used for generating candidate key point category vectors and candidate key point pixel position index vectors based on the 4-time down-sampling feature map, generating 3D regression frame coding vectors corresponding to the candidate key points based on the 4-time down-sampling feature map, the 8-time down-sampling feature map and the 16-time down-sampling feature map, and outputting the candidate key point category vectors and the 3D regression frame coding vectors to the post-processing module;
and the post-processing module is used for decoding the 3D regression frame coding vector and outputting a target detection result by combining the candidate key point category vector.
As an improvement of the above method, the feature extraction network comprises an encoder and a decoder;
the encoder is used for performing down-sampling on the input RGB image to extract high-level semantic features and outputting a 32-time down-sampling feature map;
the decoder is used for up-sampling the high-level semantic feature map output by the encoder to obtain a 4-time down-sampling feature map required by the detection head; the decoder includes three deconvolution layers: a first deconvolution layer, a second deconvolution layer, and a third deconvolution layer; a first deconvolution layer for processing the 32-fold down-sampling feature map output by the encoder and outputting a 16-fold down-sampling feature map; a second deconvolution layer for processing the 16-fold down-sampling feature map and outputting an 8-fold down-sampling feature map; and a third deconvolution layer for processing the 8-fold down-sampling feature map and outputting the 4-fold down-sampling feature map to the detection head.
As an improvement of the above method, the detection head comprises a thermodynamic diagram branch and a parameter regression branch;
the thermodynamic diagram branch is used for generating a thermodynamic diagram for target key points based on a 4-time down-sampling feature map, and finally outputting candidate key point category vectors and pixel position index vectors of the candidate key points corresponding to the 16-time down-sampling feature map, the 8-time down-sampling feature map and the 4-time down-sampling feature map by arranging all confidence values in a descending order and screening the positions with the maximum K previous confidence values as candidate key points;
the parameter regression branch introduces a lightweight feature pyramid structure for respectively taking values from corresponding feature maps according to three position indexes, and then merging the values to extract a target 3D regression frame coding vectorAnd outputting the data to a post-processing module, wherein K represents the number of the detection targets, and R represents the number of regression parameters.
As an improvement of the above method, the thermodynamic diagram branch comprises: a first convolution layer, a second convolution layer and a TopK operation unit;
the first convolution layer is used for further extracting the features of the 4-time down-sampling feature map and outputting the feature map to the second convolution layer;
the second convolution layer is used for performing convolution processing on the characteristic diagram and outputting a thermodynamic diagram, wherein any element y on the thermodynamic diagram is outputijcRepresenting a probability value that at the (i, j) pixel position of the thermodynamic diagram, a target keypoint of category c exists;
the TopK operation unit is used for arranging all probability values on the thermodynamic diagram in a descending order, taking the first K candidate points with the maximum probability values as candidate key points, converting pixel coordinates (i, j) into position indexes Index, converting channel indexes c representing categories into categories, splicing the K category values into candidate key point category vectors Classes, and outputting the candidate key point category vectors Classes to the post-processing module; and splicing and generating candidate key point pixel position Index vectors indexes according to the K position Index indexes: acquiring pixel position indexes of the candidate key points corresponding to 4 times of downsampling feature maps; the pixel position indexes 1/2 indexes of the candidate key points corresponding to the 8-time down-sampling feature map and the pixel position indexes 1/4 indexes of the 16-time down-sampling feature map can be further obtained through division operation, the three position indexes are one-dimensional vectors with the same number of elements, and three values of the three one-dimensional vectors at the same position respectively correspond to pixel position indexes of the same candidate key points at the 4-time down-sampling feature map, the 8-time down-sampling feature map and the 16-time down-sampling feature map.
As an improvement of the above method, the parametric regression branch comprises: a third convolutional layer, three parallel sampling units, a splicing unit and a 1x1 convolutional layer; the three parallel sampling units comprise: the device comprises a first sampling unit, a second sampling unit and a third sampling unit;
the third convolution layer is used for further extracting the characteristics of the 4 times down-sampling characteristic diagram and outputting the characteristic diagram to the first sampling unit;
the first sampling unit is used for sampling values from the feature map output by the third convolutional layer according to pixel position indexes of the candidate key points on the feature map sampled by 4 times;
the second sampling unit is used for sampling values from the 8-time down-sampling feature map output by the second deconvolution layer according to the pixel position index 1/2 indexes of the candidate key points on the 8-time down-sampling feature map;
the third sampling unit is used for sampling values from the 16-time down-sampling feature map output by the first deconvolution layer according to the pixel position index 1/4 indexes of the candidate key points on the 16-time down-sampling feature map;
the splicing unit is used for merging the values output by the three sampling units to realize the feature reading, alignment and fusion of the candidate target key points on the feature maps with different resolutions; outputting the fused features to a 1x1 convolutional layer;
the 1x1 convolutional layer is used for obtaining a 3D regression frame coding vector from the fusion featuresAnd outputting the data to a post-processing module.
As an improvement of the above method, the method further comprises: the method for training the monocular 3D target detection network specifically comprises the following steps:
in the parameter regression branch, the function L of attention loss is establishedreg:
In the formula (I), the compound is shown in the specification,regression loss for the ith target; n is the target number in a training batch; w is aiWeighting coefficients of the ith target regression loss in the total regression loss;
each target loss weight w in the attention loss functioniThe definition is as follows:
in the formula, PiA category confidence for the ith target;regression loss for the ith target;three-dimensional intersection ratio between the ith target prediction frame and the true value frame is obtained; beta is an equilibrium parameter.
Embodiment 2 of the present invention provides a terminal device, including: comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the above-mentioned method when executing the computer program.
Embodiment 3 of the present invention proposes a storage medium including: comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the above-mentioned method when executing the computer program.
The invention has the advantages that:
1. the method provided by the invention provides a lightweight characteristic pyramid structure, is suitable for most monocular 3D target detection methods based on key points, effectively overcomes the defects that the traditional characteristic pyramid structure influences algorithm efficiency, increases non-maximum values to inhibit post-processing and the like, and can further shorten model reasoning time delay under the condition of effectively improving algorithm accuracy indexes;
2. the method provides an attention loss function, is suitable for a large part of target detection methods, and effectively improves the precision index of the algorithm from the perspective of optimizing the training process on the premise of not influencing the model inference time delay by solving the problem of mismatching between the model output class confidence coefficient and the position precision of a detection frame (one of the reasons influencing the precision).
Drawings
FIG. 1 is a lightweight feature pyramid structure applied to a monocular 3D object detection method of the present invention;
fig. 2 is a graph of attention loss function weight coefficients.
Detailed Description
The technical solution of the present invention will be described in detail below with reference to the accompanying drawings.
The embodiment 1 of the invention provides a monocular 3D target detection method based on a lightweight characteristic pyramid structure, which comprises the following steps:
step 1) collecting vehicle-mounted camera image RGB data;
step 2) establishing a monocular 3D target detection network based on key points;
as shown in fig. 1, the network includes: a feature extraction network (backhaul), a detection head (detective head) and a post-processing module (PostProcess);
the Chinese and English symbols in FIG. 1 are shown in Table 1:
TABLE 1
English sign | Chinese interpretation | English sign | Chinese interpretation |
Backbone | Feature extraction network | Conv | Convolution with a bit line |
Detection Head | Detection head | Conv1x1 | 1x1 convolution |
Post Process | Post-treatment | Sampling | Sampling by index |
Encoder | Encoder for encoding a video signal | TopK | K maximum values before retrieval |
Decoder | Decoder | Keypoint | Key points |
H | Height of input image | Index | Index |
W | |
1/ |
1/2 times index |
D | Number of |
1/ |
1/4 times index |
Heatmap | Thermodynamic diagram | Class | Class vector |
Regression | Regression | K | Numerical variablesName (name) |
Light-FPN | Lightweight feature pyramid | Concat | Merging |
C | Number of detection categories | Decode | Decoding |
Deconv | |
3D Boxes | Three-dimensional bounding box |
Resnet-34 | Deep residual error network | Results | The result of the detection |
The feature extraction network includes an Encoder (Encoder) and a Decoder (Decoder). The encoder structure can select basic networks such as Resnet, DLA-34, Hourglass-101 and the like to carry out down-sampling on an input image to extract high-level semantic features, and the decoder is used for carrying out up-sampling on a high-level semantic feature image output by the encoder to obtain a 4-time down-sampling feature image required by the detection head; includes three deconvolution layers: a first deconvolution layer, a second deconvolution layer, and a third deconvolution layer; a first deconvolution layer for processing the 32-fold down-sampling feature map output by the encoder and outputting a 16-fold down-sampling feature map; a second deconvolution layer for processing the 16-fold down-sampling feature map and outputting an 8-fold down-sampling feature map; and a third deconvolution layer for processing the 8-fold down-sampling feature map and outputting a 4-fold down-sampling feature map.
The full convolution structure detection head includes a thermodynamic diagram branch (Heatmap) and a parametric Regression branch (Regression).
The thermodynamic diagram branch is used for predicting a target key point and outputting a key point thermodynamic diagram Y' ═ 0,1]W/4×H/4×CRepresenting the probability that the target keypoint is detected at each pixel position (the channel dimension C is responsible for the class prediction).
The thermodynamic diagram branch comprises: a first convolution layer, a second convolution layer and a TopK operation unit;
the first convolution layer is used for further extracting the features of the 4-time down-sampling feature map and outputting the feature map to the second convolution layer;
a second convolution layer for performing convolution processing on the characteristic diagram to output a thermodynamic diagram Y' for any element Y on the thermodynamic diagram Yijc: representing the probability that at the (j) pixel location of the thermodynamic diagram, there is a candidate keypoint of category c;
the TopK operation unit executes the sorting operation, and arranges all the probability values on the thermodynamic diagram in a descending order, and takes the top K values with the maximum probability values, wherein each value has the meaning same as that of the y described aboveijcLikewise, pixel coordinates (i, j) are converted to a location Index, c, which represents a category, is converted to class, and the concatenation of K such values into vectors are Classes and Indices in FIG. 1. And screening pixels with high confidence probability as candidate key points, and obtaining pixel position Index of the candidate key points corresponding to the 4-time down-sampling feature map (relative to the original image). The pixel position Index 1/2Index of the candidate key point corresponding to the 8-time down-sampling feature map and the pixel position Index 1/4Index of the 16-time down-sampling feature map can be further obtained through division operation, the above three pixel position indexes are one-dimensional vectors with the same number of elements, and three values of the three one-dimensional vectors at the same position respectively correspond to pixel position indexes of the same target key point at the 4-time down-sampling feature map, the 8-time down-sampling feature map and the 16-time down-sampling feature map.
The parameter regression branch introduces the lightweight characteristic pyramid structure provided by the invention, and the lightweight characteristic pyramid structure comprises the following steps: a third convolutional layer, three parallel sampling units, a splicing unit and a 1x1 convolutional layer; the three parallel sampling units comprise: a first sampling unit, a second sampling unit and a third sampling unit;
a third convolution layer for further extracting the feature of the 4-fold down-sampling feature map and outputting the feature map to the first sampling unit;
the first sampling unit is used for sampling values from the feature map output by the third convolutional layer according to the position indexes of the key points on the 4-time downsampling feature map;
the second sampling unit is used for sampling values from the 8-time down-sampling feature map output by the second deconvolution layer according to the position indexes 1/2 indexes of the key points on the 8-time down-sampling feature map;
the third sampling unit is used for sampling values from the 16-time down-sampling feature map output by the first deconvolution layer according to the position indexes 1/4 indexes of the key points on the 16-time down-sampling feature map;
the splicing unit is used for merging the outputs of the three sampling units (weighting, attention and other modes can also be adopted) to realize the feature reading, alignment and fusion of the candidate key points on feature maps with different resolutions; outputting the fused features to a 1x1 convolutional layer;
convolution with 1x1 for obtaining 3D regression frame code vector from the fused featuresAnd outputting the data to a post-processing module. (compare 3D frame regression map code vector in standard frameThe regression branch convolution operation is effectively reduced).
3D regression block diagram coding vector output by post-processing module to detection headDecoding is carried out, and a final target detection result is output by combining the candidate key point Class vector Class output by the thermodynamic diagram. K represents the number of detection targets, and R represents the number of regression parameters.
The regression parameters for each target under test can be represented as an 8-dimensional vector:
wherein:
δz: a residual representing a depth value z;
δh,δw,δl: a residual representing a target size dimension;
sin α, cos α: a sine-cosine value representing an azimuth;
and (3) decoding process:
the average value of the length, the width and the height of the target is obtained by data set labeling and statistics;
μzσzthe mean value and the variance of the mean value of the target depth values are obtained by data set labeling and statistics;
k is camera internal reference, xcycIs the pixel coordinates of the keypoint.
Step 3) training a monocular 3D target detection network;
the monocular 3D target detection method based on the key points has the problem that the confidence coefficient of the detection category is not matched with the geometric precision of the detection frame. The detection network outputs 3D detection frame type and confidence degree information from thermodynamic diagram branches, the probability that the corresponding pixel detects the target center point of the specified type is reflected, and the geometric information from parameter regression branches reflects the geometric information such as the size, the position, the attitude angle and the like of the target frame at the pixel. Because the thermodynamic diagram branch and the parameter regression branch in the detection head are independent from each other in the training process, the confidence of the detection frame cannot truly reflect the geometric accuracy of the detection frame.
Aiming at the mismatching problem, the invention provides an attention loss function acting on a parametric regression branch, which improves the training process by giving more attention to the real target needing further optimization, and constructs the weight of the target loss in the total regression loss according to the target class confidence and the 3D IOU (cross-over ratio) between the prediction box and the truth box, thereby realizing the quantitative definition of the attention, wherein the weight distribution follows the principle that the high class confidence is low, the 3D IOU weight is highest, the high class confidence is high, the 3D IOU is low, and the low class confidence is low, and the 3D IOU weight is low. Because the weight of each target loss in the total regression loss depends on the class confidence of the thermodynamic diagram branch output, the parameter regression branch with the attention loss function needs feedback from the thermodynamic diagram branch in the training process, so that the problem of mismatching of two prediction branches in a standard model detection head due to mutual independence of the training stages is solved, and finally the class confidence of the converged model output can simultaneously reflect the position accuracy information of the corresponding 3D frame and has strong positive correlation.
The loss function of the regression branch of the original model parameters is:
in the formula, LregThe total regression loss;regression loss for the ith target; n is the target number in a training batch;
after introducing the attention loss function, the loss function of the parametric regression branch is:
in the formula, LregThe total regression loss;regression loss for the ith target; n is the target number in a training batch; w is aiWeighting coefficients of the ith target regression loss in the total regression loss;
each target loss weight w in the attention loss functioniThe definition is as follows:
in the formula, PiA category confidence for the ith target;regression loss for the ith target;three-dimensional intersection ratio between the ith target prediction frame and the true value frame is obtained; beta is an equilibrium parameter.
The above target loss weight visualization effect is shown in fig. 2.
And 4) inputting the RGB image obtained in the step 1) into the trained monocular 3D target detection network, and outputting a target detection result.
Embodiment 2 of the present invention may also provide a computer device including: at least one processor, memory, at least one network interface, and a user interface. The various components in the device are coupled together by a bus system. It will be appreciated that a bus system is used to enable communications among the components. The bus system includes a power bus, a control bus, and a status signal bus in addition to a data bus.
The user interface may include, among other things, a display, a keyboard, or a pointing device (e.g., a mouse, trackball, touch pad, or touch screen, among others.
It will be appreciated that the memory in the embodiments disclosed herein can be either volatile memory or nonvolatile memory, or can include both volatile and nonvolatile memory. The non-volatile Memory may be a Read-Only Memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically Erasable PROM (EEPROM), or a flash Memory. Volatile Memory can be Random Access Memory (RAM), which acts as external cache Memory. By way of illustration and not limitation, many forms of RAM are available, such as Static random access memory (Static RAM, SRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic random access memory (Synchronous DRAM, SDRAM), Double Data Rate Synchronous Dynamic random access memory (ddr Data Rate SDRAM, ddr SDRAM), Enhanced Synchronous SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DRRAM). The memory described herein is intended to comprise, without being limited to, these and any other suitable types of memory.
In some embodiments, the memory stores elements, executable modules or data structures, or a subset thereof, or an expanded set thereof as follows: an operating system and an application program.
The operating system includes various system programs, such as a framework layer, a core library layer, a driver layer, and the like, and is used for implementing various basic services and processing hardware-based tasks. The application programs, including various application programs such as a Media Player (Media Player), a Browser (Browser), etc., are used to implement various application services. The program for implementing the method of the embodiment of the present disclosure may be included in an application program.
In the above embodiments, the processor may further be configured to call a program or an instruction stored in the memory, specifically, a program or an instruction stored in the application program, and the processor is configured to:
the steps of the method of example 1 were performed.
The method of embodiment 1 may be applied in or implemented by a processor. The processor may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the above method may be performed by integrated logic circuits of hardware in a processor or instructions in the form of software. The Processor may be a general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic device, or discrete hardware components. The methods, steps, and logic blocks disclosed in embodiment 1 may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in connection with embodiment 1 may be directly implemented by a hardware decoding processor, or may be implemented by a combination of hardware and software modules in the decoding processor. The software module may be located in ram, flash memory, rom, prom, or eprom, registers, etc. storage media as is well known in the art. The storage medium is located in a memory, and a processor reads information in the memory and completes the steps of the method in combination with hardware of the processor.
It is to be understood that the embodiments described herein may be implemented in hardware, software, firmware, middleware, microcode, or any combination thereof. For a hardware implementation, the Processing units may be implemented within one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), general purpose processors, controllers, micro-controllers, microprocessors, other electronic units configured to perform the functions described herein, or a combination thereof.
For a software implementation, the techniques of the present invention may be implemented by executing the functional blocks (e.g., procedures, functions, and so on) of the present invention. The software codes may be stored in a memory and executed by a processor. The memory may be implemented within the processor or external to the processor.
Embodiment 3 of the present invention provides a nonvolatile storage medium for storing a computer program. The computer program may implement the steps of the method in embodiment 1 when executed by a processor.
Finally, it should be noted that the above embodiments are only used for illustrating the technical solutions of the present invention and are not limited. Although the present invention has been described in detail with reference to the embodiments, it will be understood by those skilled in the art that various changes may be made and equivalents may be substituted without departing from the spirit and scope of the invention as defined in the appended claims.
Claims (6)
1. A monocular 3D target detection method based on a lightweight feature pyramid structure, the method comprising:
collecting RGB images of a vehicle-mounted camera;
inputting the RGB image into a pre-established and trained monocular 3D target detection network, and outputting a target detection result; the monocular 3D object detection network comprises: the system comprises a feature extraction network, a detection head and a post-processing module;
the feature extraction network is used for performing down-sampling on the RGB image to extract high-level semantic features, generating 4-time, 8-time and 16-time down-sampling feature maps and inputting the down-sampling feature maps to the detection head;
the detection head is used for generating candidate key point category vectors and candidate key point pixel position index vectors based on the 4-time down-sampling feature map, generating candidate key point 3D regression frame coding vectors based on the 4-time down-sampling feature map, the 8-time down-sampling feature map and the 16-time down-sampling feature map, and outputting the candidate key point category vectors and the 3D regression frame coding vectors to the post-processing module;
the post-processing module is used for decoding the 3D regression frame coding vector and outputting a target detection result by combining the candidate key point category vector;
the detection head comprises a thermodynamic diagram branch and a parameter regression branch;
the thermodynamic diagram branch is used for generating a thermodynamic diagram for target key points based on a 4-time down-sampling feature map, and finally outputting candidate key point category vectors and pixel position index vectors of the candidate key points corresponding to the 16-time down-sampling feature map, the 8-time down-sampling feature map and the 4-time down-sampling feature map by arranging all confidence values in a descending order and screening the positions with the maximum K previous confidence values as candidate key points;
the parameter regression branch introduces a lightweight feature pyramid structure for respectively taking values from corresponding feature maps according to three position indexes, and then merging the values to extract a target 3D regression frame coding vectorAnd output to the post-processing module; wherein K represents the number of detection targets, and R represents the number of regression parameters;
the parametric regression branch comprises: a third convolutional layer, three parallel sampling units, a splicing unit and a 1x1 convolutional layer; the three parallel sampling units comprise: the device comprises a first sampling unit, a second sampling unit and a third sampling unit;
the third convolution layer is used for further extracting the characteristics of the 4 times down-sampling characteristic diagram and outputting the characteristic diagram to the first sampling unit;
the first sampling unit is used for sampling values from the feature map output by the third convolutional layer according to pixel position indexes of the candidate key points on the feature map sampled by 4 times;
the second sampling unit is used for sampling values from the 8-time down-sampling feature map output by the second deconvolution layer according to the pixel position index 1/2 indexes of the candidate key points on the 8-time down-sampling feature map;
the third sampling unit is used for sampling values from the 16-time down-sampling feature map output by the first deconvolution layer according to the pixel position index 1/4 indexes of the candidate key points on the 16-time down-sampling feature map;
the splicing unit is used for merging the values output by the three sampling units to realize the feature reading, alignment and fusion of the candidate target key points on the feature maps with different resolutions; outputting the fused features to a 1x1 convolutional layer;
2. The monocular 3D object detection method based on a lightweight feature pyramid structure of claim 1, wherein the feature extraction network comprises an encoder and a decoder;
the encoder is used for performing down-sampling on the input RGB image to extract high-level semantic features and outputting a 32-time down-sampling feature map;
the decoder is used for up-sampling the high-level semantic feature map output by the encoder to obtain 4-time, 8-time and 16-time down-sampling feature maps required by the detection head; the decoder includes three deconvolution layers: a first deconvolution layer, a second deconvolution layer, and a third deconvolution layer; a first deconvolution layer for processing the 32-fold down-sampling feature map output by the encoder and outputting a 16-fold down-sampling feature map; a second deconvolution layer for processing the 16-fold down-sampling feature map and outputting an 8-fold down-sampling feature map; and a third deconvolution layer for processing the 8-fold down-sampling feature map and outputting the 4-fold down-sampling feature map to the detection head.
3. The monocular 3D object detection method based on a lightweight feature pyramid structure of claim 2, wherein the thermodynamic diagram branch comprises: a first convolution layer, a second convolution layer and a TopK operation unit;
the first convolution layer is used for further extracting the features of the 4-time down-sampling feature map and outputting the feature map to the second convolution layer;
the second convolution layer is used for performing convolution processing on the characteristic diagram and outputting a thermodynamic diagram, wherein any element y on the thermodynamic diagram is outputijcRepresenting a probability value that at the (i, j) pixel position of the thermodynamic diagram, a target keypoint of category c exists;
the TopK operation unit is used for arranging all probability values on the thermodynamic diagram in a descending order, taking the first K candidate points with the maximum probability values as candidate key points, converting pixel coordinates (i, j) into position indexes Index, converting channel indexes c representing categories into categories, splicing the K category values into candidate key point category vectors Classes, and outputting the candidate key point category vectors Classes to the post-processing module; and splicing and generating candidate key point pixel position Index vectors indexes according to the K position Index indexes: acquiring pixel position indexes of the candidate key points corresponding to 4 times of downsampling feature maps; the pixel position indexes 1/2 indexes of the candidate key points corresponding to the 8-time down-sampling feature map and the pixel position indexes 1/4 indexes of the 16-time down-sampling feature map can be further obtained through division operation, the three position indexes are one-dimensional vectors with the same number of elements, and three values of the three one-dimensional vectors at the same position respectively correspond to pixel position indexes of the same candidate key points at the 4-time down-sampling feature map, the 8-time down-sampling feature map and the 16-time down-sampling feature map.
4. The monocular 3D object detection method based on a lightweight feature pyramid structure as claimed in claim 3, further comprising: the method for training the monocular 3D target detection network specifically comprises the following steps:
in the parameter regression branch, the function L of attention loss is establishedreg:
In the formula (I), the compound is shown in the specification,regression loss for the ith target; n is the target number in a training batch; w is aiWeighting coefficients of the ith target regression loss in the total regression loss;
each target loss weight w in the attention loss functioniThe definition is as follows:
5. A terminal device, comprising: comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the method according to any of claims 1 to 4 when executing the computer program.
6. A storage medium, comprising: comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the method according to any of claims 1 to 4 when executing the computer program.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202110326713.7A CN112990050B (en) | 2021-03-26 | 2021-03-26 | Monocular 3D target detection method based on lightweight characteristic pyramid structure |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202110326713.7A CN112990050B (en) | 2021-03-26 | 2021-03-26 | Monocular 3D target detection method based on lightweight characteristic pyramid structure |
Publications (2)
Publication Number | Publication Date |
---|---|
CN112990050A CN112990050A (en) | 2021-06-18 |
CN112990050B true CN112990050B (en) | 2021-10-08 |
Family
ID=76333846
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202110326713.7A Active CN112990050B (en) | 2021-03-26 | 2021-03-26 | Monocular 3D target detection method based on lightweight characteristic pyramid structure |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN112990050B (en) |
Families Citing this family (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN114332792B (en) * | 2021-12-09 | 2023-05-16 | 苏州驾驶宝智能科技有限公司 | Three-dimensional scene target detection method and system based on key point multi-scale fusion |
CN114842287B (en) * | 2022-03-25 | 2022-12-06 | 中国科学院自动化研究所 | Monocular three-dimensional target detection model training method and device of depth-guided deformer |
CN114821717B (en) * | 2022-04-20 | 2024-03-12 | 北京百度网讯科技有限公司 | Target object fusion method and device, electronic equipment and storage medium |
CN115063789B (en) * | 2022-05-24 | 2023-08-04 | 中国科学院自动化研究所 | 3D target detection method and device based on key point matching |
CN115661577B (en) * | 2022-11-01 | 2024-04-16 | 吉咖智能机器人有限公司 | Method, apparatus and computer readable storage medium for object detection |
CN116403180B (en) * | 2023-06-02 | 2023-08-15 | 上海几何伙伴智能驾驶有限公司 | 4D millimeter wave radar target detection, tracking and speed measurement method based on deep learning |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN109190662A (en) * | 2018-07-26 | 2019-01-11 | 北京纵目安驰智能科技有限公司 | A kind of three-dimensional vehicle detection method, system, terminal and storage medium returned based on key point |
CN111291714A (en) * | 2020-02-27 | 2020-06-16 | 同济大学 | Vehicle detection method based on monocular vision and laser radar fusion |
CN111369617A (en) * | 2019-12-31 | 2020-07-03 | 浙江大学 | 3D target detection method of monocular view based on convolutional neural network |
CN111832655A (en) * | 2020-07-16 | 2020-10-27 | 四川大学 | Multi-scale three-dimensional target detection method based on characteristic pyramid network |
Family Cites Families (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10915793B2 (en) * | 2018-11-08 | 2021-02-09 | Huawei Technologies Co., Ltd. | Method and system for converting point cloud data for use with 2D convolutional neural networks |
CN112069896B (en) * | 2020-08-04 | 2022-12-02 | 河南科技大学 | Video target tracking method based on twin network fusion multi-template features |
-
2021
- 2021-03-26 CN CN202110326713.7A patent/CN112990050B/en active Active
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN109190662A (en) * | 2018-07-26 | 2019-01-11 | 北京纵目安驰智能科技有限公司 | A kind of three-dimensional vehicle detection method, system, terminal and storage medium returned based on key point |
CN111369617A (en) * | 2019-12-31 | 2020-07-03 | 浙江大学 | 3D target detection method of monocular view based on convolutional neural network |
CN111291714A (en) * | 2020-02-27 | 2020-06-16 | 同济大学 | Vehicle detection method based on monocular vision and laser radar fusion |
CN111832655A (en) * | 2020-07-16 | 2020-10-27 | 四川大学 | Multi-scale three-dimensional target detection method based on characteristic pyramid network |
Non-Patent Citations (2)
Title |
---|
"PointFPN: A Frustum-based Feature Pyramid Network for 3D Object Detection";Zhaoxin Fan;《2020 IEEE 32nd International Conference on Tools with Artificial Intelligence (ICTAI)》;20201224;1129-1136 * |
"基于改进特征金字塔的Mask R-CNN 目标检测方法";任之俊等;《激光与光电子学进展》;20190228;第56卷(第4期);第1-6页 * |
Also Published As
Publication number | Publication date |
---|---|
CN112990050A (en) | 2021-06-18 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN112990050B (en) | Monocular 3D target detection method based on lightweight characteristic pyramid structure | |
US11120276B1 (en) | Deep multimodal cross-layer intersecting fusion method, terminal device, and storage medium | |
CN111126359B (en) | High-definition image small target detection method based on self-encoder and YOLO algorithm | |
CN110516541B (en) | Text positioning method and device, computer readable storage medium and computer equipment | |
CN110032969B (en) | Method, apparatus, device, and medium for detecting text region in image | |
CN111950453A (en) | Optional-shape text recognition method based on selective attention mechanism | |
CN111191583A (en) | Space target identification system and method based on convolutional neural network | |
CN113221740B (en) | Farmland boundary identification method and system | |
CN113095152B (en) | Regression-based lane line detection method and system | |
CN113284163A (en) | Three-dimensional target self-adaptive detection method and system based on vehicle-mounted laser radar point cloud | |
CN115393680B (en) | 3D target detection method and system for multi-mode information space-time fusion in foggy weather scene | |
CN115147598B (en) | Target detection segmentation method and device, intelligent terminal and storage medium | |
CN114387521B (en) | Remote sensing image building extraction method based on attention mechanism and boundary loss | |
CN113298032A (en) | Unmanned aerial vehicle visual angle image vehicle target detection method based on deep learning | |
CN114037640A (en) | Image generation method and device | |
CN112733942A (en) | Variable-scale target detection method based on multi-stage feature adaptive fusion | |
Jeon et al. | ABCD: Attentive bilateral convolutional network for robust depth completion | |
CN114782865B (en) | Intersection vehicle positioning method and system based on multi-view and re-recognition | |
CN117079163A (en) | Aerial image small target detection method based on improved YOLOX-S | |
CN115294468A (en) | SAR image ship identification method for improving fast RCNN | |
CN117173657A (en) | Pre-training method for automatic driving perception model | |
CN114550163B (en) | Imaging millimeter wave three-dimensional target detection method based on deformable attention mechanism | |
CN116363361A (en) | Automatic driving method based on real-time semantic segmentation network | |
CN111797782A (en) | Vehicle detection method and system based on image features | |
Zhang et al. | SA‐BiSeNet: Swap attention bilateral segmentation network for real‐time inland waterways segmentation |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant |