CN111257866A

CN111257866A - Target detection method, device and system for linkage of vehicle-mounted camera and vehicle-mounted radar

Info

Publication number: CN111257866A
Application number: CN201811459743.XA
Authority: CN
Inventors: 邝宏武; 方梓成; 孙杰
Original assignee: Hangzhou Hikvision Digital Technology Co Ltd
Current assignee: Hangzhou Hikvision Digital Technology Co Ltd
Priority date: 2018-11-30
Filing date: 2018-11-30
Publication date: 2020-06-09
Anticipated expiration: 2038-11-30
Also published as: CN111257866B; WO2020108647A1

Abstract

The application relates to a target detection method, device and system for linkage of a vehicle-mounted camera and a vehicle-mounted radar, and belongs to the field of intelligent traffic. The method comprises the following steps: detecting each image target around the vehicle, the confidence of each image target and the position information of each image target in an image coordinate system from a video image provided by a vehicle-mounted camera; detecting each radar target around the vehicle and position information of each radar target in a radar coordinate system from a speed and distance image provided by the radar; acquiring a first perspective matrix according to the position information of the image target with the confidence coefficient exceeding a first preset threshold and the position information of the radar target with the confidence coefficient exceeding a second preset threshold; and detecting a target class from the image target with the confidence coefficient not exceeding the first preset threshold and the radar target with the confidence coefficient not exceeding the second preset threshold through the first perspective matrix. The method and the device can improve the precision of the detection target.

Description

Target detection method, device and system for linkage of vehicle-mounted camera and vehicle-mounted radar

Technical Field

The application relates to the field of intelligent traffic, in particular to a target detection method, device and system for linkage of a vehicle-mounted camera and a vehicle-mounted radar.

Background

Automatic driving is an important application in an intelligent traffic system, and requires a vehicle to detect surrounding vehicles and carry out avoidance according to the surrounding vehicles so as to avoid traffic accidents. At present, a camera and a radar are included in a vehicle, and the vehicle can detect the vehicles around the vehicle based on the camera and the radar.

At present, there is a method for detecting vehicles around an automobile, which may be: first position information of each image target around the vehicle, which may be the vehicle, is detected based on the camera, and second position information of each radar target around the vehicle, which may be the vehicle, is detected based on the radar. And mapping the first position information of each image target to a road coordinate system through a preset first perspective matrix to obtain third position information of each image target, and mapping the second position information of each radar target to the road coordinate system through a preset second perspective matrix to obtain fourth position information of each radar target. And determining at least one target pair according to the third position information of each image target and the fourth position information of each radar target, wherein each target pair comprises an image target and a radar target belonging to the same object. Since the image target and the radar target in the target pair are targets that may be detected by the camera and the radar at the same time, respectively, and therefore the target in the target pair is likely to be a vehicle, the target in the target pair is regarded as the detected vehicle.

In the process of implementing the present application, the inventors found that the above manner has at least the following defects:

the first perspective matrix is preset to reflect the conversion relationship between the image coordinate system of the camera and the road coordinate system, and the conversion relationship may change when the vehicle runs under different road conditions, and the preset first perspective matrix cannot accurately reflect the conversion relationship between the image coordinate system of the current camera and the road coordinate system, so that the precision of the third position information obtained by mapping the first position information of the image target to the road coordinate system through the preset first perspective matrix is reduced, and the precision of the detected target is further reduced.

Disclosure of Invention

In order to improve the precision of a detected target, the embodiment of the application provides a vehicle detection method and device based on a camera and a radar. The technical scheme is as follows:

according to a first aspect of the embodiments of the present application, a target detection method for linkage of a vehicle-mounted camera and a vehicle-mounted radar is provided, where the method includes:

detecting each image target around the vehicle, the confidence coefficient of each image target and the position information of each image target in an image coordinate system from a video image provided by a vehicle-mounted camera;

detecting each radar target around the vehicle and position information of each radar target in a radar coordinate system from a speed and distance image provided by the radar, wherein the confidence of the radar target is used for representing the probability that the target category of the image target or a real target corresponding to the radar target is a specified category;

acquiring a first perspective matrix according to the position information of the image target with the confidence coefficient exceeding a first preset threshold and the position information of the radar target with the confidence coefficient exceeding a second preset threshold, wherein the first perspective matrix is used for representing the conversion relation between the image coordinate system and a preset road coordinate system;

and detecting a target class from the image target with the confidence coefficient not exceeding the first preset threshold and the radar target with the confidence coefficient not exceeding the second preset threshold through the first perspective matrix.

Optionally, the method further includes:

classifying the image target with the reliability exceeding the first preset threshold value according to the detected confidence coefficient of each image target to obtain a target class of a real target corresponding to the image target, and outputting the target class, an

And classifying the radar targets with the confidence degrees exceeding the second preset threshold value according to the detected confidence degrees of the radar targets to obtain target classes of the real targets corresponding to the radar targets, and outputting the target classes.

Optionally, before the obtaining the first perspective matrix according to the position information of the image target whose confidence exceeds the first preset threshold and the position information of the radar target whose confidence exceeds the second preset threshold, the method further includes:

determining N associated target pairs from image targets with confidence degrees exceeding a first preset threshold and radar targets with confidence degrees exceeding a second preset threshold, wherein any one associated target pair comprises a radar target and an image target which meet a preset association condition, and N is a positive integer greater than or equal to 1;

the acquiring a first perspective matrix according to the position information of the image target with the confidence coefficient exceeding the first preset threshold and the position information of the radar target with the confidence coefficient exceeding the second preset threshold includes:

and determining the first perspective matrix according to the position information of the radar target and the image target in the N associated target pairs.

Optionally, the determining N associated target pairs from the image target whose confidence exceeds the first preset threshold and the radar target whose confidence exceeds the second preset threshold includes:

mapping the first image target from the image coordinate system to the road coordinate system according to first position information of the first image target and a stored second perspective matrix to obtain corresponding third position information of the first image target in the road coordinate system; the first image target is an image target with a confidence coefficient exceeding a first preset threshold value, and the first position information is the position information of the first image target in the image coordinate system;

mapping the first radar target from the radar coordinate system to the road coordinate system according to second position information of the first radar target and a stored third perspective matrix to obtain corresponding fourth position information of the first radar target in the road coordinate system; the first radar target is a radar target with a confidence coefficient exceeding a second preset threshold value, and the second position information is the position information of the first radar target in the radar coordinate system;

and performing position association on each first image target and each first radar target according to the third position information of each first image target and the fourth position information of each first radar target to obtain the N associated target pairs.

Optionally, the performing position association on each first image target and each second radar target to obtain the N associated target pairs includes:

determining a projection area of the first image target in the road coordinate system, a projection area of the first radar target in the road coordinate system, and an overlapping projection area of the first image target and the first radar target in the road coordinate system according to the third position information of the first image target and the fourth position information of the first radar target;

determining an association cost between each first image target and each second radar target according to the projection area of the first image target in the road coordinate system, the projection area of the first radar target in the road coordinate system and the overlapping projection area;

and determining one first radar target and one first image target with the minimum association cost from the first image target and the second image target as an association target pair, and further obtaining the N association target pairs.

Optionally, the determining the first perspective matrix according to the position information of the radar target and the image target in the N relevant target pairs includes:

for any one of the N associated target pairs, correcting the position information of the first image target in the associated target pair according to the position information of the first radar target in the associated target pair;

and correcting the second perspective matrix according to the corrected position information of each first image target in the N associated target pairs to obtain the first perspective matrix.

Optionally, the detecting, by the first perspective matrix, a target class from the image target whose confidence does not exceed the first preset threshold and the radar target whose confidence does not exceed the second preset threshold includes:

determining M feature fusion target pairs from image targets with confidence degrees not exceeding a first preset threshold and radar targets with confidence degrees not exceeding a second preset threshold, wherein any one feature fusion target pair comprises one radar target and one image target meeting a preset association condition, and M is a positive integer greater than or equal to 1;

for any feature fusion target pair, respectively performing convolution calculation on the echo energy features of the radar target and the image features of the image target in the feature fusion target pair, and then splicing to obtain a fusion feature map corresponding to the feature fusion target pair;

and performing convolution and full-connection calculation on the fusion characteristic graph, and inputting the fusion characteristic graph into a classification network for target classification to obtain a target class corresponding to the fusion characteristic graph.

Optionally, the determining M feature fusion target pairs from the second image target and the second radar target includes:

mapping a second image target from the image coordinate system to the road coordinate system through the first perspective matrix to obtain corresponding position information of the second image target in the road coordinate system, and mapping a second radar target from the radar coordinate system to the road coordinate system through a prestored third perspective matrix to obtain corresponding position information of the second radar target in the road coordinate system, wherein the second image target is an image target of which the confidence coefficient does not exceed a first preset threshold value, and the second radar target is a radar target of which the confidence coefficient does not exceed a second preset threshold value;

and performing position association on each second image target and each second radar target according to the corresponding position information of each second image target in the road coordinate system and the corresponding position information of each second radar target in the road coordinate system to obtain the M feature fusion target pairs.

Optionally, the detecting confidence of each image target around the vehicle from the video image provided by the vehicle-mounted camera includes:

acquiring a classification confidence coefficient, a tracking frame confidence coefficient and a position confidence coefficient of any one image target according to a current frame video image provided by the vehicle-mounted camera and a multi-frame historical frame video image close to the current frame video image;

determining the confidence of the image target according to one or more of the classification confidence, the position confidence and the tracking frame number confidence;

the detecting confidence of each radar target around the vehicle from the speed and distance image collected by the radar comprises:

and determining the confidence of the radar target according to the echo energy intensity of any one radar target in the current frame speed range image, the distance from the vehicle and the duration of the radar target in the multi-frame historical frame speed range image.

According to a second aspect of the embodiments of the present application, there is provided a target detection device in which an on-vehicle camera and an on-vehicle radar are linked, the device including:

the system comprises a first detection module, a second detection module and a third detection module, wherein the first detection module is used for detecting each image target around the vehicle, the confidence coefficient of each image target and the position information of each image target in an image coordinate system from a video image provided by a vehicle-mounted camera;

the second detection module is used for detecting each radar target around the vehicle and position information of each radar target in a radar coordinate system from a speed and distance image provided by the radar, and the confidence of the radar targets is used for indicating the probability that the target category of the image target or a real target corresponding to the radar target is a specified category;

the acquisition module is used for acquiring a first perspective matrix according to the position information of the image target with the confidence coefficient exceeding a first preset threshold and the position information of the radar target with the confidence coefficient exceeding a second preset threshold, wherein the first perspective matrix is used for representing the conversion relation between the image coordinate system and a preset road coordinate system;

and the third detection module is used for detecting a target class from the image target of which the confidence coefficient does not exceed the first preset threshold and the radar target of which the confidence coefficient does not exceed the second preset threshold through the first perspective matrix.

Optionally, the apparatus further comprises:

and the classification module is used for classifying the image targets with the reliability exceeding the first preset threshold according to the detected confidence degrees of the image targets to obtain target classes of real targets corresponding to the image targets and outputting the target classes, and classifying the radar targets with the reliability exceeding the second preset threshold according to the detected confidence degrees of the radar targets to obtain target classes of the real targets corresponding to the radar targets and outputting the target classes.

Optionally, the apparatus further comprises:

the device comprises a determining module, a judging module and a judging module, wherein the determining module is used for determining N associated target pairs from image targets with confidence degrees exceeding a first preset threshold and radar targets with confidence degrees exceeding a second preset threshold, any one associated target pair comprises a radar target and an image target meeting preset association conditions, and N is a positive integer greater than or equal to 1;

the acquisition module is configured to determine the first perspective matrix according to the position information of the radar target and the image target in the N associated target pairs.

Optionally, the determining module is configured to:

Optionally, the obtaining module is configured to:

Optionally, the third detecting module is configured to:

Optionally, the first detecting module is configured to:

the second detection module is configured to:

According to a third aspect of embodiments of the present application, there is provided an object detection system including a radar provided on a vehicle, an onboard camera provided on the vehicle, and a detection device in communication with the radar and the onboard camera,

the vehicle-mounted camera is used for shooting the periphery of the vehicle to obtain a current frame video image and providing the shot current frame video image for the detection device;

the radar is used for generating a current frame speed distance image according to the transmitted radar signal and the received echo signal and providing the current frame speed distance image for the detection device;

the detection device is used for detecting each image target around the vehicle, the confidence of each image target and the position information of each image target in an image coordinate system from the video image provided by the vehicle-mounted camera; detecting each radar target around the vehicle and position information of each radar target in a radar coordinate system from a speed and distance image provided by the radar, wherein the confidence of the radar target is used for representing the probability that the target category of the image target or a real target corresponding to the radar target is a specified category; acquiring a first perspective matrix according to the position information of the image target with the confidence coefficient exceeding a first preset threshold and the position information of the radar target with the confidence coefficient exceeding a second preset threshold, wherein the first perspective matrix is used for representing the conversion relation between the image coordinate system and a preset road coordinate system; and detecting a target class from the image target with the confidence coefficient not exceeding the first preset threshold and the radar target with the confidence coefficient not exceeding the second preset threshold through the first perspective matrix.

Optionally, the vehicle-mounted camera is disposed at the front and rear and/or left and right sides of the vehicle, and the radar is disposed at the front and rear of the vehicle.

Optionally, the radar is a millimeter wave radar.

According to a fourth aspect of embodiments herein, there is provided a non-transitory computer readable storage medium for storing a computer program which is loaded and executed by a processor to implement the instructions of the first aspect or any one of the alternative methods of the first aspect.

The technical scheme provided by the embodiment of the application can have the following beneficial effects:

detecting the position information of image targets around the vehicle in an image coordinate system and the confidence coefficient of the image targets through a vehicle-mounted camera, and detecting the position information of radar targets around the vehicle in a radar coordinate system and the confidence coefficient of each radar target through a radar; because the image target with the confidence coefficient exceeding the first preset threshold and the radar target with the confidence coefficient exceeding the second preset threshold are real targets of the appointed category, the conversion relation between the image coordinate system of the vehicle-mounted camera of the vehicle under the current road condition and the road coordinate relation can be accurately reflected according to the first perspective matrix obtained by the image target with the confidence coefficient exceeding the first preset threshold and the radar target with the confidence coefficient exceeding the second preset threshold, so that the accuracy of the real target with the appointed category, which is detected from the image target with the confidence coefficient not exceeding the first preset threshold and the radar target with the confidence coefficient not exceeding the second preset threshold, is higher according to the first perspective matrix, and the accuracy of the target detection is improved.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the application.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and together with the description, serve to explain the principles of the application.

Fig. 1 is a schematic structural diagram of a target detection system in which a vehicle-mounted camera and a vehicle-mounted radar are linked according to an embodiment of the present application;

fig. 2 is a flowchart of a target detection method in which a vehicle-mounted camera and a vehicle-mounted radar are linked according to an embodiment of the present application;

fig. 3 is a flowchart of a target detection method in which a vehicle-mounted camera and a vehicle-mounted radar are linked according to an embodiment of the present application;

FIG. 4 is a target fusion block diagram of linkage of a vehicle-mounted camera and a vehicle-mounted radar provided by the embodiment of the application;

FIG. 5 is a flowchart of a method for obtaining confidence of an image target according to an embodiment of the present disclosure;

FIG. 6 is a schematic structural diagram of a first convolutional neural network provided in an embodiment of the present application;

fig. 7 is a flowchart of a method for obtaining confidence of a radar target according to an embodiment of the present disclosure;

FIG. 8 is a flowchart of a method for obtaining a first perspective matrix according to an embodiment of the present disclosure;

FIG. 9 is a flowchart of a method for detecting a target with a specific category according to an embodiment of the present disclosure;

FIG. 10 is a flow chart of detecting a class as a target of a specified class by a convolutional neural network provided by an embodiment of the present application;

fig. 11 is a schematic structural diagram of a target detection device in which a vehicle-mounted camera and a vehicle-mounted radar are linked according to an embodiment of the present application;

FIG. 12 is a schematic structural diagram of an apparatus according to an embodiment of the present disclosure;

fig. 13 is a schematic structural diagram of an object detection system according to an embodiment of the present application;

FIG. 14 is a schematic diagram of various image objects around a vehicle detected from a video image according to an embodiment of the present application;

fig. 15 is a schematic diagram of detecting respective radar targets around a vehicle from a speed range image provided by a radar according to an embodiment of the present application.

With the above figures, there are shown specific embodiments of the present application, which will be described in more detail below. These drawings and written description are not intended to limit the scope of the inventive concepts in any manner, but rather to illustrate the inventive concepts to those skilled in the art by reference to specific embodiments.

Detailed Description

Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

In the intelligent traffic system, the vehicle can detect other vehicles around the vehicle when running on the road, so that a driver can prompt the driver of other vehicles around the vehicle when driving the vehicle to assist the driver in driving the vehicle, or the vehicle can plan a driving lane or avoid other vehicles according to the vehicles around the vehicle when automatically driving the vehicle.

Referring to fig. 1, an embodiment of the present application provides a target detection system with a vehicle-mounted camera and a vehicle-mounted radar linked, including a vehicle-mounted camera 1 disposed on a vehicle, a vehicle-mounted radar 2 disposed on the vehicle, and a detection device 3 in communication with the radar 2 and the vehicle-mounted camera 1; the vehicle-mounted camera 1 is arranged at the front and rear part and/or the left and right sides of the vehicle, and the radar 2 is arranged at the front and rear part of the vehicle;

the vehicle-mounted camera 1 is used for shooting the surrounding environment of the vehicle to obtain a video image;

the radar 2 is used for sending radar waves to the periphery of the vehicle, receiving reflected waves corresponding to the radar waves and acquiring the echo energy intensity of the reflected waves to obtain a speed distance image;

the detection device 3 is used for detecting the position information and the confidence coefficient of each image target around the vehicle in the image coordinate system based on the video image, wherein the confidence coefficient is used for indicating the probability that the target class of the real target corresponding to the image target is the specified class, and the position information and the confidence coefficient of the radar target around the vehicle in the radar coordinate system are detected based on the speed distance image, and the confidence coefficient is used for indicating the probability that the target class of the real target corresponding to the radar target is the specified class;

the detection device 3 is further configured to obtain a first perspective matrix according to the position information of the image target whose confidence coefficient exceeds a first preset threshold and the position information of the image target whose confidence coefficient exceeds a second preset threshold, where the first perspective matrix is used to represent a conversion relationship between an image coordinate system and a preset road coordinate system; and detecting a target class from the image target with the confidence coefficient not exceeding a first preset threshold value and the radar target with the confidence coefficient not exceeding a second preset threshold value through the first perspective matrix.

Optionally, the vehicle includes a detection device, and the vehicle detects surrounding vehicles through the vehicle-mounted device, the vehicle-mounted camera, and the radar. The vehicle-mounted device can detect each image target around the vehicle through the vehicle-mounted camera, and detect each radar target around the vehicle through the radar. Each image target and each radar target may be a designated class of real targets, which may be vehicles, etc.

Optionally, the radar may be a millimeter wave radar, a laser radar, or the like.

It should be noted that: if a target is detected by the vehicle-mounted camera as a real target of a specified category and is detected by the radar as a real target of a specified category at the same time, the target is more likely to be a real target of the specified category. Therefore, in order to improve the accuracy of detecting objects around the automobile, each image object detected by the vehicle-mounted camera and each radar object detected by the radar can be fused to improve the detection accuracy. The above processes of detecting the image target, the radar target and the fusion may refer to the following description of any one of the embodiments, which is not described herein.

Referring to fig. 2, an embodiment of the present application provides a target detection method based on linkage of a vehicle-mounted camera and a vehicle-mounted radar, where the method includes:

step 201: each image object around the vehicle, the confidence of each image object and the position information of each image object in an image coordinate system are detected from a video image provided by the vehicle-mounted camera.

Step 202: and detecting each radar target around the vehicle and position information of each radar target in a radar coordinate system from a speed and distance image provided by the radar, wherein the confidence of the radar target is used for indicating the probability that the target category of the image target or the real target corresponding to the radar target is the specified category.

Step 203: and acquiring a first perspective matrix according to the position information of the image target with the confidence coefficient exceeding a first preset threshold and the position information of the radar target with the confidence coefficient exceeding a second preset threshold, wherein the first perspective matrix is used for representing the conversion relation between the image coordinate system and the preset road coordinate system.

Step 204: and detecting a target class from the image target with the confidence coefficient not exceeding a first preset threshold value and the radar target with the confidence coefficient not exceeding a second preset threshold value through the first perspective matrix.

Optionally, after the step 204, a step 205 is further included, in which, according to the confidence degrees of the detected image targets, the image targets whose confidence degrees exceed the first preset threshold are classified to obtain target classes of real targets corresponding to the image targets, and the target classes are output, and according to the detected confidence degrees of the radar targets, the radar targets whose confidence degrees exceed the second preset threshold are classified to obtain target classes of real targets corresponding to the radar targets, and the target classes are output. The high-confidence image target and the radar target are classified to obtain the target of the required target class and output. Through the steps, the confidence coefficient and the target classification of the image target, the confidence coefficient and the target classification of the radar coordinate and the decision-level fusion of the high-confidence target in the step 4 can be calculated.

Referring to fig. 14, the respective image objects around the vehicle detected from the video image provided by the onboard camera, the confidence levels of the respective image objects, and the position information of the respective image objects in the image coordinate system are illustrated. The targets in the superimposed frame are image targets whose confidence levels exceed a first preset threshold, and referring to fig. 15, the detection of each radar target around the vehicle from the speed-distance image provided by the radar, and the position information of each radar target in the radar coordinate system, the confidence level of the radar target are illustrated, wherein the targets in the superimposed frame are radar targets whose confidence levels exceed a second preset threshold.

Optionally, after step 203, step 206 is further included, determining N associated target pairs from the image target whose confidence coefficient exceeds the first preset threshold and the radar target whose confidence coefficient exceeds the second preset threshold, where any one of the associated target pairs includes a radar target and an image target that satisfy a preset association condition, and N is a positive integer greater than or equal to 1; in this way, the first perspective matrix may be determined based on the position information of the radar target and the image target in the N associated target pairs.

Optionally, step 206 may include:

2061: mapping the first image target from the image coordinate system to a road coordinate system according to the first position information of the first image target and the stored second perspective matrix to obtain third position information corresponding to the first image target in the road coordinate system; the first image target is an image target with a confidence coefficient exceeding a first preset threshold, and the first position information is the position information of the first image target in the image coordinate system;

2062: mapping the first radar target from a radar coordinate system to a road coordinate system according to the second position information of the first radar target and the stored third perspective matrix to obtain corresponding fourth position information of the first radar target in the road coordinate system; the first radar target is a radar target with confidence coefficient exceeding a second preset threshold value, and the second position information is the position information of the first radar target in a radar coordinate system;

2063: and performing position association on each first image target and each first radar target according to the third position information of each first image target and the fourth position information of each first radar target to obtain N associated target pairs.

Through the steps 2061 to 2063, N associated target pairs can be obtained, and based on the N associated target pairs, the high-confidence image target detected by the vehicle-mounted camera and the high-confidence radar target detected by the radar can be dynamically aligned, so that the dynamic alignment among the image coordinate system, the road coordinate system and the radar coordinate system in fig. 4 can be realized.

Optionally, the step 2063 may be:

determining a projection area of the first image target in the road coordinate system, a projection area of the first radar target in the road coordinate system and an overlapped projection area of the first image target and the first radar target in the road coordinate system according to the third position information of the first image target and the fourth position information of the first radar target;

determining the association cost between each first image target and each second radar target according to the projection area of the first image target in the road coordinate system, the projection area of the first radar target in the road coordinate system and the overlapping projection area;

and determining a first radar target and a first image target with the minimum association cost from the first image target and the second image target as an association target pair, and further obtaining N association target pairs.

Since the association cost between each first image target and each second radar target is determined, one first radar target and one first image target with the minimum association cost are determined from the first image target and the second image target to be an association target pair, and therefore the accuracy of the association target pair can be improved.

Optionally, step 203 may include:

2031: for any one of the N associated target pairs, correcting the position information of the first image target in the associated target pair according to the position information of the first radar target in the associated target pair;

2032: and correcting the second perspective matrix according to the corrected position information of each first image target in the N associated target pairs to obtain a first perspective matrix. The position information of the radar coordinate system is more accurate relative to the position information of the image coordinate system, so that the position information of the image target is corrected by using the position information of the radar target for one associated target pair, the position information of the image target can be closer to the position information of a real target, the second perspective matrix is corrected into the first perspective matrix according to the corrected position information of each image target, and the obtained first perspective matrix can accurately reflect the conversion relation between the current image coordinate system and the road coordinate system, so that the position information when the low-confidence image target is mapped to the road coordinate system can be corrected correspondingly.

Optionally, step 204 may include:

2041: m feature fusion target pairs are determined from the image targets with the confidence degrees not exceeding the first preset threshold value and the radar targets with the confidence degrees not exceeding the second preset threshold value, any one feature fusion target pair comprises one radar target and one image target meeting preset association conditions, and M is a positive integer greater than or equal to 1.

2042: for any feature fusion target pair, respectively performing convolution calculation on the echo energy features of the radar target in the feature fusion target pair and the image features of the image target, and then splicing to obtain a fusion feature map corresponding to the feature fusion target pair;

2043: and performing convolution and full-connection calculation on the fusion characteristic graph, and inputting the fusion characteristic graph into a classification network for target classification to obtain a target class corresponding to the fusion characteristic graph.

The feature level fusion of the low-confidence target in fig. 4 can be realized through the above-mentioned steps 2041 to 2043, and specific steps 2042 and 2043 can refer to the flow shown in fig. 10.

Optionally, step 2041 may be:

mapping a second image target from the image coordinate system to the road coordinate system through a first perspective matrix to obtain corresponding position information of the second image target in the road coordinate system, and mapping a second radar target from the radar coordinate system to the road coordinate system through a prestored third perspective matrix to obtain corresponding position information of the second radar target in the road coordinate system, wherein the second image target is an image target of which the confidence coefficient does not exceed a first preset threshold value, and the second radar target is a radar target of which the confidence coefficient does not exceed a second preset threshold value;

and performing position association on each second image target and each second radar target according to the corresponding position information of each second image target in the road coordinate system and the corresponding position information of each second radar target in the road coordinate system to obtain M feature fusion target pairs.

Optionally, step 201 may include:

2011: according to a current frame video image provided by the vehicle-mounted camera and a multi-frame historical frame video image close to the current frame video image, acquiring the classification confidence coefficient, tracking frame number confidence coefficient and position confidence coefficient of any image target;

2012: determining a confidence of the image target based on one or more of the classification confidence, the position confidence, and the tracking frame number confidence.

Optionally, step 202 may be:

and determining the confidence of the radar target according to the echo energy intensity of any radar target in the current frame speed range image, the distance from the vehicle and the duration of the radar target in the multi-frame historical frame speed range image.

Referring to fig. 3 and fig. 4, an embodiment of the present application provides a target detection method for linkage of a vehicle-mounted camera and a vehicle-mounted radar, where the method may be applied to the architecture shown in fig. 1, and an execution subject of the method may be a vehicle-mounted device, and the vehicle-mounted device may be the detection device 3 in the system shown in fig. 1, and the method includes:

step 301: the method comprises the steps of detecting position information, target area and confidence coefficient of each image target around the vehicle in an image coordinate system from a video image provided by a vehicle-mounted camera of the vehicle, wherein the confidence coefficient is used for representing the probability that a target class of a real target corresponding to the image target is a specified class.

The vehicle-mounted camera can shoot the environment around the vehicle to obtain a video image of one frame, and each time the obtained video image of one frame is shot, the currently shot video image of one frame is called a first video image for convenience of description, and the first video image is input into the vehicle-mounted equipment; the vehicle-mounted equipment comprises a first convolution neural network, and the position information, the target area and the confidence coefficient of each image target with the type being the designated type in the first video image can be detected through the first convolution neural network according to the first video image.

Optionally, the image targets with the confidence degrees exceeding the first preset threshold may be classified according to the detected confidence degrees of the image targets, so as to obtain a target class of the real target corresponding to the image target, and the target class is output.

The vehicle-mounted camera comprises an image coordinate system, and the position information and the target area of each image target are the position information and the area of each image target in the image coordinate system.

Alternatively, referring to fig. 5, the vehicle device may detect the position information, the target area, and the confidence of each image target through operations 3011 to 3015, respectively:

3011: and detecting a first video image currently shot by a vehicle-mounted camera of the vehicle through a first convolutional neural network to obtain the position information and the target area of each image target with the type of the image target being the designated type.

Referring to fig. 6, the first Convolutional Neural Network includes a Convolutional Neural Network (CNN), a Region candidate Network (RPN), a Region-based Convolutional Neural Network (RCNN), and the like, where a first category set is set in the RCNN in advance, and the categories of the first category set may include designated categories, for example, the designated categories may be vehicles, the first category set may also include other categories such as houses, trees, flower stands, and street lamps.

The method comprises the following steps: firstly, when receiving a first video image input by a vehicle-mounted camera, a vehicle-mounted device inputs the first image into a first convolution neural network, and acquires the position information and the area of each target in an image coordinate system in the first video image output by the first convolution neural network and the probability of each target belonging to each category in a first category set.

In implementation, referring to fig. 6, the vehicle-mounted device may input the first video image to the CNN, perform convolution on the first video image by the CNN to obtain the feature map, obtain the first feature map corresponding to the first video image, and input the first feature map to the RPN and the RCNN, respectively. The RPN determines at least one first candidate box region in the first profile to obtain a second profile, which is also input to the RCNN. And the RCNN performs regression on any second feature map, namely the target position and the target category on the target feature in any first candidate frame according to the first feature map, and finally obtains the position information and the target area of the target object in the first candidate frame area and the probability that each target object belongs to each category in the first category set. The vehicle-mounted device acquires the position information and the target area of each target output by the RCNN and the probability that each target belongs to each category in the first category set.

Then, for each target, the vehicle-mounted equipment selects the maximum probability from the probabilities of each category corresponding to the target, takes the category corresponding to the maximum probability as the category of the target, and then selects the target with the category as the specified category from each target as the image target.

Wherein the target area of the image target may be different from the actual area of the image target.

Optionally, the RCNN may further output an image object frame corresponding to each object, that is, each image object may further have a corresponding image object frame, and the image object frame corresponding to the image object includes an image of the image object.

3012: and acquiring the classification confidence of each image target according to the video image shot by the vehicle-mounted camera before the current time.

In this step, K frames of video images recently shot before the current time may be acquired and form a video image set, for any image target, referred to as an image target to be processed for convenience of description, a confidence of the image target to be processed in each video image in the video image set is determined, and a classification confidence of the image target to be processed is acquired according to the confidence of the image target to be processed in each video image in the video image set, where K is a preset value. The classification confidence of the image target to be processed is used for representing the probability that the image target to be processed is a real vehicle on the image texture.

The video image set comprises a first video image, a second video image, … … and a Kth video image, wherein the first video image is the video image shot firstly in the video image set, and the Kth video image is the video image shot latest.

Optionally, for the operation of determining the confidence of the image target to be processed in each video image in the video image set, the implementation manner may be:

and calculating the distance between each image target in the first video image and each image target in the Kth video image according to the first position information of each image target in the first video image and the first position information of each image target in the Kth video image, and determining two image targets with the distance smaller than a preset distance threshold value as the same target in the two video images. It has been previously determined currently which objects in the kth video image and the (K-1) th video image are the same object, K being 2, 3, … …, K, so for the image object to be processed in the first video image it can be determined which video images in the set of video images include the image object to be processed and which video images do not include the image object to be processed. For a video image including an image object to be processed, the confidence level of the image object to be processed in the video image that has been currently acquired, and for a video image not including the image object to be processed, the confidence level of the image object to be processed in the video image may be set to a preset confidence level, for example, the preset confidence level may be a value such as 0 or 1.

Optionally, the vehicle-mounted terminal further stores a corresponding relationship between the target and the number of images. For each record in the correspondence, an object and the number of images including the object are stored in the record.

Accordingly, when it is determined that the image object in the first video image and the image object in the K-th video image are the same object, 1 is added to the number of images saved in the record including the image object in the correspondence relationship. When the image object in the first video image and any image object in the Kth video image are determined to be different, the image object can be a new object, the number of images including the image object is set to be 1, and the image object and the 1 are correspondingly stored in the corresponding relation.

Optionally, for the operation of obtaining the classification confidence of the image target to be processed, the classification confidence of the image target to be processed may be calculated according to the following first formula.

The first formula is:

wherein, in the first formula, C_cAs confidence of classification of the image object to be processed, C_c[k]For the confidence of the image target to be processed in the kth video image, γ is a damping coefficient, and is a preset value, for example, γ is 0.9.

3013: and obtaining the confidence coefficient of the tracking frame number of each image target according to the video image shot by the vehicle-mounted camera before the current time.

And acquiring the image number corresponding to the image target to be processed from the corresponding relation between the target and the image number, subtracting 1 from the image number to obtain the number of second video images including the image target to be processed, which are shot by the vehicle-mounted camera before the current time, and acquiring the confidence coefficient of the tracking frame number of the image target to be processed according to the number of the second video images including the image target to be processed, which are shot by the vehicle-mounted camera before the current time.

The confidence coefficient of the tracking frame number of the image target to be processed is used for representing the stability of the real target of which the image target to be processed is in a preset category.

Optionally, the confidence of the tracking frame number of the image target to be processed may be obtained according to the following second formula.

The second formula is:

wherein, in the second formula, C_TTo be treatedAnd the confidence of the tracking frame number of the image target, wherein T is the number of the second video images. In the second formula, when the number T of the second video images is less than or equal to 8, the confidence C of the tracking frame number of the image target to be processed_TWhen the number T of the second video images is more than 8, the confidence coefficient C of the tracking frame number of the image target to be processed is 0.2+ T/10_T＝1.0。

3014: and acquiring the position confidence of each image target according to the first video image.

In this step, for any image target in the first video image, for convenience of explanation, referred to as an image target to be processed, a distance between the image target to be processed and the vehicle is obtained according to the first position information of the image target to be processed, an image height corresponding to the image target to be processed and an occlusion ratio of a target frame corresponding to the image target to be processed are obtained from the first video image, the target frame includes an image corresponding to the image target to be processed, and a position confidence of the image target to be processed is obtained according to the distance, the image height and the occlusion ratio.

Alternatively, the position confidence of the image object to be processed may be calculated according to the following third formula.

The third formula is: c_p＝σ×y/h；

Wherein, in the third formula, C_pThe position confidence of the image target to be processed is obtained, h is the image height corresponding to the image target to be processed, y is the distance between the image target to be processed and the vehicle, and sigma is the shielded proportion of a target frame corresponding to the image target to be processed.

The position confidence of the image object depends on the ratio of the position information of the image object in the first video image to the occlusion of the image object: when the position information of the image target is closer to the bottom end of the first video image, the position confidence of the image target is higher; when the proportion of the image target which is occluded by other targets is smaller, the position confidence of the image target is higher.

3015: and respectively acquiring the confidence coefficient of each image target according to at least one of the classification confidence coefficient, the position confidence coefficient and the tracking frame number confidence coefficient of each image target.

For each image target, calculating the confidence of the image target in the first video image according to the following fourth formula according to the classification confidence, the position confidence and the tracking frame number confidence of the image target.

The fourth formula is: c_s＝f(C_c,C_p,C_T)＝W_c×C_c+W_p×C_p+W_T×C_T；

Wherein, in the fourth formula, C_sAs confidence of image object, W_cWeight coefficient, W, for classification confidence of image objects_pWeight coefficient, W, for the position confidence of an image object_TAnd the three weight coefficients are preset numerical values and are weight coefficients of confidence coefficients of tracking frame numbers of the image target.

The accuracy of the image target with the confidence coefficient exceeding the first preset threshold value detected by the vehicle-mounted camera is higher, and the accuracy of the image target with the confidence coefficient not exceeding the first preset threshold value is lower. Therefore, in the present embodiment, an image target whose confidence exceeds the first preset threshold may be determined as a detected real target of a specified category.

Step 302: position information of each radar target around the vehicle in a radar coordinate system and a confidence of each radar target are detected from a speed range image provided by a radar of the vehicle.

The radar has a radar coordinate system, and the detected position information of each radar target is position information in the radar coordinate system.

Optionally, according to the confidence of each detected radar target, classifying the radar target whose confidence exceeds a second preset threshold to obtain a target class of a real target corresponding to the radar target, and outputting the target class.

Alternatively, referring to fig. 7, this step may be implemented by the following operations 3021 to 3022, respectively:

3021: the position information, the area and the echo energy intensity of each radar target of which the target class around the vehicle is a specified class are detected by the radar of the vehicle.

The radar may transmit a radar wave to the surroundings of the vehicle, the radar wave may be reflected by objects around the vehicle to form reflected waves, and the radar may receive each reflected wave and obtain an echo energy intensity of each reflected wave. The position of each reflection point can be calculated according to the echo energy intensity of each reflection wave, an energy map can be drawn according to the position of each reflection point and the echo energy intensity, the energy map can be a speed distance image, the energy map comprises at least one energy block, each energy block comprises the echo energy intensity and the position corresponding to a plurality of reflection points with continuous positions, and each energy block is a target.

Based on each energy block, an object class and an area of an object corresponding to each energy block may be determined. And determining the target with the target category as the specified category as the radar target, finding out an extreme point of the echo energy intensity from an energy block of the radar target, and determining the position of the extreme point as the position information of the radar target in a radar coordinate system. The average value of the echo energy intensities in the energy blocks of the radar target can be used as the echo energy intensity of the radar target, or the echo energy intensities in the energy blocks of the radar target are sorted, and the echo energy intensity arranged in the middle position is selected as the echo energy intensity of the radar target.

The area of the radar target detected by the radar is the actual area of the radar target.

3022: and respectively acquiring the confidence coefficient of each radar target according to the position information and the echo energy intensity of each radar target.

In this step, the farthest distance that can be detected by the radar and the maximum echo energy intensity of the radar are obtained, for each radar target, the distance between the radar target and the vehicle is calculated according to the position information of the radar target, and the confidence of the radar target is calculated according to the following fifth formula according to the farthest distance, the maximum echo energy intensity, the calculated distance, and the echo energy intensity of the radar target.

The fifth formula is: c ═ a × d/d_max+b×p/p_max；

In the fifth formula, C is the confidence of the radar target, a is the weight coefficient of the radar target position, a may be greater than 0 and less than 1, d is the calculated distance, d is_maxB is a weight coefficient of the radar target echo energy intensity, b can be more than 0 and less than 1, p is the radar target echo energy intensity, p is_maxThe maximum echo energy intensity.

The accuracy of the radar target with the confidence coefficient exceeding the second preset threshold value detected by the radar is higher, and the accuracy of the radar target with the confidence coefficient not exceeding the second preset threshold value is lower. Therefore, in the present embodiment, a radar target whose confidence exceeds the second preset threshold may be determined as a detected real target of a specified category.

The execution sequence between the step 301 and the step 302 is not consecutive, and the step 301 may be executed first and then the step 302 is executed, or the step 301 may be executed first and then the step 301 may be executed, or the step 301 and the step 302 may be executed simultaneously.

Step 303: and acquiring a first perspective matrix according to the position information of the image target with the confidence coefficient exceeding a first preset threshold and the position information of the radar target with the confidence coefficient exceeding a second preset threshold, wherein the first perspective matrix is used for representing the conversion relation between the image coordinate system and the preset road coordinate system. The first and second preset thresholds may be the same or different.

Optionally, before executing this step, N associated target pairs are determined from the image target whose confidence coefficient exceeds the first preset threshold and the radar target whose confidence coefficient exceeds the second preset threshold, where any associated target pair includes a radar target and an image target that satisfy a preset association condition, and N is a positive integer greater than or equal to 1.

Referring to fig. 8, the process of determining N associated target pairs may include the following operations 3031 to 3033, the operations 3031 to 3033 being respectively:

3031: and mapping the first image target from the image coordinate system to the road coordinate system according to the first position information of the first image target and the stored second perspective matrix to obtain third position information corresponding to the first image target in the road coordinate system.

The first image target is an image target with a confidence coefficient exceeding a first preset threshold, and the first position information of the first image target is the position information of the first image target in an image coordinate system.

The vehicle-mounted terminal locally stores a second perspective matrix which is acquired last time and is used for reflecting a conversion relation between an image coordinate system of the vehicle-mounted camera and a preset road coordinate system.

The road coordinate system is different from the image coordinate system of the vehicle-mounted camera and another coordinate system except the radar coordinate system of the radar, the position of a certain point in the vehicle type can be used as a coordinate origin in the road coordinate system, the direction taking the vehicle advancing direction as a horizontal axis is used, and the vertical axis is vertical to the vehicle advancing direction.

Alternatively, the center point of the front bumper of the vehicle may be set as the origin of coordinates of the road coordinate system.

In this step, the first position information of each first image target with the confidence coefficient exceeding the first preset threshold is formed into a first matrix, and a second matrix is obtained according to a sixth formula, wherein the second matrix comprises third position information of each first image target with the confidence coefficient exceeding the first preset threshold in a road coordinate system.

The sixth formula is: a1 × B1 ═ C1; wherein A1 is the first matrix, B1 is the second perspective matrix, and C1 is the second matrix.

Optionally, the target area of each first image target whose confidence exceeds the preset threshold may be converted by using the second perspective matrix to obtain the projection area of each first image target in the road coordinate system, where the projection area of each first image target in the road coordinate system is the actual area.

3032: and mapping the first radar target from the radar coordinate system to the road coordinate system according to the second position information of the first radar target and the stored third perspective matrix to obtain corresponding fourth position information of the first radar target in the road coordinate system.

The first radar target is a radar target with the confidence coefficient exceeding a second preset threshold value, and the second position information is the position information of the first radar target in a radar coordinate system.

In this step, the second position information of each first radar target with the confidence coefficient exceeding the second preset threshold value forms a third matrix, and a fourth matrix is obtained according to a seventh formula, wherein the fourth matrix comprises fourth position information of each second radar target with the confidence coefficient exceeding the second preset threshold value in a road coordinate system.

The seventh formula is: a2 × B2 ═ C2; wherein, A2 is the third matrix, B2 is the third perspective matrix, and C2 is the fourth matrix.

The area of each radar target with the confidence coefficient exceeding the second preset threshold is the actual area of the radar target and is equal to the projection area of each first radar target in the road coordinate system, so that the area of each first radar target does not need to be converted through a third perspective matrix.

3033: and performing position association on each first image target and each first radar target according to the third position information and the projection area of each first image target and the fourth position information and the projection area of each first radar target to obtain N associated target pairs.

Optionally, the N associated target pairs may be determined through the following first and second steps, which are respectively:

the first step is as follows: and establishing a cost matrix according to the third position information and the projection area of the first image target and the fourth position information and the projection area of each radar target, wherein each first image target corresponds to one row in the cost matrix, each first radar target corresponds to one column of the cost matrix, the row corresponding to the first image target comprises a cost coefficient between the first image target and each first radar target respectively, and the cost coefficients of the first image target and the first radar target represent the probability that the first image target and the first radar target are the same target.

For example, assume that N first image objects have a confidence level exceeding a first preset threshold and N second image objects have a confidence level exceeding a second preset thresholdThe first radar target of value is X, so that the cost matrix established comprises N rows and X columns. For the ith first image target and the jth first radar target, i is 1, 2, … …, N, j is 1, 2, … …, X, according to the third position information and the projection area of the ith first image target in the road coordinate system

And fourth position information and projection area of the jth first radar target in the road coordinate system

Calculating the projection overlapping area S between the ith first image target and the jth first radar target_ij。

Then according to the projection area of the ith first image target in the road coordinate system

Projection area of jth first radar target in road coordinate system

And a projected overlap area S between the ith first image target and the jth first radar target_ijCalculating a cost coefficient d between the ith first image target and the jth first radar target according to an eighth formula_ij. The cost coefficient d between the ith first image target and the jth first radar target is calculated_ijAs an element of the ith row and the jth column of the cost matrix.

The eighth formula is:

the second step is that: and selecting a maximum cost coefficient from a line of cost coefficients corresponding to the first image target, and forming a pair of associated target pairs by the first image target and the first radar target corresponding to the maximum cost coefficient.

Optionally, after the associated target pair is obtained, the first perspective matrix is determined according to the position information of the radar target and the image target in the N associated target pairs.

Optionally, for any one of the N associated target pairs, modifying the position information of the first image target in the associated target pair according to the position information of the first radar target in the associated target pair; and correcting the second perspective matrix according to the corrected position information of each first image target in the N associated target pairs to obtain a first perspective matrix.

The probability that the first radar target detected by the radar is a real target of a specified category is higher than the probability that the first image target detected by the vehicle-mounted camera is a real target of a specified category. Therefore, in this step, for each associated target pair, the first image target and the first radar target included in the associated target pair are the same target, the third position information of the first image target may be corrected to the fourth position information of the first radar target, so as to obtain the fourth position information of each first image target. And constructing a fifth matrix according to the fourth position information of each first image object, and acquiring a first perspective matrix according to the fifth matrix and the first matrix formed by the first position information of each first image object according to a ninth formula.

The ninth formula is: a1 × B3 ═ C3; wherein C3 is the fifth matrix and B3 is the first perspective matrix.

The first image target with the confidence coefficient exceeding the first preset threshold and the first radar target with the confidence coefficient exceeding the second preset threshold are detected real targets in the designated category. Therefore, in this step, the first perspective matrix is obtained according to the position information of the first image target with the confidence coefficient exceeding the first preset threshold and the position information of the second radar target with the confidence coefficient exceeding the second preset threshold, so that the first perspective matrix can reflect the conversion relation between the image coordinate system and the road coordinate system of the vehicle-mounted camera driven by the vehicle under the current road condition.

Optionally, the second perspective matrix saved by the vehicle-mounted device may be updated to the first perspective matrix.

Step 304: and detecting real targets with the specified category from the image targets with the confidence coefficient not exceeding a first preset threshold value and the radar targets with the confidence coefficient not exceeding a second preset threshold value through the first perspective matrix.

Alternatively, referring to fig. 9 and fig. 10, this step may be implemented by the operations 3041 to 3043, which are respectively:

3041: and mapping the position information of each second image target of which the confidence coefficient does not exceed a first preset threshold value into a road coordinate system through the first perspective matrix to obtain the corresponding position information of each second image target in the road coordinate system.

In this step, the position information of each second image target whose confidence coefficient does not exceed the first preset threshold value is formed into a sixth matrix, and a seventh matrix is obtained according to a tenth formula, where the seventh matrix includes the position information of each second image target whose confidence coefficient does not exceed the first preset threshold value in the road coordinate system.

The tenth formula is: a5 × B3 ═ C4; wherein, A5 is the sixth matrix, B3 is the first perspective matrix, and C4 is the seventh matrix.

Optionally, the target area of each second image target whose confidence does not exceed the first preset threshold may also be converted by the first perspective matrix to obtain the projection area of each second image target in the road coordinate system, where the projection area of each second image target in the road coordinate system is the actual area.

3042: and mapping each second radar target of which the confidence coefficient does not exceed a second preset threshold value into the road coordinate system through the third perspective matrix to obtain corresponding position information of each second radar target in the road coordinate system.

In this step, the position information of each second radar target whose confidence coefficient does not exceed the second preset threshold value forms an eighth matrix, and a ninth matrix is obtained according to an eleventh formula, where the ninth matrix includes the position information of each second radar target whose confidence coefficient does not exceed the second preset threshold value in the road coordinate system.

The eleventh formula is: a6 × B2 ═ C5; wherein, A6 is the eighth matrix, B2 is the third perspective matrix, and C2 is the ninth matrix.

The area of each second radar target of which the confidence coefficient does not exceed the second preset threshold is the actual area of the radar target and is equal to the projection area of each second radar target in the road coordinate system, so that the area of each second radar target does not need to be converted through the first perspective matrix.

3043: and determining M characteristic fusion target pairs according to the position information and the projection area of each second image target in the road coordinate system, the confidence coefficient of which does not exceed the first preset threshold, and the position information and the projection area of each second radar target in the road coordinate system, the confidence coefficient of which does not exceed the second preset threshold, wherein the radar target pairs comprise image targets and radar targets which are the same target.

Optionally, the M feature fusion target pairs may be determined through the following first and second steps, which are respectively:

the first step is as follows: establishing a second price matrix according to the position information and the projection area of each second image target in the road coordinate system, the confidence of which does not exceed the first preset threshold, and the position information and the projection area of each second radar target in the road coordinate system, the confidence of which does not exceed the second preset threshold, wherein each second image target, the confidence of which does not exceed the first preset threshold, corresponds to one row in the second price matrix, each second radar target, the confidence of which does not exceed the second preset threshold, corresponds to one column in the second price matrix, one row corresponding to the second image target comprises second cost coefficients between the image target and each radar target with the confidence coefficient not exceeding a preset threshold value, and the second cost coefficient of the second image target and the second radar target represents the probability that the second image target with the confidence coefficient not exceeding the first preset threshold value and the second radar target with the confidence coefficient not exceeding the second preset threshold value are the same target.

For example, assume that there are M second image targets whose confidence levels do not exceed the first preset threshold, and Y second radar targets whose confidence levels exceed the second preset threshold, and the cost matrix thus established includes M rows and Y columns. For the pth second image target and the qth second radar target, p is 1, 2, … …, M, q is 1, 2, … …, Y, according to which the pth second image target is in the road coordinate systemPosition information and projected area S of_p ^CAnd the position information and the projected area S of the q-th second radar target in the road coordinate system_q ^RCalculating the projection overlapping area S between the pth second image target and the qth second radar target_pq。

Then according to the projection area S of the p-th second image target in the road coordinate system_p ^CThe projection area S of the qth second radar target in the road coordinate system_q ^RAnd a projected overlap area S between the pth second image target and the qth second radar target_pqCalculating a cost coefficient d between the p-th second image target and the q-th second radar target according to the twelfth formula_pq. The cost coefficient d between the p second image target and the q second radar target_pqAs an element of the p-th row and q-th column of the cost matrix.

The twelfth formula is: d_pq＝S_pq/(S_p ^C+S_q ^R－S_pq)。

The second step is that: and selecting the maximum second cost coefficient from a line of second cost coefficients corresponding to the second image target, and forming a feature fusion target pair by the second image target and the second radar target corresponding to the maximum second cost coefficient.

3044: and detecting a real target with the target class being the designated class from the M feature fusion target pairs through a second convolutional neural network.

The vehicle-mounted equipment comprises a second convolutional neural network, wherein a second category set is arranged in the second convolutional neural network in advance, and the categories of the second category set can comprise a specified category, a non-specified category and other categories. For example, assume that the designated category is a vehicle or a motor vehicle and the non-preset category may be a non-motor vehicle.

For each feature fusion target pair, the feature fusion target pair includes a second image target and a second radar target, the vehicle-mounted device may input an image corresponding to the second image target in the first video image and an energy block corresponding to the second radar target to a second convolutional neural network, as shown in fig. 10, extract energy features from image features extracted from an image of the second image target and the energy block of the second radar target through the second convolutional neural network, splice the image features and the energy features, obtain a feature sequence after splicing, and output a probability that the feature fusion target pair belongs to each category in the second category set after performing multilayer convolutional calculation and full connection layer calculation according to the feature sequence through the second convolutional neural network; and selecting the class with the highest probability as the target class of the feature fusion target pair, and when the target class of the feature fusion target pair is the designated class, taking the second radar target in the feature fusion target pair as a real target of which the detected class is the designated class. Of course, the second convolutional network may also be formed by a plurality of sub-networks, which respectively perform: image features extracted from an image of a second image target; extracting echo energy features from an energy block of a second radar target; splicing the image characteristics and the energy characteristics to obtain a characteristic sequence; and according to the feature sequence, performing multilayer convolution calculation and full-connection layer calculation, and outputting the probability that the feature fusion target pair belongs to each class in the second class set.

When the type of the feature fusion target pair is the designated type, it is indicated that the second image target in the feature fusion target pair and the target type of the second radar target are both the designated type, that is, the vehicle-mounted camera and the radar simultaneously detect the real target of which the same target is the designated type, and since the radar detection precision is higher than that of the vehicle-mounted camera, the second radar target in the feature fusion target pair is used as the real target of which the detected target type is the designated type.

Optionally, the second convolutional neural network is obtained by training a sample set in advance, where the sample set includes a plurality of preset feature fusion target pairs and a target class corresponding to each feature fusion target. In training, the sample set is input to a second convolutional neural network for training.

The beneficial effects of the embodiment of the application are as follows: detecting image targets around the vehicle and confidence degrees of the image targets through a vehicle-mounted camera, and detecting radar targets around the vehicle and confidence degrees of the radar targets through a radar; because the first image target with the confidence coefficient exceeding the first preset threshold and the first radar target with the confidence coefficient exceeding the second preset threshold are real targets of a specified category, the first perspective matrix obtained according to the first image target with the confidence coefficient exceeding the first preset threshold and the first radar target with the confidence coefficient exceeding the second preset threshold can reflect the conversion relation between the image coordinate system and the road coordinate system of the vehicle-mounted camera of the vehicle under the current road condition, so that the accuracy of the real target with the target category as the specified category detected from the second image target with the confidence coefficient not exceeding the first preset threshold and the second radar target with the confidence coefficient not exceeding the second preset threshold according to the first perspective matrix is higher, and the accuracy of the target detection is improved.

The following are embodiments of the apparatus of the present application that may be used to perform embodiments of the method of the present application. For details which are not disclosed in the embodiments of the apparatus of the present application, reference is made to the embodiments of the method of the present application.

Referring to fig. 11, an embodiment of the present application provides a target detection device 400 in which an on-vehicle camera and an on-vehicle radar are linked, where the device 400 includes:

a first detection module 401, configured to detect, from a video image provided by a vehicle-mounted camera, each image object around a vehicle, a confidence of each image object, and position information of each image object in an image coordinate system;

a second detection module 402, configured to detect, from a speed and distance image provided by the radar, radar targets around the vehicle and position information of the radar targets in a radar coordinate system, where the confidence of the radar targets is used to indicate a probability that a target category of a real target corresponding to the image target or the radar target is a specified category;

an obtaining module 403, configured to obtain a first perspective matrix according to position information of an image target whose confidence coefficient exceeds a first preset threshold and position information of a radar target whose confidence coefficient exceeds a second preset threshold, where the first perspective matrix is used to represent a conversion relationship between an image coordinate system and a preset road coordinate system;

a third detecting module 404, configured to detect, through the first perspective matrix, a target class from the image target whose confidence does not exceed the first preset threshold and the radar target whose confidence does not exceed the second preset threshold.

Optionally, the apparatus 400 further includes:

Optionally, the determining module is configured to:

Optionally, the obtaining module 403 is configured to:

Optionally, the third detecting module 404 is configured to:

Optionally, the first detecting module 401 is configured to:

the second detecting module 402 is configured to:

The beneficial effects of the embodiment of the application are as follows: the first detection module detects image targets around the vehicle and confidence degrees of the image targets through a vehicle-mounted camera, and the second detection module detects radar targets around the vehicle and the confidence degrees of each radar target through a radar; because the image target with the confidence coefficient exceeding the first preset threshold and the radar target with the confidence coefficient exceeding the second preset threshold are real targets of the appointed category, the first perspective matrix acquired by the acquisition module for the real targets of the appointed category according to the image target with the confidence coefficient exceeding the first preset threshold and the radar target with the confidence coefficient exceeding the second preset threshold can reflect the conversion relation between the image coordinate system and the road coordinate system of the vehicle-mounted camera of the vehicle under the current road condition, so that the third detection module detects the real targets of the appointed category from the image target with the confidence coefficient not exceeding the first preset threshold and the radar target with the confidence coefficient not exceeding the second preset threshold according to the first perspective matrix, and the precision of the real targets of the appointed category is higher, and the precision of the target detection is improved.

With regard to the apparatus in the above-described embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.

Fig. 12 shows a block diagram of a terminal 500 according to an exemplary embodiment of the present application. The terminal 500 may be a vehicle-mounted terminal, and the terminal 500 may also be referred to as user equipment, a portable terminal, or other names.

In general, the terminal 500 includes: a processor 501 and a memory 502.

The processor 501 may include one or more processing cores, such as a 4-core processor, an 8-core processor, and so on. The processor 501 may be implemented in at least one hardware form of a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), and a PLA (Programmable Logic Array). The processor 501 may also include a main processor and a coprocessor, where the main processor is a processor for processing data in an awake state, and is also called a Central Processing Unit (CPU); a coprocessor is a low power processor for processing data in a standby state. In some embodiments, the processor 501 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content required to be displayed on the display screen. In some embodiments, processor 501 may also include an AI (Artificial Intelligence) processor for processing computational operations related to machine learning.

Memory 502 may include one or more computer-readable storage media, which may be non-transitory. Memory 502 may also include high-speed random access memory, as well as non-volatile memory, such as one or more magnetic disk storage devices, flash memory storage devices. In some embodiments, a non-transitory computer readable storage medium in memory 502 is used to store at least one instruction for execution by processor 501 to implement the camera and radar-based vehicle detection methods provided by the method embodiments herein.

In some embodiments, the terminal 500 may further optionally include: a peripheral interface 503 and at least one peripheral. The processor 501, memory 502 and peripheral interface 503 may be connected by a bus or signal lines. Each peripheral may be connected to the peripheral interface 503 by a bus, signal line, or circuit board. Specifically, the peripheral device includes: at least one of radio frequency circuitry 504, touch screen display 505, camera 506, audio circuitry 507, positioning components 508, and power supply 509.

The peripheral interface 503 may be used to connect at least one peripheral related to I/O (Input/Output) to the processor 501 and the memory 502. In some embodiments, the processor 501, memory 502, and peripheral interface 503 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 501, the memory 502, and the peripheral interface 503 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.

The Radio Frequency circuit 504 is used for receiving and transmitting RF (Radio Frequency) signals, also called electromagnetic signals. The radio frequency circuitry 504 communicates with communication networks and other communication devices via electromagnetic signals. The rf circuit 504 converts an electrical signal into an electromagnetic signal to transmit, or converts a received electromagnetic signal into an electrical signal. Optionally, the radio frequency circuit 504 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a subscriber identity module card, and so forth. The radio frequency circuitry 504 may communicate with other terminals via at least one wireless communication protocol. The wireless communication protocols include, but are not limited to: the world wide web, metropolitan area networks, intranets, generations of mobile communication networks (2G, 3G, 4G, and 5G), Wireless local area networks, and/or WiFi (Wireless Fidelity) networks. In some embodiments, the rf circuit 504 may further include NFC (Near Field Communication) related circuits, which are not limited in this application.

The display screen 505 is used to display a UI (User Interface). The UI may include graphics, text, icons, video, and any combination thereof. When the display screen 505 is a touch display screen, the display screen 505 also has the ability to capture touch signals on or over the surface of the display screen 505. The touch signal may be input to the processor 501 as a control signal for processing. At this point, the display screen 505 may also be used to provide virtual buttons and/or a virtual keyboard, also referred to as soft buttons and/or a soft keyboard. In some embodiments, the display screen 505 may be one, providing the front panel of the terminal 500; in other embodiments, the display screens 505 may be at least two, respectively disposed on different surfaces of the terminal 500 or in a folded design; in still other embodiments, the display 505 may be a flexible display disposed on a curved surface or on a folded surface of the terminal 500. Even more, the display screen 505 can be arranged in a non-rectangular irregular figure, i.e. a shaped screen. The Display screen 505 may be made of LCD (liquid crystal Display), OLED (Organic Light-Emitting Diode), and the like.

The camera assembly 506 is used to capture images or video. Optionally, camera assembly 506 includes a front camera and a rear camera. Generally, a front camera is disposed at a front panel of the terminal, and a rear camera is disposed at a rear surface of the terminal. In some embodiments, the number of the rear cameras is at least two, and each rear camera is any one of a main camera, a depth-of-field camera, a wide-angle camera and a telephoto camera, so that the main camera and the depth-of-field camera are fused to realize a background blurring function, and the main camera and the wide-angle camera are fused to realize panoramic shooting and VR (Virtual Reality) shooting functions or other fusion shooting functions. In some embodiments, camera assembly 506 may also include a flash. The flash lamp can be a monochrome temperature flash lamp or a bicolor temperature flash lamp. The double-color-temperature flash lamp is a combination of a warm-light flash lamp and a cold-light flash lamp, and can be used for light compensation at different color temperatures.

Audio circuitry 507 may include a microphone and a speaker. The microphone is used for collecting sound waves of a user and the environment, converting the sound waves into electric signals, and inputting the electric signals to the processor 501 for processing, or inputting the electric signals to the radio frequency circuit 504 to realize voice communication. For the purpose of stereo sound collection or noise reduction, a plurality of microphones may be provided at different portions of the terminal 500. The microphone may also be an array microphone or an omni-directional pick-up microphone. The speaker is used to convert electrical signals from the processor 501 or the radio frequency circuit 504 into sound waves. The loudspeaker can be a traditional film loudspeaker or a piezoelectric ceramic loudspeaker. When the speaker is a piezoelectric ceramic speaker, the speaker can be used for purposes such as converting an electric signal into a sound wave audible to a human being, or converting an electric signal into a sound wave inaudible to a human being to measure a distance. In some embodiments, audio circuitry 507 may also include a headphone jack.

The positioning component 508 is used to locate the current geographic position of the terminal 500 for navigation or LBS (location based Service). The positioning component 508 may be a positioning component based on the GPS (global positioning System) in the united states, the beidou System in china, or the galileo System in russia.

Power supply 509 is used to power the various components in terminal 500. The power source 509 may be alternating current, direct current, disposable or rechargeable. When power supply 509 includes a rechargeable battery, the rechargeable battery may be a wired rechargeable battery or a wireless rechargeable battery. The wired rechargeable battery is a battery charged through a wired line, and the wireless rechargeable battery is a battery charged through a wireless coil. The rechargeable battery may also be used to support fast charge technology.

In some embodiments, terminal 500 also includes one or more sensors 510. The one or more sensors 510 include, but are not limited to: acceleration sensor 511, gyro sensor 512, pressure sensor 513, fingerprint sensor 514, optical sensor 515, and proximity sensor 516.

The acceleration sensor 511 may detect the magnitude of acceleration on three coordinate axes of the coordinate system established with the terminal 500. For example, the acceleration sensor 511 may be used to detect components of the gravitational acceleration in three coordinate axes. The processor 501 may control the touch screen 505 to display the user interface in a landscape view or a portrait view according to the gravitational acceleration signal collected by the acceleration sensor 511. The acceleration sensor 511 may also be used for acquisition of motion data of a game or a user.

The gyro sensor 512 may detect a body direction and a rotation angle of the terminal 500, and the gyro sensor 512 may cooperate with the acceleration sensor 511 to acquire a 3D motion of the user on the terminal 500. The processor 501 may implement the following functions according to the data collected by the gyro sensor 512: motion sensing (such as changing the UI according to a user's tilting operation), image stabilization at the time of photographing, game control, and inertial navigation.

The pressure sensor 513 may be disposed on a side bezel of the terminal 500 and/or an underlying layer of the touch display screen 505. When the pressure sensor 513 is disposed on the side frame of the terminal 500, a user's holding signal of the terminal 500 may be detected, and the processor 501 performs left-right hand recognition or shortcut operation according to the holding signal collected by the pressure sensor 513. When the pressure sensor 513 is disposed at the lower layer of the touch display screen 505, the processor 501 controls the operability control on the UI interface according to the pressure operation of the user on the touch display screen 505. The operability control comprises at least one of a button control, a scroll bar control, an icon control and a menu control.

The fingerprint sensor 514 is used for collecting a fingerprint of the user, and the processor 501 identifies the identity of the user according to the fingerprint collected by the fingerprint sensor 514, or the fingerprint sensor 514 identifies the identity of the user according to the collected fingerprint. Upon recognizing that the user's identity is a trusted identity, the processor 501 authorizes the user to perform relevant sensitive operations including unlocking the screen, viewing encrypted information, downloading software, paying, and changing settings, etc. The fingerprint sensor 514 may be provided on the front, back, or side of the terminal 500. When a physical button or a vendor Logo is provided on the terminal 500, the fingerprint sensor 514 may be integrated with the physical button or the vendor Logo.

The optical sensor 515 is used to collect the ambient light intensity. In one embodiment, the processor 501 may control the display brightness of the touch display screen 505 based on the ambient light intensity collected by the optical sensor 515. Specifically, when the ambient light intensity is high, the display brightness of the touch display screen 505 is increased; when the ambient light intensity is low, the display brightness of the touch display screen 505 is turned down. In another embodiment, processor 501 may also dynamically adjust the shooting parameters of camera head assembly 506 based on the ambient light intensity collected by optical sensor 515.

A proximity sensor 516, also referred to as a distance sensor, is typically disposed on the front panel of the terminal 500. The proximity sensor 516 is used to collect the distance between the user and the front surface of the terminal 500. In one embodiment, when the proximity sensor 516 detects that the distance between the user and the front surface of the terminal 500 gradually decreases, the processor 501 controls the touch display screen 505 to switch from the bright screen state to the dark screen state; when the proximity sensor 516 detects that the distance between the user and the front surface of the terminal 500 becomes gradually larger, the processor 501 controls the touch display screen 505 to switch from the screen-rest state to the screen-on state.

Those skilled in the art will appreciate that the configuration shown in fig. 12 is not intended to be limiting of terminal 500 and may include more or fewer components than those shown, or some components may be combined, or a different arrangement of components may be used.

Referring to fig. 13, an embodiment of the present application provides an object detection system 600, which includes a radar 601 disposed on a vehicle, an onboard camera 602 disposed on the vehicle, and a detection device 603 communicating with the radar 601 and the onboard camera 602,

the vehicle-mounted camera 602 is configured to shoot the surroundings of the vehicle to obtain a current frame video image, and provide the shot current frame video image to the detection device;

the radar 601 is configured to generate a current frame rate range image according to a transmitted radar signal and a received echo signal, and provide the current frame rate range image to the detection device;

the detection device 603 is configured to detect, from the video image provided by the vehicle-mounted camera 602, each image object around the vehicle, a confidence of each image object, and position information of each image object in an image coordinate system; detecting each radar target around the vehicle and position information of each radar target in a radar coordinate system from a speed and distance image provided by the radar 601, wherein the confidence level of the radar target is used for representing the probability that the target class of the image target or a real target corresponding to the radar target is a specified class; acquiring a first perspective matrix according to the position information of the image target with the confidence coefficient exceeding a first preset threshold and the position information of the radar target with the confidence coefficient exceeding a second preset threshold, wherein the first perspective matrix is used for representing the conversion relation between the image coordinate system and a preset road coordinate system; and detecting a target class from the image target with the confidence coefficient not exceeding the first preset threshold and the radar target with the confidence coefficient not exceeding the second preset threshold through the first perspective matrix.

Optionally, the vehicle-mounted camera 602 is disposed at the front and rear and/or left and right sides of the vehicle, and the radar 601 is disposed at the front and rear of the vehicle.

Optionally, the radar 601 is a millimeter wave radar.

Other embodiments of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. This application is intended to cover any variations, uses, or adaptations of the invention following, in general, the principles of the application and including such departures from the present disclosure as come within known or customary practice within the art to which the invention pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the application being indicated by the following claims.

It will be understood that the present application is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the application is limited only by the appended claims.

Claims

1. A target detection method for linkage of a vehicle-mounted camera and a vehicle-mounted radar is characterized by comprising the following steps:

2. The method of claim 1, wherein the method further comprises:

3. The method of claim 1, wherein before the obtaining the first perspective matrix according to the position information of the image target whose confidence exceeds the first preset threshold and the position information of the radar target whose confidence exceeds the second preset threshold, the method further comprises:

4. The method of claim 3, wherein determining N associated target pairs from the image target with the confidence level exceeding a first preset threshold and the radar target with the confidence level exceeding a second preset threshold comprises:

5. The method of claim 4, wherein said positionally associating each of said first image targets with each of said second radar targets to obtain said N associated target pairs comprises:

6. The method of claim 4, wherein determining the first perspective matrix from position information of radar targets and image targets in the N correlated target pairs comprises:

7. The method of any one of claims 1 to 6, wherein said detecting, by said first perspective matrix, a target class from image targets having a confidence level that does not exceed said first preset threshold and radar targets having a confidence level that does not exceed said second preset threshold comprises:

8. The method of claim 7, wherein determining M feature fusion target pairs from the second image target and the second radar target comprises:

9. The method of claim 1, wherein detecting the confidence level of each image object around the vehicle from the video image provided by the onboard camera comprises:

10. The utility model provides a target detection device of on-vehicle camera and on-vehicle radar linkage which characterized in that, the device includes:

11. The apparatus of claim 10, wherein the apparatus further comprises:

12. The apparatus of claim 10, wherein the apparatus further comprises:

13. The apparatus of claim 12, wherein the determination module is to:

14. The apparatus of claim 13, wherein the acquisition module is to:

15. The apparatus of any of claims 10 to 14, wherein the third detection module is to:

16. The apparatus of claim 15, wherein the third detection module is to:

17. The apparatus of claim 10, wherein the first detection module is to:

the second detection module is configured to:

18. An object detection system comprising a radar arranged on a vehicle, an on-board camera arranged on the vehicle, and a detection device in communication with the radar and the on-board camera,

19. The system of claim 18, wherein the onboard camera is disposed at a front and rear portion and/or left and right sides of the vehicle, and the radar is disposed at the front and rear portion of the vehicle.

20. The system of claim 18 or 19, wherein the radar is a millimeter wave radar.