CN115503747A

CN115503747A - Road condition identification and reminding system based on intelligent automobile steer-by-wire system

Info

Publication number: CN115503747A
Application number: CN202211182285.6A
Authority: CN
Inventors: 郑宏宇; 陈超宁; 靳立强; 肖峰; 刘哲
Original assignee: Jilin University
Current assignee: Jilin University
Priority date: 2022-09-27
Filing date: 2022-09-27
Publication date: 2022-12-23

Abstract

The invention discloses a road condition identification and reminding system based on an intelligent automobile steer-by-wire system, which comprises a road condition information acquisition module, a road condition identification module, a driver reminding control module and a driver reminding execution module, wherein the road condition is respectively identified through the combination of a binocular camera, a laser radar and a series of vehicle sensors under different light intensity and working condition conditions, and the identification results are fused to obtain the road condition information in front of a vehicle; and the front road condition information is provided for the driver in advance through various modes of animation display, steering wheel vibration, voice reminding and signal lamp flashing so that the driver can make a correct driving decision, traffic accidents are avoided, and the life and property safety of the driver is guaranteed.

Description

Road condition identification and reminding system based on intelligent automobile steer-by-wire system

The technical field is as follows:

the invention belongs to the field of intelligent automobile identification, relates to the technology of environment perception and driver reminding, and particularly relates to a road condition identification and reminding system based on an intelligent automobile steer-by-wire system.

Background art:

with the increasing of the automobile keeping quantity, the number of drivers is more and more, but the driving levels of different drivers are uneven, and the number of traffic accidents caused by the factors of the drivers is increased day by day. The intelligent degree of the intelligent automobile depends on the sensing, planning, decision-making and automatic execution capabilities, the sensing capability is the basis of all intelligence of the automobile and serves as eyes of the intelligent automobile, the sensing algorithm is the only information source capable of picking up and reading actual road conditions and detecting and identifying potential dangers, the sensing algorithm reads the local road conditions where the intelligent automobile is located through a sensor technology and makes decisions on the basis so that the intelligent automobile can complete driving tasks. Currently, the local traffic information mainly refers to lanes and their boundaries, obstacles on the road surface, pedestrians, and the like. In addition, the detection, identification and avoidance of the abnormal protrusion and the abnormal depression of the road surface are only lacked. The abnormal depressions and the abnormal bulges of the road surface not only bring huge financial expenses and a large number of traffic accidents to each country, but also seriously threaten the lives of drivers and passengers. With the increasing number of vehicle users in recent years, drivers and passengers die at home and abroad due to traffic accidents caused by abnormal depressions and abnormal bulges on the road surface. It can be seen that the detection and avoidance of road surface abnormal depressions and road surface abnormal elevations using advanced techniques and methods has become increasingly urgent. In addition, the road shoulder and the deceleration strip are common constituent elements of the road surface and generally not taken care of in the intelligent automobile sensing research, but when the speed of the vehicle passing through the road shoulder and the deceleration strip is too high, the destructive power of the vehicle structure is huge, so that the road shoulder and the deceleration strip need to be taken care of.

And the driver makes driving decisions under the condition of fully knowing the road condition in front of the vehicle, so as to generate driving operation. If the driver does not grasp the front road condition information sufficiently, the information of abnormal depressions on the road surface, abnormal bulges on the road surface, deceleration strips, road shoulders, pedestrians, non-motor vehicles and the like are omitted, so that the driver can mistakenly drive or fail to decelerate in time and take avoidance measures, therefore, the driver needs to be provided with reminding information of the front vehicle condition, the driver can grasp the front road condition sufficiently in time, traffic accidents are avoided, and the life and property safety of the driver is guaranteed.

The steer-by-wire system is considered to be a necessary assembly for realizing the automatic driving of the intelligent automobile, the mechanical connection between a steering wheel and a steering wheel is eliminated, the decoupling between the operation of the steering wheel and the rotation of the steering wheel is realized, and a great design space is provided for the transmission ratio and the control of a steering wheel motor. The control of a steering wheel road feel motor is one of the main research contents of steer-by-wire, and the current research focuses on designing road feel feedback torque, which is to provide the road feel feedback torque for a driver, such a feedback torque has a problem in that it has no advance notice function and the driver cannot know the road condition ahead because the road feel feedback torque is a feedback of the road condition on which the wheels are currently passing, not the road condition on the road surface ahead of the vehicle. The operation of the driver can be further adjusted only by enabling the driver to know the road condition ahead in advance, so that the driver can be informed of the road condition ahead in advance, and the steering-by-wire technology is based on, so that the steering wheel road feel motor can be conveniently used for providing the road condition ahead information for the driver.

The invention content is as follows:

in order to solve the technical problems, the invention provides a road condition identification and reminding system based on an intelligent automobile steer-by-wire system, which is used for identifying road conditions respectively through the combination of a binocular camera, a laser radar and a series of vehicle sensors under different light intensities and working conditions, and fusing identification results to obtain road condition information in front of a vehicle. And the front road condition information is provided for the driver in advance through various modes of animation display, steering wheel vibration, voice reminding and signal lamp flashing so that the driver can make a correct driving decision, traffic accidents are avoided, and the life and property safety of the driver is guaranteed.

In order to achieve the purpose, the invention is realized according to the following technical scheme:

a road condition identification and reminding system based on an intelligent automobile steer-by-wire system comprises a road condition information acquisition module, a road condition identification module, a driver reminding control module and a driver reminding execution module; the road condition identification module is respectively connected with the road condition information acquisition module and the driver reminding control module; the driver reminding control module is also connected with the driver reminding execution module;

technical scheme road condition information acquisition module include two binocular cameras, a long-range laser radar, 17 vehicle sensor, a light sensor, 17 vehicle sensor include: four suspension displacement sensors, a front axle height sensor, a rear axle height sensor, an inertia measuring unit, a yaw angle sensor, a three-axis acceleration sensor, four vehicle body acceleration sensors and four tire pressure detection sensors, which are respectively positioned at the left front part, the left rear part, the right front part and the right rear part of the vehicle; the binocular camera and the long-distance laser radar are respectively used for acquiring an image and a three-dimensional point cloud of a front road surface; the suspension displacement sensors are respectively used for acquiring vertical displacement signals of the suspension; the front axle and the rear axle height sensors are respectively used for acquiring a ground clearance signal of the front axle and a ground clearance signal of the rear axle; the inertia measurement unit is used for acquiring pitching angular displacement and rolling angular displacement of the car body; the yaw angle sensor is used for acquiring a yaw velocity signal of the vehicle; the three-axis acceleration sensor is used for acquiring transverse acceleration, longitudinal acceleration and vertical acceleration signals of the vehicle; the four vehicle body acceleration sensors are respectively arranged at the positions 5 cm away from the top of the shock absorber on the left front vehicle body, the right front vehicle body, the left rear vehicle body and the right rear vehicle body and are respectively used for acquiring vertical vibration acceleration signals of the left front vehicle body, the right front vehicle body, the left rear vehicle body and the right rear vehicle body; the tire pressure detection sensors are respectively used for collecting tire pressure signals of left front tires, right front tires, left rear tires and right rear tires; the light sensor is used for acquiring an illumination intensity signal of the surrounding environment of the vehicle and comparing the illumination intensity signal with an illumination intensity threshold value; the illumination intensity threshold is calculated according to the following expression:

in the formula, L _uth Representing the threshold value of the intensity of illumination, k _pr Expressing the efficiency influence coefficient of the binocular camera, wherein the value range of the efficiency influence coefficient is 0 to 1; k is a radical of _ex Expressing the influence coefficient of radar efficiency, and the value range of the influence coefficient is 0 to 1; l is _uth0 The initial value of the illumination intensity is represented and can be taken within the range from 0.001 lux to 0.25 lux;

according to the technical scheme, the two binocular cameras are calibrated before being installed and are respectively installed at the upper left corner and the upper right corner of a windshield, the ground clearance of the installation position is hc, the horizontal distance between the two binocular cameras is Lc, and the hc and the Lc are related to the size parameter of the front windshield of the vehicle and are determined by a vehicle manufacturer; the visual distance of the two binocular cameras is 250 meters, and the left and right visual angle ranges are 120 degrees; the binocular cameras can rotate up and down, images in different angle ranges can be shot through the up-and-down rotation, the downward shooting angle of each binocular camera is a negative angle, the upward shooting angle of each binocular camera is a positive angle, the inclination angle of each binocular camera is Ac, the rotation angle range of each binocular camera is negative 30 degrees to positive 60 degrees, namely the downward maximum rotation angle is 30 degrees, the upward maximum rotation angle is 60 degrees, each two binocular cameras shoot a frame of image every 1 millisecond, the first frame shoots an image in the angle range of negative 30 degrees to 0 degrees, the second frame shoots an image in the angle range of negative 10 degrees to positive 20 degrees, the third frame shoots an image in the angle range of positive 10 degrees to positive 40 degrees, the fourth frame shoots an image in the angle range of positive 30 degrees to positive 60 degrees, the fifth frame shoots an image in the angle range of positive 10 degrees to positive 40 degrees, and the sixth frame shoots an image in the angle range of negative 10 degrees to positive 20 degrees; shooting the six frames of images in sequence to form a shooting cycle, and shooting the images by reciprocating rotation of the camera up and down according to the sequence of the shooting cycle; the long-distance laser radar is installed at the middle position of the front bumper, the ground clearance of the installation position is hr, and hr is related to the size parameter of the front bumper of the vehicle and is determined by a vehicle manufacturer; the long-distance laser radar can rotate up and down to acquire point cloud data of environments with different angle ranges in a scanning mode, the downward scanning angle of the long-distance laser radar is recorded as a negative angle, the upward scanning angle is recorded as a positive angle, the inclination angle of the long-distance laser radar is recorded as Ar, the range of the up-and-down rotation angle of the long-distance laser radar is from negative 20 degrees to positive 10 degrees, namely the long-distance laser radar can rotate 20 degrees downwards maximally and 10 degrees upwards maximally, and the detection distance is 300 meters; the sequence of the long-distance laser radar scanning environment is as follows: firstly, sweeping once when the inclination angle is minus 20 degrees, then sweeping once when the inclination angle is 0 degrees, then sweeping once when the inclination angle is 10 degrees, then sweeping once when the inclination angle is 0 degrees, finally rotating to the position of minus 20 degrees, and reciprocating up and down to sweep according to the sequence; the long-distance laser radar generates three-dimensional point cloud data in the area in front of the vehicle every time of scanning, and each point cloud data is provided with a position label and a distance label: x, y, z, d; x, y, z and d respectively represent a horizontal axis coordinate, a vertical axis coordinate and a distance from the vehicle; the road condition information acquisition module inputs all acquired information into the road condition identification module;

the road condition identification module in the technical scheme identifies the road surface characteristics of the front road surface according to input information, identifies whether pedestrians, bicycles, motorcycles, abnormal depressions of the road surface, abnormal bulges of the road surface, speed bumps, lane lines and road shoulders exist on the front road surface, and inputs the identification result into a driver reminding control module; the road condition identification module is provided with three identification submodules, namely an image identification submodule, a three-dimensional point cloud identification submodule and a vehicle sensor signal identification submodule, and the specific identification process of each identification submodule is as follows:

the image recognition submodule performs road surface feature and target recognition according to the image acquired by the binocular camera, and the specific recognition process of the image recognition submodule is as follows:

step S1: respectively preprocessing each frame of image shot by the left camera and the right camera, firstly respectively removing the transverse distortion and the longitudinal distortion of the acquired images of the left camera and the right camera to obtain corrected images, and then correcting the images of the left camera and the right camera to enable the image planes of the left camera and the right camera to be parallel;

step S2: performing feature matching on the left and right camera images obtained in the step S2 through a semi-global block matching algorithm, searching corresponding points in the left and right images, solving parallax, finally obtaining depth information of each pixel point of the images, and forming a depth map;

and step S3: fusing pixels of the left image and the right image by adopting a wavelet transformation method on the basis of performing feature matching on the left image and the right image, so as to fuse the left image and the right image into a picture and obtain a fused image;

and step S4: inputting the fusion image obtained in the step S3 into a pre-trained deep learning model of the single-point multi-box detector to detect the types of target objects, wherein the types of the target objects comprise pedestrians, bicycles, motorcycles, abnormal depressions on the road surface, abnormal bulges on the road surface, speed bumps, lane lines and road shoulders; outputting the target object class name and position contained in the input image by the single-point multi-box detector deep learning model, and marking the regression frame and name of the target object in the image;

step S5: searching corresponding depth values in the depth map according to the target object positions obtained in the step S4 to obtain a distance value between each target object and the vehicle;

step S6: recording each pixel point of the fused image as a target point, and obtaining a three-dimensional space coordinate of the target point through coordinate transformation based on the internal parameters and the focal length of the left camera and the right camera and the image coordinates of the target point in the left camera and the right camera, namely obtaining the three-dimensional coordinate of each pixel point in the fused image;

step S7: the two side length sizes of the regression frame can be obtained by calculating the distance between the three-dimensional space coordinates of the pixel points corresponding to the four vertexes of the regression frame of the target object; if the target object type is the abnormal depression of the road surface, the length of the transverse side and the length of the vertical side of the regression frame are respectively the long side and the wide side of the abnormal depression of the road surface, two points with the largest longitudinal coordinate difference in the longitudinal direction in the pixel points at the abnormal depression of the road surface are found, and the longitudinal coordinate difference between the two points is used as the depression depth h of the abnormal depression of the road surface _ac (ii) a If the target object type is the abnormal bulge of the road surface, the length of the vertical edge side of the regression frame is the height h of the abnormal bulge of the road surface _tc (ii) a If the targetThe object type is one of a deceleration strip and a road shoulder, the length of the vertical edge of the regression frame is the height of the target objects, and h is used for the height of the target objects _jc 、h _sc Represents;

step S8: if the fact that the road surface abnormal depression exists on the front road surface is identified, cutting the image according to the regression frame edge of the road surface abnormal depression, extracting the outline of the road surface abnormal depression by adopting a Gaussian Laplace edge detection operator on the cut image containing the regression frame, extracting outline pixel points of the road surface abnormal depression, obtaining three-dimensional space coordinates of the outline pixel points of the road surface abnormal depression according to the three-dimensional space coordinates of each pixel point obtained in the step S6, accumulating the areas of rectangles taking two transversely adjacent pixel points on the outline and two longitudinal pixel points corresponding to the two pixel points as vertexes, and obtaining the accurate area D of the road surface abnormal depression _ac ；

The three-dimensional point cloud identification submodule carries out pavement feature and target identification according to the acquired laser radar three-dimensional point cloud data, and the specific identification process of the three-dimensional point cloud identification submodule is as follows:

step 1: dividing and filtering the point clouds according to the z coordinate value of the three-dimensional point cloud, taking the point cloud with the z coordinate value difference smaller than 0.5 m as a ground range and keeping the point cloud, deleting the point cloud with the maximum z coordinate value larger than 2.5 m, and defining the rest point clouds as target object point clouds and keeping the point clouds;

and 2, step: inputting the point cloud data retained in the step 1 into a pre-trained PointNet + + deep learning classification model, wherein target objects which can be identified by the model comprise pedestrians, bicycles, motorcycles, abnormal depressions on road surfaces, abnormal bulges on road surfaces, deceleration strips, lane lines and road shoulders, and the output of the model is the category and the position of the target objects contained in the input point cloud;

and step 3: extracting the geometrical dimensions of the target object point cloud obtained in the step 2 by adopting a principal component analysis method and a projection method, and respectively obtaining the geometrical dimensions of each target object; for abnormal depression of the road surface, obtaining the depression area D _ar And a depression depth h _ar (ii) a For the abnormal bumps on the road surface, the bump height h is obtained _tr (ii) a For deceleration strips and road shoulders, the height values are obtained and are respectively used for h _jr And h _sr Represents;

the specific identification process of the vehicle sensor signal identification submodule is as follows:

inputting signals acquired by a vehicle sensor into a wavelet neural network model to identify the abnormal depression depth or the abnormal protrusion height of the road surface, wherein the wavelet neural network model is trained in advance by using known road surface height data and corresponding vehicle sensor signals, and a mapping relation between the vehicle sensor signals and the road surface height is established, and the sensor signals used here comprise the following 16 signals: vertical displacement of a left front suspension, vertical displacement of a right front suspension, vertical displacement of a left rear suspension, vertical displacement of a right rear suspension, ground clearance of a front axle, ground clearance of a rear axle, pitch angular displacement of a vehicle body, roll angular displacement of the vehicle body, yaw angular velocity signals, lateral acceleration, longitudinal acceleration and vertical acceleration, vertical vibration acceleration of a left front vehicle body, vertical vibration acceleration of a right front vehicle body, tire pressure of a left front wheel and tire pressure of a right front wheel; the network structure of the wavelet neural network model has 3 layers in total, and comprises an input layer, a hidden layer and an output layer; the input layer has 16 neuron nodes corresponding to the 16 sensor signals and inputs the sequence x _i I =1,2,3,.., 16; the hidden layer has 48 neuron nodes, each node of the hidden layer is composed of a wavelet function, and the expression of the wavelet function is as follows:

the input sequence is processed by wavelet function in the hidden layer to obtain the output value of the hidden layer, the output value u of the jth neuron node of the hidden layer _j Is of the formula:

in the formula, a and b are respectively a scaling factor and a translation factor of a wavelet function, n is the number of nodes of an input layer, the numerical value is 16, m is the number of nodes of a hidden layer, and the numerical value is 48;

the output layer only has one neuron node, and the output of the node is as follows:

in the formula, w _j Is the weight coefficient of the jth hidden layer output; the output of the output layer represents the height h of the road surface _x If the unit is meter, 2 significant figures are reserved after decimal point, and the significant figures have positive and negative scores, if the significant figures are positive figures, the significant figures are the road surface bulges, but whether the bulges are the road surface abnormal bulges or not can not be directly judged, and deceleration strips or road shoulders cannot be excluded; if the negative number indicates that the depression is abnormal depression of the road surface, the height h of the road surface bulge _ts And road surface abnormal depression depth h _as The conversion is obtained by the following formula:

the input signals of the wavelet neural network model have two choices, which input is determined to be used according to the recognition mode, the first input is that the vertical displacement of a left rear suspension, the vertical displacement of a right rear suspension and the ground clearance of a rear axle are set to be 0, and the vertical displacement of a left front suspension, the vertical displacement of a right front suspension, the ground clearance of a front axle, the pitching angular displacement of a vehicle body, the rolling angular displacement of the vehicle body, a yaw angular velocity signal, the lateral acceleration, the longitudinal acceleration and the vertical acceleration, the vertical vibration acceleration of a left front vehicle body, the vertical vibration acceleration of a right front vehicle body, the tire pressure of a left front wheel and the tire pressure of a right front wheel are all normally input; the second input is that the vertical displacement of the left front suspension, the vertical displacement of the right front suspension and the ground clearance of the front axle are set to be 0, and the vertical displacement of the left rear suspension, the vertical displacement of the right rear suspension, the ground clearance of the rear axle, the pitching angular displacement of the automobile body, the rolling angular displacement of the automobile body, the yawing angular velocity signal, the transverse acceleration, the longitudinal acceleration and the vertical acceleration, the vertical vibration acceleration of the left rear automobile body, the vertical vibration acceleration of the right rear automobile body, the tire pressure of the left rear wheel and the tire pressure of the right rear wheel are all normally input;

the beneficial effect of using the 16 sensor signals as the input of the wavelet neural network model to identify the road surface is that the selected 16 sensor signals completely cover response signals of a suspension, an axle, a vehicle body and tires, cover three types of signals of displacement, speed and acceleration, contain comprehensive information and are beneficial to accurately identifying the road surface;

the wavelet neural network has the advantages that the 16 sensor signals are mutually influenced and have certain coupling, compared with a common neural network, the wavelet neural network has stronger learning capacity, higher precision and higher convergence speed, is suitable for processing the condition of complex multi-input signals and is beneficial to quickly identifying the road surface, the relation between the 16 sensor signals and the road surface is not linear, and the wavelet neural network can well fit the nonlinear relation;

the road condition identification module further comprises a data memory, wherein the data memory stores historical abnormal maps; the historical abnormal map comprises the concave depth, concave area and position of the abnormal road surface concave on the road historically identified by the road condition identification module, and the convex height and position information of the abnormal road surface convex, and is used for correcting the current identification result;

the road condition identification module has 3 identification modes: a first identification mode, a second identification mode and a third identification mode;

the driver reminding control module generates reminding information which needs to be provided for a driver based on a recognition result input by the road condition recognition module, wherein the reminding information comprises schematic animation, steering wheel vibration information, voice reminding information and signal lamp flicker information, and sends a control signal of the corresponding reminding information to the driver reminding control module;

the technical scheme is that the driver reminding execution module comprises a vehicle central control display screen, a loudspeaker, a signal lamp, a steering wheel of a steer-by-wire system and a road feel simulation motor assembly, wherein the loudspeaker is installed in a headrest of a driver seat, the signal lamp is installed in the geometric center of the steering wheel, and the driver is correspondingly reminded according to related control signals given by the driver reminding control module.

The three recognition modes and mode selection conditions of the road condition recognition module in the technical scheme are as follows:

(1) The first recognition mode:

when the vehicle runs forwards and the illumination intensity is greater than the illumination intensity threshold value, selecting a first identification mode;

in a first identification mode, calling an image identification submodule, a three-dimensional point cloud identification submodule and a vehicle sensor signal identification submodule to work, wherein a first input signal is selected by inputting a wavelet neural network model in the vehicle sensor signal identification submodule, and the height h of the raised road surface is calculated _ts1 And the abnormal depression depth h of the road surface _as1 And h is combined _ts1 、h _as1 Outputting the data to a driver reminding control module; the identification results respectively obtained by the image identification submodule and the three-dimensional point cloud identification submodule comprise: the height value of the abnormal road surface protrusion, the depth value of the abnormal road surface depression, the area value of the abnormal road surface depression, the height value of the deceleration strip, the height value of the road shoulder, and the positions of pedestrians, bicycles, motorcycles, abnormal road surface depressions, abnormal road surface protrusions, deceleration strips, lane lines and road shoulders on the front road surface and the distance between the pedestrians, bicycles, motorcycles, abnormal road surfaces, abnormal road surface depressions, abnormal road surface protrusions, deceleration strips, lane lines and road shoulders and the vehicle;

multiplying the recognition result of the image recognition submodule by a weight coefficient W _c Multiplying the identification result of the three-dimensional point cloud identification submodule by a weight coefficient W _r Adding the obtained final result to obtain a fused final result, and outputting the fused result to a driver reminding control module, wherein the fused road surface abnormal bulge height value, the road surface abnormal depression depth value, the road surface abnormal depression area value, the deceleration strip height value and the road shoulder height value are respectively used for h _t1 、h _a1 、D _a1 、h _j1 、h _s1 Represents;

W _c 、W _r the weight coefficients of the image recognition submodule and the three-dimensional point cloud recognition submodule are respectively calculated according to the following formula:

in the formula, k _v Is a vehicle speed influence factor, the value is 0.0003, and the unit is h/km; v _x The unit is km/h, and the unit is longitudinal vehicle speed which can be obtained by a vehicle speed sensor; k is a radical of formula _s Is the angular velocity influence factor of the steering wheel, the value is 0.00001, the unit is s/deg;

the steering wheel rotation angular speed can be obtained by a steering wheel rotation angle sensor, and the unit is deg/s;

(2) Second recognition mode

When the vehicle runs forwards and the illumination intensity is smaller than the illumination intensity threshold value, selecting a second identification mode;

in a second recognition mode, calling a three-dimensional point cloud recognition submodule and a vehicle sensor signal recognition submodule to work, wherein a recognition result obtained through the three-dimensional point cloud recognition submodule comprises: the road surface abnormal bulge height value, the road surface abnormal depression depth value, the road surface abnormal depression area value, the deceleration strip height value, the road shoulder height value, the pedestrian, the bicycle, the motorcycle, the road surface abnormal depression, the road surface abnormal bulge, the deceleration strip, the lane line, the position of the road shoulder on the front road surface and the distance between the pedestrian and the vehicle are output to the driver reminding control module;

the wavelet neural network model in the vehicle sensor signal identification submodule is input to select a first input signal, and the road surface protrusion height h is obtained through calculation _ts1 And the abnormal depression depth h of the road surface _as1 And h is _ts1 And h _as1 Outputting the data to a driver reminding control module;

(3) Third recognition mode

Selecting a third identification mode when the vehicle runs in a reverse mode;

in a third identification mode, the road condition identification module only calls the vehicle sensor signal identification submodule to work, the input of the wavelet neural network model in the vehicle sensor signal identification submodule selects a second input signal, and the height h of the raised road surface is calculated _ts2 And the abnormal depression depth h of the road surface _as2 And h is _ts2 、h _as2 Output to the driver reminding control unit module; because the abnormal depressed area of the road surface cannot be known through the third identification mode, the abnormal depressed area of the road surface is defined as a fixed value of 0.05 in the third identification mode and is output to the driver reminding control module to facilitate the relevant calculation.

The driver reminding control module generates control signals of four reminding modes, wherein the control signals comprise schematic animation of a vehicle central control display screen, frequency of vibration reminding of a steering wheel, content of voice reminding, frequency and color of signal lamp flicker reminding;

when the front road surface is identified to have the protrusion or the depression and the pedestrian, the bicycle or the motorcycle is appeared, the driver reminding control module sends the generated schematic animation signal to the driver reminding execution module so as to remind the driver of the depression or the protrusion on the front road surface and remind the driver of paying attention to the pedestrian and the non-motor vehicle appeared in the front; the schematic animation contains the following information: the height and position of the abnormal bump on the road surface, the distance from the vehicle or the depth of the abnormal bump on the road surface, the shape and position of the abnormal bump on the road surface, the distance information between the vehicle and the abnormal bump on the road surface, and the distance between a pedestrian or a bicycle or a motorcycle in front and the vehicle;

the logic that the driver reminding control submodule controls the steering wheel to vibrate for reminding is as follows:

when the road surface abnormal bulge with the bulge height larger than the bulge height threshold value appears on the road surface at the front side, the steering wheel is controlled to start vibration reminding, and the vibration frequency is F _t Hertz; the vibration lasts for t1 second, the preset value of t1 is 3, and the vibration can be modified by a driver independently; the frequency F of the vibration prompt of the steering wheel is determined according to the following formula _t ：

F _t ＝K _d (C ₁ V _x +K _rt h _t )F ₀

In the formula, K _d The driver preference factor is dimensionless, the initial value is 1, and the driver can modify the factor within the range of 0.2 to 1.0; c ₁ The unit is h/km, and the specific value is determined by an applicator; k _rt The convex road surface correction coefficient is determined by an applicator; h is _t Representing the height of the abnormal bump on the road surface; f0 is an initial reference frequency selected from a range of 500 Hz to 3000 Hz;

when the pavement abnormal depression with the depression depth larger than the depression depth threshold value and the depression area larger than the depression area threshold value appears on the pavement on the front side, the steering wheel is controlled to start vibration reminding, and the vibration frequency is F _a Hertz; the vibration lasts for t2 seconds, the preset value of t2 is 3, and the vibration can be modified by a driver independently; the frequency F of the vibration alert of the steering wheel is determined according to the following formula _a ：

F _a ＝K _d (C ₁ V _x +K _ra h _a D _a )F ₀

In the formula, K _ra The correction coefficient of the concave pavement is determined by an applicator; h is _a Depth of depression representing abnormal depression of road surface, D _a Is the depressed area of the abnormal depression of the road surface;

the logic of the voice reminding content played by the driver reminding control module in the technical scheme is as follows:

when the road surface abnormal bulge with the bulge height larger than the bulge height threshold value appears on the road surface at the front side, the voice reminding content is as follows: please note that a road surface abnormal bulge with the height of h1 m is arranged on the road surface d1 m in front, and the road surface abnormal bulge is please avoid in time, the content is played once, the voice reminding volume is Sv1 decibel, and d1 and h1 are the distance and the height between the road surface abnormal bulge and the vehicle respectively; when the road surface that appears the sunken degree of depth on the road surface of front and is greater than sunken degree of depth threshold value and sunken area and is greater than sunken area threshold value is unusually sunken, the content is reminded to pronunciation: please note that, there is a road surface abnormal recess with a height of h2 m and a recess area of D2 square meters on the road surface D2 m ahead, please avoid in time, the content is played once, the voice reminding volume is Sv1 db, D2, h2, D2 are the distance between the road surface abnormal recess and the vehicle, the recess depth and the recess area respectively; the preset value Sv1 is 25 decibels, and a driver is allowed to set an integer value in a range of 20 decibels to 70 decibels to replace the preset value; the convex height threshold, the concave area threshold and the concave depth threshold are set by a driver according to requirements;

the logic of the driver reminding control module for controlling the signal lamp is as follows:

when the road surface abnormal bulge with the bulge height larger than the bulge height threshold value appears on the road surface at the front side, the signal lamp lights red, starts to flicker at the frequency of F1 Hz, and goes out after the vehicle passes through the road surface abnormal bulge; when abnormal pavement depressions with the depression depth larger than the depression depth threshold and the depression area larger than the depression area threshold appear on the pavement above, the signal lamp lights red and starts to flicker at the frequency of F2 Hz, and the signal lamp is turned off after the vehicle passes through the abnormal pavement depressions; the preset values of F1 and F2 are respectively 1 Hz and 2 Hz, and the values can be modified by a driver independently;

the road condition identification module of the technical scheme has the specific process of using the historical abnormal map to correct the current identification result as follows:

downloading information of abnormal depressions and abnormal bulges of the road surface in the historical abnormal map;

respectively comparing the position of each road surface abnormal recess or road surface abnormal protrusion in the history with the positions of the road surface abnormal recesses or road surface abnormal protrusions identified by the three identification submodules;

if the same position in history also has a road surface abnormal depression or a road surface abnormal bulge, the recognition accuracy of the current recognition sub-module at the position is considered to be 100%, and the recognition result does not need to be changed;

if the abnormal depression or the abnormal protrusion of the road surface is not found at the same position in history, considering safety considerations, the identification accuracy of the current identification submodule at the position is also considered to be 100%, and the identification result does not need to be changed;

if the road surface abnormal recess or the road surface abnormal protrusion is recorded at a certain historical position and the recognition submodule does not recognize that the road surface abnormal recess or the road surface abnormal protrusion exists at the position currently, the recognition submodule operates again for detection, and if the recognition result of the second time does not exist, the position is considered to have no road surface abnormal recess or road surface abnormal protrusion; if the second recognition result is present, the first detection is considered as false detection, the second detection result is taken as the standard, the recognition accuracy of the current recognition sub-module at the position is considered as 100%, the recognition result does not need to be changed, and meanwhile, the coefficient n of the false detection of the binocular camera and the laser radar in one week is respectively counted _1c 、n _1r And total number of detections n _2c 、n _2r And calculating the false detection rate Pwc = n of the binocular camera _1c /n _2c False detection rate Pwr = n of laser radar _1r /n _2r Thereby adjusting the efficiency influence coefficient k of the binocular camera _pr And a radar efficiency influence coefficient k _ex The expression of the binocular camera efficiency influence coefficient is as follows:

in the formula, K _pr0 The initial efficiency factor of the binocular camera can be taken as a value within the range of 0.3 to 0.95, and the specific value is determined by an applicator;

the expression of the radar efficiency influence coefficient is as follows:

in the formula, K _ex0 The value of the radar initial efficiency factor can be within the range of 0.2 to 0.98, and the specific value is determined by an application person.

The invention has the beneficial effects that:

1. according to the invention, the road condition is respectively identified through the combination of the binocular camera, the laser radar and the series of vehicle sensors under different light intensities and working conditions, and the identification results are fused, so that the obtained road condition information in front of the vehicle is more accurate, and the method has wider driving environment and driving working condition adaptability.

2. The invention adopts various modes of animation display, steering wheel vibration, voice reminding and signal lamp flashing to provide the front road condition information for the driver in advance, thereby improving the mastering degree of the driver on the front road condition.

3. The steering wheel vibration, voice reminding and signal lamp flicker reminding modes fully consider the preference and the will of the driver, and the specific vibration intensity, the voice reminding volume and the signal lamp flicker frequency are all adjustable, so that the steering wheel vibration, voice reminding and signal lamp flicker reminding method can be suitable for different drivers.

Description of the drawings:

the invention will be further described with reference to the accompanying drawings and examples, in which:

FIG. 1 is a block diagram of the system components of the present invention;

FIG. 2 is a schematic view of the installation location of the binocular camera and the long range lidar of the present invention;

fig. 3 is a schematic diagram illustrating a selection process of an identification mode of the traffic identification module according to the present invention;

fig. 4 is a schematic diagram of three identification modes of the road condition identification module according to the present invention;

the specific implementation mode is as follows:

in order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is further described in detail with reference to the following embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

Referring to fig. 1, the road condition identification and reminding system based on the intelligent automobile steer-by-wire system is characterized by comprising: the system comprises a road condition information acquisition module, a road condition identification module, a driver reminding control module and a driver reminding execution module; the road condition identification module is respectively connected with the road condition information acquisition module and the driver reminding control module; the driver reminding control module is also connected with the driver reminding execution module;

the road condition information acquisition module comprises two binocular cameras, a long-distance laser radar, 17 vehicle sensors and an optical sensor, wherein the vehicle sensors comprise four suspension displacement sensors, a front axle height sensor, a rear axle height sensor, an inertia measurement unit, a yaw angle sensor, a three-axis acceleration sensor, four vehicle body acceleration sensors and four tire pressure detection sensors, which are respectively positioned at the left front, the left back, the right front and the right back of a vehicle; the binocular camera and the long-distance laser radar are respectively used for acquiring an image and a three-dimensional point cloud of a front road surface; the suspension displacement sensors are respectively used for acquiring vertical displacement signals of the suspension; the front axle height sensor and the rear axle height sensor are respectively used for acquiring a ground clearance signal of a front axle and a ground clearance signal of a rear axle; the inertia measurement unit is used for acquiring pitching angular displacement and rolling angular displacement of the car body; the yaw angle sensor is used for acquiring a yaw velocity signal of the vehicle; the three-axis acceleration sensor is used for acquiring transverse acceleration, longitudinal acceleration and vertical acceleration signals of the vehicle; the four vehicle body acceleration sensors are respectively arranged at the positions 5 cm away from the top of the shock absorber on the left front vehicle body, the right front vehicle body, the left rear vehicle body and the right rear vehicle body and are respectively used for acquiring vertical vibration acceleration signals of the left front vehicle body, the right front vehicle body, the left rear vehicle body and the right rear vehicle body; the tire pressure detection sensors are respectively used for collecting tire pressure signals of left front tires, right front tires, left rear tires and right rear tires; the light sensor is used for acquiring an illumination intensity signal of the surrounding environment of the vehicle and comparing the illumination intensity signal with an illumination intensity threshold value; the illumination intensity threshold is calculated by the following expression:

referring to fig. 2, the two binocular cameras and the laser radar according to the present invention are installed at positions:

the two binocular cameras are calibrated before being installed, the left binocular camera 100 and the right binocular camera 200 are respectively installed at the upper left corner and the upper right corner of a windshield, the ground clearance of the installation position is hc, the horizontal distance between the two binocular cameras is Lc, and the hc and the Lc are related to the size parameter of the front windshield of the vehicle and are determined by a vehicle manufacturer; the visual distance of the two binocular cameras is 250 meters, and the left and right visual angle ranges are 120 degrees; the binocular cameras can rotate up and down, images in different angle ranges can be shot through the up-and-down rotation, the downward shooting angle of each binocular camera is a negative angle, the upward shooting angle of each binocular camera is a positive angle, the inclination angle of each binocular camera is Ac, the rotation angle range of each binocular camera is negative 30 degrees to positive 60 degrees, namely the downward maximum rotation angle is 30 degrees, the upward maximum rotation angle is 60 degrees, each two binocular cameras shoot a frame of image every 1 millisecond, the first frame shoots an image in the angle range of negative 30 degrees to 0 degrees, the second frame shoots an image in the angle range of negative 10 degrees to positive 20 degrees, the third frame shoots an image in the angle range of positive 10 degrees to positive 40 degrees, the fourth frame shoots an image in the angle range of positive 30 degrees to positive 60 degrees, the fifth frame shoots an image in the angle range of positive 10 degrees to positive 40 degrees, and the sixth frame shoots an image in the angle range of negative 10 degrees to positive 20 degrees; shooting the six frames of images in sequence to form a shooting cycle, and shooting the images by reciprocating rotation of the camera up and down according to the sequence of the shooting cycle; the long-distance laser radar 300 is installed at the right middle position of a front bumper, the ground clearance of the installation position is hr, and hr is related to the size parameter of the front bumper of the vehicle and is determined by a vehicle manufacturer; the long-distance laser radar can rotate up and down to acquire point cloud data of environments in different angle ranges in a scanning mode, the downward scanning angle of the long-distance laser radar is recorded as a negative angle, the upward scanning angle is recorded as a positive angle, the inclination angle of the long-distance laser radar is recorded as Ar, the range of the up-and-down rotation angle of the long-distance laser radar is from negative 20 degrees to positive 10 degrees, namely the long-distance laser radar can rotate 20 degrees at the maximum downward and 10 degrees at the maximum upward, and the detection distance is 300 meters; the sequence of the long-distance laser radar scanning environment is as follows: firstly, sweeping once when the inclination angle is minus 20 degrees, then sweeping once when the inclination angle is 0 degrees, then sweeping once when the inclination angle is 10 degrees, then sweeping once when the inclination angle is 0 degrees, finally rotating to the position of minus 20 degrees, and reciprocating up and down to sweep according to the sequence; the long-distance laser radar generates three-dimensional point cloud data in the area in front of the vehicle every time of scanning, and each point cloud data is provided with a position label and a distance label: x, y, z, d; x, y, z and d respectively represent a horizontal axis coordinate, a vertical axis coordinate and a distance from the vehicle;

referring to fig. 3, the selection process of the three recognition modes of the road condition recognition module of the present invention is as follows:

when the vehicle runs forwards and the illumination intensity is greater than the illumination intensity threshold value, selecting a first identification mode; when the vehicle runs forwards and the illumination intensity is smaller than the illumination intensity threshold value, selecting a second identification mode; selecting a third identification mode when the vehicle runs in reverse;

referring to fig. 4, the three identification modes of the traffic identification module of the present invention are specifically:

in a first identification mode, calling an image identification submodule, a three-dimensional point cloud identification submodule and a vehicle sensor signal identification submodule to work, wherein a first input signal is selected by inputting a wavelet neural network model in the vehicle sensor signal identification submodule, and the height h of the raised road surface is calculated _ts1 And the abnormal depression depth h of the road surface _as1 And h is _ts1 、h _as1 Outputting the data to a driver reminding control module; the identification results respectively obtained by the image identification submodule and the three-dimensional point cloud identification submodule comprise: the height value of the road surface abnormal bulge, the depth value of the road surface abnormal depression, the value of the area of the road surface abnormal depression, the height value of the deceleration strip, the height value of the road shoulder, and the positions of pedestrians, bicycles, motorcycles, abnormal depressions of the road surface, abnormal bulges of the road surface, the deceleration strip, lane lines and the road shoulder on the front road surface and the distance from the vehicle;

multiplying the recognition result of the image recognition submodule by a weight coefficient W _c Multiplying the identification result of the three-dimensional point cloud identification submodule by a weight coefficient W _r Adding to obtain the fused finalAnd finally, outputting the fusion result to a driver reminding control module, wherein the height value of the road surface abnormal bulge, the depth value of the road surface abnormal depression, the area value of the road surface abnormal depression, the height value of the deceleration strip and the height value of the road shoulder obtained after fusion are respectively used for h _t1 、h _a1 、D _a1 、h _j1 、h _s1 Represents;

W _c 、W _r the weight coefficients of the image identification submodule and the three-dimensional point cloud identification submodule are respectively calculated according to the following formula:

in the formula, k _v The vehicle speed influence factor is determined by an operator, and the specific value is determined by the operator in the unit of h/km; v _x The unit is km/h, and the unit is longitudinal vehicle speed which can be obtained by a vehicle speed sensor; k is a radical of formula _s The specific value is determined by an operator, and the unit is s/deg;

in a second identification mode, calling a three-dimensional point cloud identification submodule and a vehicle sensor signal identification submodule to work, wherein an identification result obtained by the three-dimensional point cloud identification submodule comprises: the road surface abnormal bulge height value, the road surface abnormal depression depth value, the road surface abnormal depression area value, the deceleration strip height value, the road shoulder height value, the pedestrian, the bicycle, the motorcycle, the road surface abnormal depression, the road surface abnormal bulge, the deceleration strip, the lane line, the position of the road shoulder on the front road surface and the distance between the pedestrian and the vehicle are output to the driver reminding control module;

in a third identification mode, the road condition identification module only calls the vehicle sensor signal identification submodule to work, the input of the wavelet neural network model in the vehicle sensor signal identification submodule selects a second input signal, and the height h of the raised road surface is calculated _ts2 And the abnormal depression depth h of the road surface _as2 And h is _ts2 And h _as2 Output to the driver reminding control unit module; since the abnormal road surface depression area cannot be known through the third recognition mode, the abnormal road surface depression area is defined as a fixed value of 0.05 in the third recognition mode and is output to the driver reminding control module for facilitating correlation calculation.

The foregoing discussion is that of the preferred embodiments of the present invention only, and is intended to be illustrative and explanatory only and not limiting of the invention itself. It is intended that the invention not be limited to the particular embodiments disclosed, but that the invention be defined by the claims appended hereto. Furthermore, references made in the foregoing specification to specific embodiments are not to be interpreted as limitations on the scope of the invention or on the definition of terms used in the claims. Various other embodiments and various modifications to the disclosed embodiments will be apparent to those skilled in the art. All such embodiments, changes and modifications that do not depart from the basic inventive concepts are intended to be included within the scope of the appended claims.

Claims

1. A road condition identification and reminding system based on an intelligent automobile steer-by-wire system is characterized by comprising a road condition information acquisition module, a road condition identification module, a driver reminding control module and a driver reminding execution module; the road condition identification module is respectively connected with the road condition information acquisition module and the driver reminding control module; the driver reminding control module is also connected with the driver reminding execution module; road conditions information acquisition module includes two binocular cameras, a long-range laser radar, 17 vehicle sensor, a light sensor, 17 vehicle sensor includes: four suspension displacement sensors, a front axle height sensor, a rear axle height sensor, an inertia measuring unit, a yaw angle sensor, a three-axis acceleration sensor, four vehicle body acceleration sensors and four tire pressure detection sensors, which are respectively positioned at the left front part, the left rear part, the right front part and the right rear part of the vehicle; the binocular camera and the long-distance laser radar are respectively used for acquiring an image and a three-dimensional point cloud of a front road surface; the suspension displacement sensors are respectively used for acquiring vertical displacement signals of the suspension; the front axle and the rear axle height sensors are respectively used for acquiring a ground clearance signal of the front axle and a ground clearance signal of the rear axle; the inertia measurement unit is used for acquiring pitching angular displacement and rolling angular displacement of the car body; the yaw angle sensor is used for acquiring a yaw velocity signal of the vehicle; the three-axis acceleration sensor is used for acquiring signals of transverse acceleration, longitudinal acceleration and vertical acceleration of the vehicle; the four vehicle body acceleration sensors are respectively arranged at the positions 5 cm away from the top of the shock absorber on the left front vehicle body, the right front vehicle body, the left rear vehicle body and the right rear vehicle body and are respectively used for acquiring vertical vibration acceleration signals of the left front vehicle body, the right front vehicle body, the left rear vehicle body and the right rear vehicle body; the four tire pressure detection sensors are respectively used for collecting tire pressure signals of left front tires, right front tires, left rear tires and right rear tires; the light sensor is used for acquiring an illumination intensity signal of the surrounding environment of the vehicle and comparing the illumination intensity signal with an illumination intensity threshold value; the illumination intensity threshold is calculated according to the following expression:

the two binocular cameras are calibrated before installation and are respectively installed at the upper left corner and the upper right corner of the windshield, the ground clearance of the installation position is hc, the horizontal distance between the two binocular cameras is Lc, and the hc and the Lc are related to the size parameter of the front windshield of the vehicle and are determined by a vehicle manufacturer; the visual distance of the two binocular cameras is 250 meters, and the left and right visual angle ranges are 120 degrees; the binocular cameras can rotate up and down, images in different angle ranges can be shot through the up-and-down rotation, the downward shooting angle of each binocular camera is a negative angle, the upward shooting angle of each binocular camera is a positive angle, the inclination angle of each binocular camera is Ac, the rotation angle range of each binocular camera is negative 30 degrees to positive 60 degrees, namely the downward maximum rotation angle is 30 degrees, the upward maximum rotation angle is 60 degrees, each two binocular cameras shoot a frame of image every 1 millisecond, the first frame shoots an image in the angle range of negative 30 degrees to 0 degrees, the second frame shoots an image in the angle range of negative 10 degrees to positive 20 degrees, the third frame shoots an image in the angle range of positive 10 degrees to positive 40 degrees, the fourth frame shoots an image in the angle range of positive 30 degrees to positive 60 degrees, the fifth frame shoots an image in the angle range of positive 10 degrees to positive 40 degrees, and the sixth frame shoots an image in the angle range of negative 10 degrees to positive 20 degrees; shooting the six frames of images in sequence to form a shooting cycle, and shooting the images by reciprocating rotation of the camera up and down according to the sequence of the shooting cycle; the long-distance laser radar is installed at the middle position of the front bumper, the ground clearance of the installation position is hr, and hr is related to the size parameter of the front bumper of the vehicle and is determined by a vehicle manufacturer; the long-distance laser radar can rotate up and down to acquire point cloud data of environments with different angle ranges in a scanning mode, the downward scanning angle of the long-distance laser radar is recorded as a negative angle, the upward scanning angle is recorded as a positive angle, the inclination angle of the long-distance laser radar is recorded as Ar, the range of the up-and-down rotation angle of the long-distance laser radar is from negative 20 degrees to positive 10 degrees, namely the long-distance laser radar can rotate 20 degrees downwards maximally and 10 degrees upwards maximally, and the detection distance is 300 meters; the sequence of the long-distance laser radar scanning environment is as follows: firstly, sweeping once when the inclination angle is minus 20 degrees, then sweeping once when the inclination angle is 0 degrees, then sweeping once when the inclination angle is 10 degrees, then sweeping once when the inclination angle is 0 degrees, finally rotating to the position of minus 20 degrees, and reciprocating up and down to sweep according to the sequence; the long-distance laser radar generates three-dimensional point cloud data in the front area of the vehicle every time of scanning, and each point cloud data is provided with a position label and a distance label: x, y, z, d; x, y, z and d respectively represent a horizontal axis coordinate, a vertical axis coordinate and a distance from the vehicle; the road condition information acquisition module inputs all acquired information into the road condition identification module;

the road condition identification module identifies the road surface characteristics of the front road surface according to the input information, identifies whether pedestrians, bicycles, motorcycles, abnormal depressions of the road surface, abnormal bulges of the road surface, speed bumps, lane lines and road shoulders exist on the front road surface, and inputs the identification result into the driver reminding control module; the road condition identification module is provided with three identification submodules, namely an image identification submodule, a three-dimensional point cloud identification submodule and a vehicle sensor signal identification submodule, and the specific identification process of each identification submodule is as follows:

step S1: respectively preprocessing each frame of image shot by the left camera and the right camera, firstly respectively removing the transverse distortion and the longitudinal distortion of the obtained images of the left camera and the right camera to obtain corrected images, and then correcting the images of the left camera and the right camera to ensure that the image planes of the left camera and the right camera are parallel;

and step S4: inputting the fusion image obtained in the step S3 into a pre-trained deep learning model of the single-point multi-box detector to detect the classes of target objects, wherein the classes of the target objects comprise pedestrians, bicycles, motorcycles, abnormal depressions on road surfaces, abnormal bulges on road surfaces, speed bumps, lane lines and road shoulders; outputting the target object class name and position contained in the input image by the single-point multi-box detector deep learning model, and marking a regression frame and a name of the target object in the image;

step S7: the two side length sizes of the regression frame can be obtained by calculating the distance between the three-dimensional space coordinates of the pixel points corresponding to the four vertexes of the regression frame of the target object; if the target object category is abnormal pavement depression, the side length of the transverse edge and the side length of the vertical edge of the regression frame are respectively the long edge and the wide edge of the abnormal pavement depression, two points with the largest longitudinal coordinate difference in the longitudinal direction in the pixel points of the abnormal pavement depression are found, and the longitudinal coordinate difference between the two points is used as the depression depth h of the abnormal pavement depression _ac (ii) a If the target object type is the abnormal bulge of the road surface, the length of the vertical edge side of the regression frame is the height h of the abnormal bulge of the road surface _tc (ii) a If the target object is one of a speed bump and a road shoulder, the length of the vertical edge of the regression frame is the height of the target objects, and h is used for the length of the vertical edge _jc 、h _sc Represents;

step 2: inputting the point cloud data retained in the step 1 into a pre-trained PointNet + + deep learning classification model, wherein target objects which can be identified by the model comprise pedestrians, bicycles, motorcycles, abnormal depressions on road surfaces, abnormal bulges on road surfaces, deceleration strips, lane lines and road shoulders, and the output of the model is the category and the position of the target objects contained in the input point cloud;

and step 3: extracting the geometrical dimensions of the target object point clouds obtained in the step 2 by adopting a principal component analysis method and a projection method, and respectively obtaining the geometrical dimensions of each target object; for abnormal depression of the road surface, obtaining the depression area D _ar And a depression depth h _ar (ii) a For the abnormal bumps on the road surface, the bump height h is obtained _tr (ii) a For deceleration strips and road shoulders, the height values are obtained and are respectively used for h _jr And h _sr Represents;

inputting signals acquired by a vehicle sensor into a wavelet neural network model to identify the abnormal depression depth or the abnormal protrusion height of the road surface, wherein the wavelet neural network model is trained in advance by using known road surface height data and corresponding vehicle sensor signals, and a mapping relation between the vehicle sensor signals and the road surface height is established, and the sensor signals used here comprise the following 16 signals: vertical displacement of the left front suspension, vertical displacement of the right front suspension, vertical displacement of the left rear suspension, vertical displacement of the right rear suspension, ground clearance of the front axle, ground clearance of the rear axle, pitch angle displacement of the vehicle body, roll angle displacement of the vehicle body, yaw rate signal, lateral acceleration, longitudinal acceleration and vertical acceleration, left front suspension, left rear suspension, right front suspension, left front suspension, right rear suspension, vertical displacement of the vehicle body, yaw rate signal, lateral acceleration, pitch angle displacement of the vehicle body, yaw rate signal, lateral acceleration, longitudinal acceleration, and vertical accelerationVertical vibration acceleration of the vehicle body, vertical vibration acceleration of the right front vehicle body, left front wheel tire pressure and right front wheel tire pressure; the network structure of the wavelet neural network model has 3 layers in total, and comprises an input layer, a hidden layer and an output layer; the input layer has 16 neuron nodes respectively corresponding to the 16 sensor signals, and an input sequence x _i I =1,2,3, ·,16; the hidden layer has 48 neuron nodes, each node of the hidden layer is composed of a wavelet function, and the expression of the wavelet function is as follows:

in the formula, w _j Is the weight coefficient of the jth hidden layer output; the output of the output layer represents the height h of the road surface _x If the unit is meter, 2 significant figures are reserved after decimal point, and the significant figures have positive and negative scores, if the significant figures are positive figures, the significant figures are the road surface bulges, but whether the bulges are the road surface abnormal bulges or not can not be directly judged, and deceleration strips or road shoulders cannot be excluded; if the negative number indicates that the depression is abnormal depression of the road surface, the height h of the road surface protrusion _ts And road surface abnormalityDepth h of depression _as The conversion is obtained by the following formula:

the input signals of the wavelet neural network model have two choices, which input is determined to be used according to the recognition mode, the first input is that the vertical displacement of a left rear suspension, the vertical displacement of a right rear suspension and the ground clearance of a rear axle are set to be 0, and the vertical displacement of a left front suspension, the vertical displacement of a right front suspension, the ground clearance of a front axle, the pitching angular displacement of a vehicle body, the rolling angular displacement of the vehicle body, a yaw angular velocity signal, the lateral acceleration, the longitudinal acceleration and the vertical acceleration, the vertical vibration acceleration of a left front vehicle body, the vertical vibration acceleration of a right front vehicle body, the tire pressure of a left front wheel and the tire pressure of a right front wheel are normally input; the second input is that the vertical displacement of the left front suspension, the vertical displacement of the right front suspension and the ground clearance of the front axle are set to be 0, and the vertical displacement of the left rear suspension, the vertical displacement of the right rear suspension, the ground clearance of the rear axle, the pitching angular displacement of the automobile body, the rolling angular displacement of the automobile body, the yawing angular velocity signal, the transverse acceleration, the longitudinal acceleration and the vertical acceleration, the vertical vibration acceleration of the left rear automobile body, the vertical vibration acceleration of the right rear automobile body, the tire pressure of the left rear wheel and the tire pressure of the right rear wheel are normally input;

the road condition identification module also comprises a data memory, and the data memory stores a historical abnormal map; the historical abnormal map comprises the concave depth, concave area and position of the abnormal concave road surface on the road historically identified by the road condition identification module, and the convex height and position information of the abnormal convex road surface, and is used for correcting the current identification result;

the road condition identification module has 3 identification modes: a first identification pattern, a second identification pattern and a third identification pattern;

the driver reminding control module generates reminding information which needs to be provided for a driver based on a recognition result input by the road condition recognition module, wherein the reminding information comprises schematic animation, steering wheel vibration information, voice reminding information and signal lamp flicker information, and sends a control signal of the corresponding reminding information to the driver reminding control module; the driver reminding execution module comprises a vehicle central control display screen, a loudspeaker, a signal lamp, a steering wheel of a steer-by-wire system and a road feel simulation motor assembly, wherein the loudspeaker is installed in a headrest of a driver seat, the signal lamp is installed in the geometric center of the steering wheel, and the driver is correspondingly reminded according to related control signals given by the driver reminding control module.

2. The intelligent automobile steer-by-wire system-based road condition identification and reminding system according to claim 1, wherein the three identification modes and mode selection conditions of the road condition identification module are as follows:

(1) The first recognition mode:

in a first identification mode, calling an image identification submodule, a three-dimensional point cloud identification submodule and a vehicle sensor signal identification submodule to work, wherein a first input signal is selected by inputting a wavelet neural network model in the vehicle sensor signal identification submodule, and the height h of the raised road surface is calculated _ts1 And the abnormal depression depth h of the road surface _as1 And h is combined _ts1 And h _as1 Outputting the data to a driver reminding control module; the identification results respectively obtained by the image identification submodule and the three-dimensional point cloud identification submodule comprise: the height value of the abnormal road surface protrusion, the depth value of the abnormal road surface depression, the area value of the abnormal road surface depression, the height value of the deceleration strip, the height value of the road shoulder, and the positions of pedestrians, bicycles, motorcycles, abnormal road surface depressions, abnormal road surface protrusions, deceleration strips, lane lines and road shoulders on the front road surface and the distance between the pedestrians, bicycles, motorcycles, abnormal road surfaces, abnormal road surface depressions, abnormal road surface protrusions, deceleration strips, lane lines and road shoulders and the vehicle;

multiplying the recognition result of the image recognition submodule by a weight coefficient W _c Identification of three-dimensional point cloudsMultiplying the recognition result of the submodule by a weight coefficient W _r Adding the obtained final result to obtain a fused final result, and outputting the fused result to a driver reminding control module, wherein the fused road surface abnormal bulge height value, the road surface abnormal depression depth value, the road surface abnormal depression area value, the deceleration strip height value and the road shoulder height value are respectively used for h _t1 、h _a1 、D _a1 、h _j1 And h _s1 Represents;

W _c and W _r The weight coefficients of the image identification submodule and the three-dimensional point cloud identification submodule are respectively calculated according to the following formula:

in the formula, k _v The vehicle speed influence factor is determined by an operator, and the specific value is determined by the operator in the unit of h/km; v _x The unit is km/h, and the unit is longitudinal vehicle speed which can be obtained by a vehicle speed sensor; k is a radical of _s The specific value is determined by an operator, and the unit is s/deg;

(2) The second recognition mode:

in a second recognition mode, calling a three-dimensional point cloud recognition submodule and a vehicle sensor signal recognition submodule to work, wherein a recognition result obtained through the three-dimensional point cloud recognition submodule comprises: the road surface abnormal bulge height value, the road surface abnormal depression depth value, the road surface abnormal depression area value, the deceleration strip height value, the road shoulder height value, the positions of pedestrians, bicycles, motorcycles, abnormal depressions of the road surface, abnormal bulges of the road surface, deceleration strips, lane lines and road shoulders on the front road surface and the distance between the road shoulders and the vehicle, and the recognition results are output to the driver reminding control module;

the wavelet neural network model in the vehicle sensor signal identification submodule inputs and selects a first input signal, and the road surface protrusion height h is obtained through calculation _ts1 And the abnormal depression depth h of the road surface _as1 And h is _ts1 And h _as1 Outputting the data to a driver reminding control module;

(3) The third recognition mode:

selecting a third identification mode when the vehicle runs in reverse;

in a third identification mode, the road condition identification module only calls the vehicle sensor signal identification submodule to work, the input of the wavelet neural network model in the vehicle sensor signal identification submodule selects a second input signal, and the height h of the raised road surface is calculated _ts2 And the abnormal depression depth h of the road surface _as2 And h is _ts2 And h _as2 Output to the driver reminding control unit module; because the abnormal depressed area of the road surface cannot be known through the third identification mode, the abnormal depressed area of the road surface is defined as a fixed value of 0.05 in the third identification mode and is output to the driver reminding control module to facilitate the relevant calculation.

3. The intelligent automobile steer-by-wire system-based road condition identification and reminding system as claimed in claim 1, wherein the driver reminding control module generates control signals of four reminding modes, including a schematic animation of a vehicle central control display screen, a frequency of a steering wheel vibration reminding, a content of a voice reminding, and a frequency and a color of a signal lamp flicker reminding;

the driver reminding control submodule controls the steering wheel to vibrate and remind the steering wheel according to the following logic:

when the road surface abnormal bulge with the bulge height larger than the bulge height threshold value appears on the road surface at the front side, the steering wheel is controlled to start vibration reminding, and the vibration frequency is F _t Hertz; the vibration lasts for t1 second, the preset value of t1 is 3, and the vibration can be modified by a driver independently; the frequency F of the vibration alert of the steering wheel is determined according to the following formula _t ：

F _t ＝K _d (C ₁ V _x +K _rt h _t )F ₀

In the formula, K _d The driver preference factor is dimensionless, the initial value is set to 1 and can be modified by the driver within the range of 0.2 to 1.0; c ₁ The unit is h/km, and the specific value is determined by an applicator; k _rt The convex road surface correction coefficient is determined by an applicator; h is _t Representing the height of the abnormal bump on the road surface; f0 is an initial reference frequency selected from a range of 500 Hz to 3000 Hz;

F _a ＝K _d (C ₁ V _x +K _ra h _a D _a )F ₀

the logic of the voice reminding content played by the driver reminding control module is as follows:

when the road surface abnormal bulge with the bulge height larger than the bulge height threshold value appears on the road surface at the front side, the voice reminding content is as follows: please note that a road surface abnormal bulge with the height of h1 m is arranged on the road surface d1 m in front, and the road surface abnormal bulge is please avoid in time, the content is played once, the voice reminding volume is Sv1 decibel, and d1 and h1 are the distance and the height between the road surface abnormal bulge and the vehicle respectively; when the road surface that appears the sunken degree of depth on the road surface of front and is greater than sunken degree of depth threshold value and sunken area and is greater than sunken area threshold value is unusually sunken, the content is reminded to pronunciation: please note that, a road surface D2 meters ahead is provided with a road surface abnormal recess with a height of h2 meters and a recess area of D2 square meters, and please avoid in time, the content is played once, the voice reminding volume is Sv1 decibel, and D2, h2 and D2 are the distance between the road surface abnormal recess and the vehicle, the recess depth and the recess area of the road surface abnormal recess respectively; the preset value Sv1 is 25 decibels, and a driver is allowed to set an integer value in a range of 20 decibels to 70 decibels to replace the preset value; the convex height threshold, the concave area threshold and the concave depth threshold are all set by a driver according to requirements;

when the road surface abnormal bulge with the bulge height larger than the bulge height threshold value appears on the road surface at the front side, the signal lamp lights a red light, starts to flicker at the frequency of F1 Hz, and is turned off after the vehicle passes through the road surface abnormal bulge; when abnormal pavement depressions with the depression depth larger than the depression depth threshold and the depression area larger than the depression area threshold appear on the pavement above, the signal lamp lights red and starts to flicker at the frequency of F2 Hz, and the signal lamp is turned off after the vehicle passes through the abnormal pavement depressions; the preset values of F1 and F2 are 1 hz and 2 hz, respectively, and the values can also be modified autonomously by the driver.

4. The system of claim 1, wherein the road condition identification module corrects the current identification result by using a historical abnormal map in the following specific process:

if the road surface abnormal recess or the road surface abnormal protrusion is recorded at a certain historical position and the recognition submodule does not recognize that the road surface abnormal recess or the road surface abnormal protrusion exists at the position currently, the recognition submodule operates again for detection, and if the recognition result of the second time does not exist, the position is considered to have no road surface abnormal recess or road surface abnormal protrusion; if the second recognition result is present, the first detection is regarded as false detection, the second detection result is taken as the reference, the recognition accuracy of the current recognition sub-module at the position is regarded as 100%, the recognition result does not need to be changed, and meanwhile, the coefficients n of the false detection of the binocular camera and the laser radar in one week are respectively counted _1c And n _1r And the total number of detections n _2c And n _2r And calculating the false detection rate Pwc = n of the binocular camera _1c /n _2c False detection rate Pwr = n of laser radar _1r /n _2r Thereby adjusting the efficiency influence coefficient k of the binocular camera _pr And a radar efficiency influence coefficient k _ex The expression of the binocular camera efficiency influence coefficient is as follows:

in the formula, K _pr0 The initial efficiency factor of the binocular camera can be selected from the range of 0.3 to 0.95,the specific value is determined by an applicator;

the expression of the radar efficiency influence coefficient is as follows:

in the formula, K _ex0 The value of the radar initial efficiency factor can be in the range of 0.2 to 0.98, and the specific value is determined by an application person.